Bocklitz-Lab
/

lit2vec-subfield-classifier-model

@@ -1,325 +1,224 @@
----
-language:
-  - en
-library_name: keras
-pipeline_tag: text-classification
-license: cc-by-4.0
-tags:
-  - chemistry
-  - scientific-literature
-  - multilabel
-  - classification
-  - embeddings
-  - keras
-datasets:
-  - Bocklitz-Lab/lit2vec-subfield-classifier-dataset
-model-index:
-- name: lit2vec-subfield-classifier-model
-  results:
-  - task:
-      name: Multi-label Text Classification
-      type: text-classification
-    dataset:
-      name: Lit2Vec Subfield Classifier Dataset
-      type: Bocklitz-Lab/lit2vec-subfield-classifier-dataset
-      split: test
-      size: 7482
-    metrics:
-      - type: micro_f1
-        value: 0.81
-      - type: macro_f1
-        value: 0.75
-      - type: weighted_f1
-        value: 0.80
-      - type: pr_auc
-        value: 0.8688
-      - type: roc_auc
-        value: 0.9725
----
-# Lit2Vec Subfield Classifier (MLP)
-> Multi-label classifier for chemistry subfields using dense text embeddings.
-**Repo:** [https://huggingface.co/Bocklitz-Lab/lit2vec-subfield-classifier-model](https://huggingface.co/Bocklitz-Lab/lit2vec-subfield-classifier-model)
-**Dataset:** [https://huggingface.co/datasets/Bocklitz-Lab/lit2vec-subfield-classifier-dataset](https://huggingface.co/datasets/Bocklitz-Lab/lit2vec-subfield-classifier-dataset)
----
-## Model Summary
-This model is a **Keras MLP** for **multi-label** classification of chemistry-related scientific texts.
-It consumes a **dense embedding vector** and predicts one or more **subfields** (e.g., *Catalysis*, *Energy Chemistry*, *Materials Science*).
-* **Input:** dense embedding vector (`embedding` from the dataset)
-* **Output:** 18 sigmoid probabilities (one per subfield)
-* **Task:** multilabel text classification (thresholded at 0.5 by default)
----
-## Intended Use & Limitations
-**Intended use**
-* Subfield tagging for chemistry abstracts/summaries
-* Metadata enrichment for literature databases
-* Retrieval, filtering, and analytics
-**Limitations**
-* Trained only on chemistry texts → may not generalize to other domains
-* Requires the **same embedding space** as the dataset encoder (raw text is **not** accepted directly)
-* Long-tail subfields (few examples) may have lower F1
----
-## Labels
-| ID | Subfield                           |
-| -- | ---------------------------------- |
-| 0  | Catalysis                          |
-| 1  | Organic Chemistry                  |
-| 2  | Polymer Chemistry                  |
-| 3  | Inorganic Chemistry                |
-| 4  | Materials Science                  |
-| 5  | Analytical Chemistry               |
-| 6  | Physical Chemistry                 |
-| 7  | Biochemistry                       |
-| 8  | Environmental Chemistry            |
-| 9  | Energy Chemistry                   |
-| 10 | Medicinal Chemistry                |
-| 11 | Chemical Engineering               |
-| 12 | Supramolecular Chemistry           |
-| 13 | Radiochemistry & Nuclear Chemistry |
-| 14 | Forensic & Legal Chemistry         |
-| 15 | Food Chemistry                     |
-| 16 | Chemical Education                 |
-| 17 | Others                             |
-The repo includes `label_mapping.json`.
----
-## Training Details
-* **Framework:** TensorFlow/Keras
-* **Architecture:**
-  * Input → Dense(256, ReLU) → (BatchNorm) → Dropout(0.3)
-  * Dense(256, ReLU) → (BatchNorm) → Dropout(0.3)
-  * Output: Dense(18, **sigmoid**)
-* **Loss:** **Weighted** Binary Cross-Entropy (per-class weights from train frequency)
-* **Optimizer:** Adam (ReduceLROnPlateau)
-* **Callbacks:** EarlyStopping (restore best), ReduceLROnPlateau, optional W\&B logging
-* **Validation:** 5-fold CV on train+val; final training on official splits
-* **Best epoch (val):** 11 (from W\&B)
----
-## Evaluation
-**Validation (final run):**
-* PR-AUC: **0.8688**
-* ROC-AUC: **0.9725**
-* Binary Accuracy: **0.9597**
-**Test (held-out split, threshold = 0.5):**
-* **Micro F1:** **0.81**
-* **Macro F1:** **0.75**
-* **Weighted F1:** **0.80**
-* **Samples F1:** **0.80**
-**Per-label (F1, support):**
-| Subfield                           | F1       | Support |
-| ---------------------------------- | -------- | ------- |
-| Catalysis                          | 0.80     | 197     |
-| Organic Chemistry                  | 0.70     | 245     |
-| Polymer Chemistry                  | 0.72     | 120     |
-| Inorganic Chemistry                | 0.71     | 203     |
-| Materials Science                  | 0.80     | 917     |
-| Analytical Chemistry               | 0.71     | 633     |
-| Physical Chemistry                 | 0.63     | 240     |
-| Biochemistry                       | **0.92** | 2106    |
-| Environmental Chemistry            | 0.79     | 508     |
-| Energy Chemistry                   | 0.79     | 166     |
-| Medicinal Chemistry                | 0.82     | 1343    |
-| Chemical Engineering               | **0.53** | 413     |
-| Supramolecular Chemistry           | 0.68     | 34      |
-| Radiochemistry & Nuclear Chemistry | 0.65     | 20      |
-| Forensic & Legal Chemistry         | 0.70     | 16      |
-| Food Chemistry                     | 0.83     | 282     |
-| Chemical Education                 | 0.85     | 20      |
-| Others                             | 0.83     | 19      |
-**Notes**
-* Strong performance on frequent classes like **Biochemistry**, **Medicinal Chemistry**, **Food Chemistry**.
-* Lower F1 on long-tail or heterogeneous labels like **Chemical Engineering** and **Physical Chemistry**.
-If included in the repo, the plot `f1_vs_freq.png` shows F1 vs. training label frequency.
----
-## Usage
-> The model expects the **same embedding space** as the dataset’s `embedding`. If you want to apply it to new texts, you must compute embeddings with the **same encoder** used to create the dataset.
-### Quick start (load from Hub, inference)
-```python
-# pip install -r requirements.txt
-import json
-import numpy as np
-from typing import List, Tuple
-from huggingface_hub import hf_hub_download
-from tensorflow import keras
-from sentence_transformers import SentenceTransformer
-REPO_ID = "Bocklitz-Lab/lit2vec-subfield-classifier-model"
-EMBED_MODEL = "intfloat/e5-large-v2"   # must match what you used to train!
-TEXT_PREFIX = {"abstract": "abstract: ", "summary": "summary: "}  # keep consistent with your pipeline
-THRESHOLD = 0.5  # decision threshold for multilabel
-# ----- Load model + label mapping -----
-model_path = hf_hub_download(REPO_ID, filename="mlp_model.h5")
-label_map_path = hf_hub_download(REPO_ID, filename="label_mapping.json")
-with open(label_map_path, "r", encoding="utf-8") as f:
-    mapping = json.load(f)
-index_to_label = {int(k): v for k, v in mapping["index_to_label"].items()}
-model = keras.models.load_model(model_path, compile=False)  # inference only
-encoder = SentenceTransformer(EMBED_MODEL)
-def encode_text(text: str, text_type: str = "summary") -> np.ndarray:
-    """
-    Encode text into a normalized embedding compatible with the classifier.
-    text_type: "summary" or "abstract" (affects prefix)
-    """
-    prefix = TEXT_PREFIX.get(text_type, "")
-    emb = encoder.encode([prefix + text], normalize_embeddings=True)  # shape: (1, D)
-    return emb.astype("float32")
-def predict_labels_from_text(text: str, text_type: str = "summary", threshold: float = THRESHOLD
-                            ) -> Tuple[List[int], List[str], np.ndarray]:
-    """
-    Returns (predicted_ids, predicted_labels, probabilities)
-    """
-    x = encode_text(text, text_type=text_type)       # (1, D)
-    probs = model.predict(x, verbose=0)[0]           # (18,)
-    pred_ids = [i for i, p in enumerate(probs) if p > threshold]
-    pred_labels = [index_to_label[i] for i in pred_ids]
-    return pred_ids, pred_labels, probs
-# ----- Example -----
-if __name__ == "__main__":
-    sample_text = (
-        "The adsorption capacity of Helix aspera shell for Pb2+, Zn2+ and Ni2+ has been studied. This shell has the potential of adsorbing Pb2+, Zn2+ and Ni2+ from aqueous solution. The adsorption potentials of Helix aspera shell is largely influenced by the ionic character of the ions and occurred according to the order Pb2+ > Ni2+ > Zn2+. The adsorption of Pb(II), Zn(II) and Ni(II) ions from aqueous solutions by Helix aspera shell is thermodynamically feasible and is consistent with the models of Langmuir and Freundlich adsorption isotherms. From the results of the study, the shell of Helix aspera is recommended for use in the removal of Pb2+, Zn2+ and Ni2+ from aqueous solution."
-    )
-    ids, labels, probs = predict_labels_from_text(sample_text, text_type="abstract", threshold=0.5)
-    print("Predicted IDs:", ids)
-    print("Predicted Labels:", labels)
-    print("Top scores:", sorted(((index_to_label[i], float(p)) for i, p in enumerate(probs)),
-                               key=lambda x: x[1], reverse=True)[:5])
-```
-### Batch inference
-```python
-X = np.load("embeddings_batch.npy").astype("float32")  # shape (N, D)
-probs = model.predict(X, verbose=0)  # shape (N, 18)
-labels_per_row = [[index_to_label[i] for i, p in enumerate(row) if p > 0.5] for row in probs]
-```
-> **Tip:** If you need to **fine-tune** the model, recompile it and reuse the weighted BCE:
->
-> * At load time, pass `compile=False` and then `model.compile(loss="binary_crossentropy", ...)` for inference-only.
-> * For training with class weights, reintroduce the weighted loss you used in the training script.
----
-## Files in this repository
-* `mlp_model.h5` – Keras model weights/graph
-* `label_mapping.json` – name ↔ id mapping
-* `training_history.json` – training curves (optional)
-* `f1_vs_freq.png` – F1 vs frequency plot (optional)
-* `README.md` – this model card
----
-## Dataset
-**Lit2Vec Subfield Classifier Dataset**
-* \~39.9k CC-BY scientific texts with **embeddings** and **subfield labels**
-* Splits: \~80/10/10 (train/val/test)
-* [https://huggingface.co/datasets/Bocklitz-Lab/lit2vec-subfield-classifier-dataset](https://huggingface.co/datasets/Bocklitz-Lab/lit2vec-subfield-classifier-dataset)
----
-## Model Index
-```yaml
-model-index:
-- name: Lit2Vec Subfield Classifier (MLP)
-  results:
-  - task:
-      type: text-classification
-      name: Multi-label text classification
-    dataset:
-      name: Lit2Vec Subfield Classifier Dataset
-      type: Bocklitz-Lab/lit2vec-subfield-classifier-dataset
-      split: test
-    metrics:
-    - type: micro_f1
-      value: 0.81
-    - type: macro_f1
-      value: 0.75
-    - type: weighted_f1
-      value: 0.80
-    - type: pr_auc
-      value: 0.8688
-    - type: roc_auc
-      value: 0.9725
-```
----
-## License
-* **Model:** CC BY 4.0
-* **Dataset:** CC BY 4.0
----
-## Citations
-**Dataset**
-```bibtex
-@dataset{lit2vec_classifier_2025,
-  author       = {Mahmoud Amiri, Thomas Bocklitz},
-  title        = {Lit2Vec Subfield Classifier Dataset},
-  year         = {2025},
-  publisher    = {Hugging Face},
-  howpublished = {\url{https://huggingface.co/datasets/Bocklitz-Lab/lit2vec-subfield-classifier-dataset}},
-  note         = {Submitted to Nature Scientific Data}
-}
-```
-**Model**
-```bibtex
-@misc{lit2vec_mlp_classifier_2025,
-  title        = {Lit2Vec Subfield Classifier Model},
-  author       = {Mahmoud Amiri and Thomas Bocklitz},
-  year         = {2025},
-  publisher    = {Hugging Face},
-  howpublished = {\url{https://huggingface.co/Bocklitz-Lab/lit2vec-subfield-classifier-model}}
-}
-```

+---
+language:
+  - en
+library_name: keras
+pipeline_tag: text-classification
+license: apache-2.0
+tags:
+  - chemistry
+  - scientific-literature
+  - multilabel
+  - classification
+  - embeddings
+  - keras
+datasets:
+  - Bocklitz-Lab/lit2vec-subfield-classifier-dataset
+model-index:
+- name: lit2vec-subfield-classifier-model
+  results:
+  - task:
+      name: Multi-label Text Classification
+      type: text-classification
+    dataset:
+      name: Lit2Vec Subfield Classifier Dataset
+      type: Bocklitz-Lab/lit2vec-subfield-classifier-dataset
+      split: test
+      size: 3990
+    metrics:
+      - type: micro_f1
+        value: 0.8078
+      - type: macro_f1
+        value: 0.7476
+      - type: weighted_f1
+        value: 0.8003
+      - type: micro_roc_auc
+        value: 0.9844
+      - type: macro_roc_auc
+        value: 0.9778
+      - type: micro_pr_auc
+        value: 0.9014
+      - type: macro_pr_auc
+        value: 0.8425
+---
+# Lit2Vec Subfield Classifier (MLP)
+A multi-label classifier that assigns chemistry abstracts and summaries to one or more of 18 subfields. It is a small Keras MLP that works on top of `intfloat/e5-large-v2` embeddings and is used to produce the `predicted_subfield` field of the Lit2Vec workflow ([paper](https://arxiv.org/abs/2604.12498), [code](https://github.com/Bocklitz-Lab/Lit2Vec-code)). Training code: [lit2vec-subfield-classifier](https://github.com/Bocklitz-Lab/lit2vec-subfield-classifier). Dataset: [lit2vec-subfield-classifier-dataset](https://huggingface.co/datasets/Bocklitz-Lab/lit2vec-subfield-classifier-dataset). Demo: [Space](https://huggingface.co/spaces/Bocklitz-Lab/lit2vec-subfield-classifier-space).
+- **Input:** a 1,024-dimensional, L2-normalised e5-large-v2 embedding of the text `abstract: <text>` or `summary: <text>`
+- **Output:** 18 sigmoid scores, one per subfield; a label is assigned when its score is above 0.5
+- **Not accepted:** raw text. You have to embed it first, as shown below.
+## Labels
+| ID | Subfield |
+|---:|---|
+| 0 | Catalysis |
+| 1 | Organic Chemistry |
+| 2 | Polymer Chemistry |
+| 3 | Inorganic Chemistry |
+| 4 | Materials Science |
+| 5 | Analytical Chemistry |
+| 6 | Physical Chemistry |
+| 7 | Biochemistry |
+| 8 | Environmental Chemistry |
+| 9 | Energy Chemistry |
+| 10 | Medicinal Chemistry |
+| 11 | Chemical Engineering |
+| 12 | Supramolecular Chemistry |
+| 13 | Radiochemistry & Nuclear Chemistry |
+| 14 | Forensic & Legal Chemistry |
+| 15 | Food Chemistry |
+| 16 | Chemical Education |
+| 17 | Others |
+The mapping is also in `label_mapping.json`. The training labels were generated by GPT-4o from a controlled vocabulary. They were not curated by experts (see Limitations).
+## Quick start
+```python
+# pip install -r requirements.txt
+import json
+import numpy as np
+from huggingface_hub import hf_hub_download
+from tensorflow import keras
+from sentence_transformers import SentenceTransformer
+REPO_ID = "Bocklitz-Lab/lit2vec-subfield-classifier-model"
+EMBED_MODEL = "intfloat/e5-large-v2"
+TEXT_PREFIX = {"abstract": "abstract: ", "summary": "summary: "}  # the prefixes used for the training embeddings
+THRESHOLD = 0.5
+model = keras.models.load_model(hf_hub_download(REPO_ID, "mlp_model.h5"), compile=False)
+with open(hf_hub_download(REPO_ID, "label_mapping.json"), encoding="utf-8") as f:
+    index_to_label = {int(k): v for k, v in json.load(f)["index_to_label"].items()}
+encoder = SentenceTransformer(EMBED_MODEL)
+def predict(text: str, text_type: str = "abstract", threshold: float = THRESHOLD):
+    x = encoder.encode([TEXT_PREFIX[text_type] + text], normalize_embeddings=True).astype("float32")
+    probs = model.predict(x, verbose=0)[0]
+    labels = [index_to_label[i] for i, p in enumerate(probs) if p > threshold]
+    return labels, probs
+labels, probs = predict("The adsorption capacity of Helix aspera shell for Pb2+, Zn2+ and Ni2+ has been studied ...")
+print(labels)
+```
+The text prefix matters. The embeddings in the training dataset were computed with `abstract: ` for abstracts and `summary: ` for summaries. If you embed text without the prefix, or with e5's usual `passage: `, the vectors are close but not identical (cosine similarity of about 0.998), and scores can shift slightly.
+For many texts, embed them in a batch and call `model.predict(X)` on the stacked array.
+## Training
+| | |
+|---|---|
+| Framework | TensorFlow / Keras 2.15 |
+| Architecture | Dense(256, ReLU) - BatchNorm - Dropout(0.3), twice, then Dense(18, sigmoid); 334,866 parameters |
+| Loss | Binary cross-entropy weighted per class by (negatives / positives) of the training split |
+| Optimiser | Adam, learning rate 1e-3, batch size 32, at most 50 epochs |
+| Callbacks | EarlyStopping on validation loss (patience 5, best weights restored), ReduceLROnPlateau (factor 0.5, patience 3) |
+| Run | stopped after 17 epochs; the weights of epoch 12 (lowest validation loss) are the ones in this repository |
+| Model selection | 5-fold cross-validation on train and validation, then one final training on the official splits |
+| Seed | 42 |
+The full settings are in `config.yaml`, and `training_history.json` holds the curves of the final run.
+## Evaluation
+Validation metrics at epoch 12: loss 1.064, binary accuracy 0.964, ROC-AUC 0.972, PR-AUC 0.854 (macro-averaged over the labels, as computed by Keras).
+Test set (3,990 records, threshold 0.5):
+| Metric | Value |
+|---|---:|
+| Micro F1 | 0.808 |
+| Macro F1 | 0.748 |
+| Weighted F1 | 0.800 |
+| Samples F1 | 0.804 |
+| Micro precision / recall | 0.854 / 0.766 |
+| Exact-match accuracy | 0.508 |
+| ROC-AUC (micro / macro) | 0.984 / 0.978 |
+| PR-AUC (micro / macro) | 0.901 / 0.843 |
+We recomputed these numbers from the weights in this repository and the test split of the dataset, and they match the evaluation outputs in the training repository.
+### The test split overlaps with the training split
+Every paper in the dataset appears twice, once with its abstract and once with its GPT-4o summary, and both records carry the same labels. The dataset was split randomly by record, not by paper. As a result, 3,573 of the 3,990 test records (90%) have the other text of the same paper in the training or validation split. The scores above are therefore somewhat optimistic. On test records whose paper does not occur in the training split the model scores:
+| Test records | n | Micro F1 | Macro F1 | Weighted F1 |
+|---|---:|---:|---:|---:|
+| All | 3,990 | 0.808 | 0.748 | 0.800 |
+| Paper not in the training split | 838 | 0.804 | 0.707 | 0.794 |
+| Paper not in training or validation | 417 | 0.790 | 0.680 | 0.775 |
+The micro-averaged scores change little. Macro F1 drops more, partly because the rare classes have only 4 to 6 examples in the smaller subsets. For an estimate of how the model behaves on new papers, use the second row.
+### F1 per label (full test set)
+| Subfield | F1 | Support |
+|---|---:|---:|
+| Catalysis | 0.80 | 197 |
+| Organic Chemistry | 0.70 | 245 |
+| Polymer Chemistry | 0.72 | 120 |
+| Inorganic Chemistry | 0.71 | 203 |
+| Materials Science | 0.80 | 917 |
+| Analytical Chemistry | 0.71 | 633 |
+| Physical Chemistry | 0.63 | 240 |
+| Biochemistry | 0.92 | 2,106 |
+| Environmental Chemistry | 0.79 | 508 |
+| Energy Chemistry | 0.79 | 166 |
+| Medicinal Chemistry | 0.82 | 1,343 |
+| Chemical Engineering | 0.53 | 413 |
+| Supramolecular Chemistry | 0.68 | 34 |
+| Radiochemistry & Nuclear Chemistry | 0.65 | 20 |
+| Forensic & Legal Chemistry | 0.70 | 16 |
+| Food Chemistry | 0.83 | 282 |
+| Chemical Education | 0.85 | 20 |
+| Others | 0.83 | 19 |
+Support is the number of test records that carry the label. The model is strongest on frequent classes such as Biochemistry, Medicinal Chemistry and Materials Science, and weakest on Chemical Engineering and Physical Chemistry.
+## Intended use
+Tagging chemistry abstracts and summaries with broad subfields for browsing, filtering and retrieval, for example the subfield labels in the Lit2Vec corpus.
+## Limitations
+- **Labels are model-generated.** The training labels come from GPT-4o. Two domain experts reviewed a random sample of 45 annotated abstracts and found them generally reasonable (see the paper). The classifier learns the labelling habits and the errors of GPT-4o, so treat the output as navigation metadata and not as ground truth.
+- **Rare classes** (Supramolecular, Radiochemistry, Forensic, Chemical Education, Others) have very few examples, and their scores are noisy.
+- **Overlapping splits.** See the section above.
+- **Domain.** It was trained on chemistry abstracts and summaries. It may not work on other fields or on long full text.
+- **Embedding space.** It only works with the e5-large-v2 embedding described above.
+## Files
+- `mlp_model.h5`: the Keras model
+- `label_mapping.json`: label name and ID mapping
+- `training_history.json`: training and validation curves of the final run
+- `config.yaml`: training settings
+- `requirements.txt`: packages for the quick start
+## Citation
+Please cite the Lit2Vec paper:
+```bibtex
+@misc{amiri2026lit2vec,
+  title         = {Lit2Vec: A Reproducible Workflow for Building a Legally Screened Chemistry Corpus from S2ORC for Downstream Retrieval and Text Mining},
+  author        = {Amiri, Mahmoud and Jafari, Jamile Mohammad and Mostafapour, Sara and Bocklitz, Thomas},
+  year          = {2026},
+  eprint        = {2604.12498},
+  archivePrefix = {arXiv},
+  doi           = {10.48550/arXiv.2604.12498},
+  url           = {https://arxiv.org/abs/2604.12498}
+}
+```
+## License
+The model weights are released under Apache-2.0. The training dataset is released under CC BY 4.0. Its labels and summaries were generated with GPT-4o, so check OpenAI's terms of use if you use them to train other models.
+## Acknowledgements
+The embeddings come from `intfloat/e5-large-v2`. The abstracts come from S2ORC (Allen Institute for AI and Semantic Scholar). The work was developed in the Bocklitz Lab at the Leibniz Institute of Photonic Technology (Leibniz IPHT) and Friedrich Schiller University Jena and was funded by the BMFTR within the programme Photonics Research Germany (13N15466, 13N15710, 13N15715). Questions: mahmoud.amiri@uni-jena.de.

config.yaml CHANGED Viewed

@@ -1,3 +1,5 @@
 seed: 42
 mixed_precision: false

+# Training settings of this model. The training code is in
+# https://github.com/Bocklitz-Lab/lit2vec-subfield-classifier (src/train.py, configs/train.yaml).
 seed: 42
 mixed_precision: false

requirements.txt CHANGED Viewed

@@ -1,15 +1,8 @@
-numpy==1.26
-scikit-learn==1.2
-tensorflow==2.15.0
-matplotlib==3.7
-wandb==0.16
-datasets==2.19
-pyarrow==12
 sentence-transformers==2.7.0
-huggingface_hub>=0.23.0
-# Torch is required by sentence-transformers
-# CPU-only build (safer if you don’t need GPU)
-torch==2.3.1+cpu
-torchvision==0.18.1+cpu
-torchaudio==2.3.1+cpu
---index-url https://download.pytorch.org/whl/cpu

+# Packages for the quick start in README.md (inference only)
+tensorflow-cpu==2.15.0
+numpy==1.26.4
 sentence-transformers==2.7.0
+huggingface_hub>=0.23.0,<1.0
+# sentence-transformers installs PyTorch. For a smaller CPU-only build, install it first with:
+#   pip install torch --index-url https://download.pytorch.org/whl/cpu