| --- |
| language: |
| - en |
| base_model: allenai/scibert_scivocab_cased |
| pipeline_tag: token-classification |
| tags: |
| - biomedical |
| - relation-extraction |
| - named-entity-recognition |
| - drugprot |
| - scibert |
| --- |
| |
| # DrugProt SciBERT: joint chemical/gene NER and chemical → gene relation extraction |
|
|
| A SciBERT model fine-tuned on the DrugProt corpus (BioCreative VII Track 1). It finds **chemical** and |
| **gene/protein** mentions in biomedical text and classifies the relation a sentence states from a chemical to a |
| gene/protein into the 13 DrugProt relation types. |
|
|
| This is a research checkpoint. It is **not** a Hugging Face `transformers` model class and cannot be loaded with |
| `AutoModel` or `pipeline`. |
|
|
| ## Architecture |
|
|
| - **Encoder:** `allenai/scibert_scivocab_cased` (BERT-base, cased vocabulary), fine-tuned. |
| - **NER head:** span classification. Every candidate span of up to 24 sub-words is represented as |
| [first; mean; last] encoder vectors and classified as `None`, `CHEMICAL` or `GENE` (softmax). |
| - **Relation head (typed entity markers):** for each same-sentence (chemical, gene) pair, the sentence is encoded |
| again with typed text markers around the two mentions (`@ * chemical * … @`, `# ^ gene ^ … #`). The concatenated |
| hidden states of the chemical and gene start markers are classified over the 13 relation types with a per-label |
| sigmoid (multi-label) and a decision threshold of 0.5. |
| - **Training:** joint NER + relation loss (L = L_NER + 1.0 · L_REL), AdamW with linear warm-up/decay |
| (encoder lr 3e-5, head lr 1e-3), batch size 16, max sequence length 256, 8 epochs with early stopping, seed 42. |
|
|
| ## Data |
|
|
| - **Corpus:** DrugProt v1.2 (Miranda-Escalada, Krallinger et al., BioCreative VII), PubMed abstracts annotated by |
| experts, [doi:10.5281/zenodo.5119892](https://doi.org/10.5281/zenodo.5119892). |
| - **Training:** the official training set (3,500 abstracts) minus 350 abstracts held out for checkpoint selection. |
| - **Checkpoint selection:** on those 350 held-out training abstracts (epoch 7 of 8). |
| - **Evaluation:** the official **development** set (750 abstracts), used for neither training nor selection. The |
| official test-set gold labels were not used. |
|
|
| ## Evaluation (official DrugProt development set, 750 abstracts) |
|
|
| | Task | Precision | Recall | F1 | |
| |---|---|---|---| |
| | NER, all entities (strict span + type) | 88.95 | 90.52 | **89.73** | |
| | NER, `CHEMICAL` | 88.86 | 91.92 | 90.36 | |
| | NER, `GENE` | 89.05 | 88.98 | 89.02 | |
| | Relations with gold entities (official BioCreative VII scorer, micro) | 77.4 | 71.9 | **74.5** | |
| | Relations end-to-end (predicted entities, strict) | 66.29 | 62.43 | **64.30** | |
|
|
| Relation F1 by type (gold entities, official scorer): INHIBITOR 84.7, ANTAGONIST 88.2, AGONIST 77.6, |
| INDIRECT-DOWNREGULATOR 74.7, INDIRECT-UPREGULATOR 71.9, ACTIVATOR 69.5, SUBSTRATE 69.4, PART-OF 69.0, |
| AGONIST-INHIBITOR 66.7, DIRECT-REGULATOR 60.0, PRODUCT-OF 59.7. The two rarest types, AGONIST-ACTIVATOR and |
| SUBSTRATE_PRODUCT-OF, were never predicted (F1 0.0). |
| |
| Notes on these numbers: |
| - One training run (one seed); no confidence intervals. |
| - Only same-sentence relations are modelled. About 2% of gold dev relations (78 of 3,761) cannot be produced by |
| this design (cross-sentence pairs, or entities wider than 24 sub-words or with boundaries inside a word). They |
| count as misses. |
| - The end-to-end score is a strict internal match of predicted entity spans and labels; the official scorer |
| requires gold entity ids and so covers only the gold-entity setting. |
| |
| ## Files |
| |
| | File | Contents | |
| |---|---| |
| | `model.safetensors` | All weights: encoder, NER head and relation head | |
| | `model_config.json` | Encoder configuration, head settings, label order, relation-marker token ids, training settings | |
| | `tokenizer.json`, `tokenizer_config.json` | The cased SciBERT tokenizer used in training (`do_lower_case=False`) | |
|
|
| ## Usage |
|
|
| The checkpoint is read by the `cancer_twins.scientific_ai` research code that trained it, which is not distributed |
| here. With that package installed: |
|
|
| ```python |
| from huggingface_hub import snapshot_download |
| from transformers import AutoTokenizer |
| from cancer_twins.scientific_ai.model import load_model |
| |
| checkpoint = snapshot_download("long-unicam/drugprot-scibert-ner-rel") |
| model, config = load_model(checkpoint) # weights + model_config.json |
| tokenizer = AutoTokenizer.from_pretrained(checkpoint, use_fast=True) # the bundled SciBERT tokenizer |
| ``` |
|
|
| Inference then uses `cancer_twins.scientific_ai.data.build_examples` (sentence and span preparation) and |
| `cancer_twins.scientific_ai.predict.predict_examples` (NER, then relations over the predicted or given entities). |
| Requirements: Python 3.12, PyTorch 2.x, `transformers` 5.x, `safetensors`. Label order in `model_config.json` must |
| match the code; `load_model` checks it. |
|
|
| ## Limitations and intended use |
|
|
| - **Research use only.** Outputs are text-mined statements of what a sentence says, not verified biological |
| facts, and must not be used for clinical or treatment decisions. |
| - Trained on PubMed abstracts in English; performance on full texts, other genres or other entity definitions is |
| unknown and likely lower. |
| - Rare relation types are learned poorly or not at all (see above), and DIRECT-REGULATOR and PRODUCT-OF are weak. |
| - Negation, speculation and hypotheses in the text are not modelled separately; a relation that a sentence denies |
| may still be predicted. |
| - DrugProt annotations reflect curators' judgements under the DrugProt guidelines; "no relation" means "not |
| annotated", not a confirmed absence of interaction. |
|
|
