File size: 5,590 Bytes
2673314 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 | ---
language:
- en
base_model: allenai/scibert_scivocab_cased
pipeline_tag: token-classification
tags:
- biomedical
- relation-extraction
- named-entity-recognition
- drugprot
- scibert
---
# DrugProt SciBERT: joint chemical/gene NER and chemical → gene relation extraction
A SciBERT model fine-tuned on the DrugProt corpus (BioCreative VII Track 1). It finds **chemical** and
**gene/protein** mentions in biomedical text and classifies the relation a sentence states from a chemical to a
gene/protein into the 13 DrugProt relation types.
This is a research checkpoint. It is **not** a Hugging Face `transformers` model class and cannot be loaded with
`AutoModel` or `pipeline`.
## Architecture
- **Encoder:** `allenai/scibert_scivocab_cased` (BERT-base, cased vocabulary), fine-tuned.
- **NER head:** span classification. Every candidate span of up to 24 sub-words is represented as
[first; mean; last] encoder vectors and classified as `None`, `CHEMICAL` or `GENE` (softmax).
- **Relation head (typed entity markers):** for each same-sentence (chemical, gene) pair, the sentence is encoded
again with typed text markers around the two mentions (`@ * chemical * … @`, `# ^ gene ^ … #`). The concatenated
hidden states of the chemical and gene start markers are classified over the 13 relation types with a per-label
sigmoid (multi-label) and a decision threshold of 0.5.
- **Training:** joint NER + relation loss (L = L_NER + 1.0 · L_REL), AdamW with linear warm-up/decay
(encoder lr 3e-5, head lr 1e-3), batch size 16, max sequence length 256, 8 epochs with early stopping, seed 42.
## Data
- **Corpus:** DrugProt v1.2 (Miranda-Escalada, Krallinger et al., BioCreative VII), PubMed abstracts annotated by
experts, [doi:10.5281/zenodo.5119892](https://doi.org/10.5281/zenodo.5119892).
- **Training:** the official training set (3,500 abstracts) minus 350 abstracts held out for checkpoint selection.
- **Checkpoint selection:** on those 350 held-out training abstracts (epoch 7 of 8).
- **Evaluation:** the official **development** set (750 abstracts), used for neither training nor selection. The
official test-set gold labels were not used.
## Evaluation (official DrugProt development set, 750 abstracts)
| Task | Precision | Recall | F1 |
|---|---|---|---|
| NER, all entities (strict span + type) | 88.95 | 90.52 | **89.73** |
| NER, `CHEMICAL` | 88.86 | 91.92 | 90.36 |
| NER, `GENE` | 89.05 | 88.98 | 89.02 |
| Relations with gold entities (official BioCreative VII scorer, micro) | 77.4 | 71.9 | **74.5** |
| Relations end-to-end (predicted entities, strict) | 66.29 | 62.43 | **64.30** |
Relation F1 by type (gold entities, official scorer): INHIBITOR 84.7, ANTAGONIST 88.2, AGONIST 77.6,
INDIRECT-DOWNREGULATOR 74.7, INDIRECT-UPREGULATOR 71.9, ACTIVATOR 69.5, SUBSTRATE 69.4, PART-OF 69.0,
AGONIST-INHIBITOR 66.7, DIRECT-REGULATOR 60.0, PRODUCT-OF 59.7. The two rarest types, AGONIST-ACTIVATOR and
SUBSTRATE_PRODUCT-OF, were never predicted (F1 0.0).
Notes on these numbers:
- One training run (one seed); no confidence intervals.
- Only same-sentence relations are modelled. About 2% of gold dev relations (78 of 3,761) cannot be produced by
this design (cross-sentence pairs, or entities wider than 24 sub-words or with boundaries inside a word). They
count as misses.
- The end-to-end score is a strict internal match of predicted entity spans and labels; the official scorer
requires gold entity ids and so covers only the gold-entity setting.
## Files
| File | Contents |
|---|---|
| `model.safetensors` | All weights: encoder, NER head and relation head |
| `model_config.json` | Encoder configuration, head settings, label order, relation-marker token ids, training settings |
| `tokenizer.json`, `tokenizer_config.json` | The cased SciBERT tokenizer used in training (`do_lower_case=False`) |
## Usage
The checkpoint is read by the `cancer_twins.scientific_ai` research code that trained it, which is not distributed
here. With that package installed:
```python
from huggingface_hub import snapshot_download
from transformers import AutoTokenizer
from cancer_twins.scientific_ai.model import load_model
checkpoint = snapshot_download("long-unicam/drugprot-scibert-ner-rel")
model, config = load_model(checkpoint) # weights + model_config.json
tokenizer = AutoTokenizer.from_pretrained(checkpoint, use_fast=True) # the bundled SciBERT tokenizer
```
Inference then uses `cancer_twins.scientific_ai.data.build_examples` (sentence and span preparation) and
`cancer_twins.scientific_ai.predict.predict_examples` (NER, then relations over the predicted or given entities).
Requirements: Python 3.12, PyTorch 2.x, `transformers` 5.x, `safetensors`. Label order in `model_config.json` must
match the code; `load_model` checks it.
## Limitations and intended use
- **Research use only.** Outputs are text-mined statements of what a sentence says, not verified biological
facts, and must not be used for clinical or treatment decisions.
- Trained on PubMed abstracts in English; performance on full texts, other genres or other entity definitions is
unknown and likely lower.
- Rare relation types are learned poorly or not at all (see above), and DIRECT-REGULATOR and PRODUCT-OF are weak.
- Negation, speculation and hypotheses in the text are not modelled separately; a relation that a sentence denies
may still be predicted.
- DrugProt annotations reflect curators' judgements under the DrugProt guidelines; "no relation" means "not
annotated", not a confirmed absence of interaction.
|