---
language:
- en
base_model: allenai/scibert_scivocab_cased
pipeline_tag: token-classification
tags:
- biomedical
- relation-extraction
- named-entity-recognition
- drugprot
- scibert
---

# DrugProt SciBERT: joint chemical/gene NER and chemical → gene relation extraction

A SciBERT model fine-tuned on the DrugProt corpus (BioCreative VII Track 1). It finds **chemical** and
**gene/protein** mentions in biomedical text and classifies the relation a sentence states from a chemical to a
gene/protein into the 13 DrugProt relation types.

This is a research checkpoint. It is **not** a Hugging Face `transformers` model class and cannot be loaded with
`AutoModel` or `pipeline`.

## Architecture

- **Encoder:** `allenai/scibert_scivocab_cased` (BERT-base, cased vocabulary), fine-tuned.
- **NER head:** span classification. Every candidate span of up to 24 sub-words is represented as
  [first; mean; last] encoder vectors and classified as `None`, `CHEMICAL` or `GENE` (softmax).
- **Relation head (typed entity markers):** for each same-sentence (chemical, gene) pair, the sentence is encoded
  again with typed text markers around the two mentions (`@ * chemical * … @`, `# ^ gene ^ … #`). The concatenated
  hidden states of the chemical and gene start markers are classified over the 13 relation types with a per-label
  sigmoid (multi-label) and a decision threshold of 0.5.
- **Training:** joint NER + relation loss (L = L_NER + 1.0 · L_REL), AdamW with linear warm-up/decay
  (encoder lr 3e-5, head lr 1e-3), batch size 16, max sequence length 256, 8 epochs with early stopping, seed 42.

## Data

- **Corpus:** DrugProt v1.2 (Miranda-Escalada, Krallinger et al., BioCreative VII), PubMed abstracts annotated by
  experts, [doi:10.5281/zenodo.5119892](https://doi.org/10.5281/zenodo.5119892).
- **Training:** the official training set (3,500 abstracts) minus 350 abstracts held out for checkpoint selection.
- **Checkpoint selection:** on those 350 held-out training abstracts (epoch 7 of 8).
- **Evaluation:** the official **development** set (750 abstracts), used for neither training nor selection. The
  official test-set gold labels were not used.

## Evaluation (official DrugProt development set, 750 abstracts)

| Task | Precision | Recall | F1 |
|---|---|---|---|
| NER, all entities (strict span + type) | 88.95 | 90.52 | **89.73** |
| NER, `CHEMICAL` | 88.86 | 91.92 | 90.36 |
| NER, `GENE` | 89.05 | 88.98 | 89.02 |
| Relations with gold entities (official BioCreative VII scorer, micro) | 77.4 | 71.9 | **74.5** |
| Relations end-to-end (predicted entities, strict) | 66.29 | 62.43 | **64.30** |

Relation F1 by type (gold entities, official scorer): INHIBITOR 84.7, ANTAGONIST 88.2, AGONIST 77.6,
INDIRECT-DOWNREGULATOR 74.7, INDIRECT-UPREGULATOR 71.9, ACTIVATOR 69.5, SUBSTRATE 69.4, PART-OF 69.0,
AGONIST-INHIBITOR 66.7, DIRECT-REGULATOR 60.0, PRODUCT-OF 59.7. The two rarest types, AGONIST-ACTIVATOR and
SUBSTRATE_PRODUCT-OF, were never predicted (F1 0.0).

Notes on these numbers:
- One training run (one seed); no confidence intervals.
- Only same-sentence relations are modelled. About 2% of gold dev relations (78 of 3,761) cannot be produced by
  this design (cross-sentence pairs, or entities wider than 24 sub-words or with boundaries inside a word). They
  count as misses.
- The end-to-end score is a strict internal match of predicted entity spans and labels; the official scorer
  requires gold entity ids and so covers only the gold-entity setting.

## Files

| File | Contents |
|---|---|
| `model.safetensors` | All weights: encoder, NER head and relation head |
| `model_config.json` | Encoder configuration, head settings, label order, relation-marker token ids, training settings |
| `tokenizer.json`, `tokenizer_config.json` | The cased SciBERT tokenizer used in training (`do_lower_case=False`) |

## Usage

The checkpoint is read by the `cancer_twins.scientific_ai` research code that trained it, which is not distributed
here. With that package installed:

```python
from huggingface_hub import snapshot_download
from transformers import AutoTokenizer
from cancer_twins.scientific_ai.model import load_model

checkpoint = snapshot_download("long-unicam/drugprot-scibert-ner-rel")
model, config = load_model(checkpoint)                                   # weights + model_config.json
tokenizer = AutoTokenizer.from_pretrained(checkpoint, use_fast=True)     # the bundled SciBERT tokenizer
```

Inference then uses `cancer_twins.scientific_ai.data.build_examples` (sentence and span preparation) and
`cancer_twins.scientific_ai.predict.predict_examples` (NER, then relations over the predicted or given entities).
Requirements: Python 3.12, PyTorch 2.x, `transformers` 5.x, `safetensors`. Label order in `model_config.json` must
match the code; `load_model` checks it.

## Limitations and intended use

- **Research use only.** Outputs are text-mined statements of what a sentence says, not verified biological
  facts, and must not be used for clinical or treatment decisions.
- Trained on PubMed abstracts in English; performance on full texts, other genres or other entity definitions is
  unknown and likely lower.
- Rare relation types are learned poorly or not at all (see above), and DIRECT-REGULATOR and PRODUCT-OF are weak.
- Negation, speculation and hypotheses in the text are not modelled separately; a relation that a sentence denies
  may still be predicted.
- DrugProt annotations reflect curators' judgements under the DrugProt guidelines; "no relation" means "not
  annotated", not a confirmed absence of interaction.