DrugProt SciBERT Stage B (entity markers, joint NER/REL) checkpoint and model card

2673314 verified 6 days ago

5.59 kB

language:
  - en
base_model: allenai/scibert_scivocab_cased
pipeline_tag: token-classification
tags:
  - biomedical
  - relation-extraction
  - named-entity-recognition
  - drugprot
  - scibert

DrugProt SciBERT: joint chemical/gene NER and chemical → gene relation extraction

A SciBERT model fine-tuned on the DrugProt corpus (BioCreative VII Track 1). It finds chemical and gene/protein mentions in biomedical text and classifies the relation a sentence states from a chemical to a gene/protein into the 13 DrugProt relation types.

This is a research checkpoint. It is not a Hugging Face transformers model class and cannot be loaded with AutoModel or pipeline.

Architecture

Encoder: allenai/scibert_scivocab_cased (BERT-base, cased vocabulary), fine-tuned.
NER head: span classification. Every candidate span of up to 24 sub-words is represented as [first; mean; last] encoder vectors and classified as None, CHEMICAL or GENE (softmax).
Relation head (typed entity markers): for each same-sentence (chemical, gene) pair, the sentence is encoded again with typed text markers around the two mentions (@ * chemical * … @, # ^ gene ^ … #). The concatenated hidden states of the chemical and gene start markers are classified over the 13 relation types with a per-label sigmoid (multi-label) and a decision threshold of 0.5.
Training: joint NER + relation loss (L = L_NER + 1.0 · L_REL), AdamW with linear warm-up/decay (encoder lr 3e-5, head lr 1e-3), batch size 16, max sequence length 256, 8 epochs with early stopping, seed 42.

Data

Corpus: DrugProt v1.2 (Miranda-Escalada, Krallinger et al., BioCreative VII), PubMed abstracts annotated by experts, doi:10.5281/zenodo.5119892.
Training: the official training set (3,500 abstracts) minus 350 abstracts held out for checkpoint selection.
Checkpoint selection: on those 350 held-out training abstracts (epoch 7 of 8).
Evaluation: the official development set (750 abstracts), used for neither training nor selection. The official test-set gold labels were not used.

Evaluation (official DrugProt development set, 750 abstracts)

Task	Precision	Recall	F1
NER, all entities (strict span + type)	88.95	90.52	89.73
NER, `CHEMICAL`	88.86	91.92	90.36
NER, `GENE`	89.05	88.98	89.02
Relations with gold entities (official BioCreative VII scorer, micro)	77.4	71.9	74.5
Relations end-to-end (predicted entities, strict)	66.29	62.43	64.30

Relation F1 by type (gold entities, official scorer): INHIBITOR 84.7, ANTAGONIST 88.2, AGONIST 77.6, INDIRECT-DOWNREGULATOR 74.7, INDIRECT-UPREGULATOR 71.9, ACTIVATOR 69.5, SUBSTRATE 69.4, PART-OF 69.0, AGONIST-INHIBITOR 66.7, DIRECT-REGULATOR 60.0, PRODUCT-OF 59.7. The two rarest types, AGONIST-ACTIVATOR and SUBSTRATE_PRODUCT-OF, were never predicted (F1 0.0).

Notes on these numbers:

One training run (one seed); no confidence intervals.
Only same-sentence relations are modelled. About 2% of gold dev relations (78 of 3,761) cannot be produced by this design (cross-sentence pairs, or entities wider than 24 sub-words or with boundaries inside a word). They count as misses.
The end-to-end score is a strict internal match of predicted entity spans and labels; the official scorer requires gold entity ids and so covers only the gold-entity setting.

Files

File	Contents
`model.safetensors`	All weights: encoder, NER head and relation head
`model_config.json`	Encoder configuration, head settings, label order, relation-marker token ids, training settings
`tokenizer.json`, `tokenizer_config.json`	The cased SciBERT tokenizer used in training (`do_lower_case=False`)

Usage

The checkpoint is read by the cancer_twins.scientific_ai research code that trained it, which is not distributed here. With that package installed:

from huggingface_hub import snapshot_download
from transformers import AutoTokenizer
from cancer_twins.scientific_ai.model import load_model

checkpoint = snapshot_download("long-unicam/drugprot-scibert-ner-rel")
model, config = load_model(checkpoint)                                   # weights + model_config.json
tokenizer = AutoTokenizer.from_pretrained(checkpoint, use_fast=True)     # the bundled SciBERT tokenizer

Inference then uses cancer_twins.scientific_ai.data.build_examples (sentence and span preparation) and cancer_twins.scientific_ai.predict.predict_examples (NER, then relations over the predicted or given entities). Requirements: Python 3.12, PyTorch 2.x, transformers 5.x, safetensors. Label order in model_config.json must match the code; load_model checks it.

Limitations and intended use

Research use only. Outputs are text-mined statements of what a sentence says, not verified biological facts, and must not be used for clinical or treatment decisions.
Trained on PubMed abstracts in English; performance on full texts, other genres or other entity definitions is unknown and likely lower.
Rare relation types are learned poorly or not at all (see above), and DIRECT-REGULATOR and PRODUCT-OF are weak.
Negation, speculation and hypotheses in the text are not modelled separately; a relation that a sentence denies may still be predicted.
DrugProt annotations reflect curators' judgements under the DrugProt guidelines; "no relation" means "not annotated", not a confirmed absence of interaction.