tmy100000001/LitDD_BERT

LitDD BERT

The abstract classifier of LitDD, a pipeline that maps PubMed literature to developmental-disorder entries in Gene2Phenotype (DDG2P). It is described in PubMed-wide Classification and Disease-level Mapping of Peer-reviewed Literature for Genetic Developmental Disorders in Gene2Phenotype (Yates et al., under review). Code: https://github.com/biomedicalinformaticsgroup/LitDD_mining (archived at https://doi.org/10.5281/zenodo.17779641).

Given a title and abstract (TIAB), LitDD BERT answers one question:

Does this record describe human case evidence for a genetic developmental disorder?

It is the first stage of the pipeline. It screens the 31.9 million eligible PubMed records (English, published after 1980) and passes the positives to a gene gate, which restricts each abstract to the G2P entries of the DDG2P genes it names, and then to an LLM (openai/gpt-oss-20b) that chooses among those entries.

Versions

The released checkpoint is revision v2.5, which has the same weights as branch main. The checkpoint described in the originally submitted manuscript is kept at tag v1-seed42; it is not interchangeable with this one (it classifies far more random PubMed records as positive) and should be used only to reproduce that version.

Requires transformers >= 4.48

This is a ModernBERT architecture. Earlier versions fail with KeyError: 'modernbert'.

Usage

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

model_id, revision = "tmy100000001/LitDD_BERT", "v2.5"
tok = AutoTokenizer.from_pretrained(model_id, revision=revision)
model = AutoModelForSequenceClassification.from_pretrained(model_id, revision=revision).eval()

tiab = ("A Novel Homozygous Nonsense HYDIN Gene Mutation p.(Arg951*) in Primary Ciliary "
        "Dyskinesia. We report a consanguineous family in which two affected siblings ...")

enc = tok(tiab, truncation=True, max_length=8192, return_tensors="pt")
with torch.no_grad():
    pred = model(**enc).logits.argmax(-1).item()
print(model.config.id2label[pred])   # RELEVANT / NOT_RELEVANT

Label mapping: 0 = NOT_RELEVANT, 1 = RELEVANT. The text is the title and abstract joined by one space. For corpus-scale inference the repository's litdd.pipeline.bert_predict_vllm serves the model with vLLM in fp32.

Intended use

Use it for: first-pass screening of PubMed titles and abstracts for human case evidence of a genetic developmental disorder, at corpus scale.

Do not use it for:

• deciding which disease a paper describes: that is the LLM step's job, and the classifier is not disease-specific;
• clinical or diagnostic decision-making;
• a precision-sensitive task on its own. At PubMed prevalence the classifier's positives include many records that the gene gate and LLM step remove.

Training

Base model	thomas-sounack/BioClinical-ModernBERT-large
Task	binary sequence classification
Training set	37,335 (TIAB, label) rows, 24.0% positive: the training portion of the clinician-annotated set; curated positives (DDG2P, HPO annotations, ClinGen) on genes of the training fold of a gene-grouped split, including 232 reviewed papers that report variants in molecular rather than case-report terms; and 20,000 PubMed abstracts drawn at random, stratified by decade, as negatives, excluding all evaluation records
Hyperparameters	learning rate 3e-5, weight decay 0.1, 5 epochs, batch 32, fp32, seed 44
Selection	5-fold StratifiedGroupKFold cross-validation within the training set; refit on the full training set; test set evaluated once

Evaluation

All figures are from the paper's Supplementary Information.

Held-out annotated test set (2,739 records): precision 0.892, recall 0.937, F1 0.914. Across seeds 42, 43 and 44 with the same recipe: F1 0.920 (SD 0.004).

Random PubMed records classified positive: 0.51% (95% CI 0.38–0.69%). Without the 20,000 corpus negatives the same recipe reaches the same test F1 (0.916) but classifies 22.6% of random records as positive, so test F1 alone does not show how the classifier behaves across PubMed.

Stricter held-out splits (means of three seeds):

Split	Precision	Recall	F1
Title and abstract (standard)	0.897	0.944	0.920
Gene held out	0.822	0.964	0.887
Disease (G2P entry) held out	0.834	0.930	0.879
Time split (train ≤2019, test ≥2020)	0.908	0.961	0.934

Recall on independently curated papers not used in training or testing (DDG2P, HPO annotations, ClinGen): 0.763 [0.742, 0.782] on all 1,752 papers and 0.915 [0.895, 0.931] on the 972 that name a DDG2P gene in the title or abstract. Most missed papers name no gene, so the pipeline could not map them to a disease in any case.

Base encoders fine-tuned identically (means of three seeds; test set as above):

Encoder	Precision	Recall	F1
BiomedBERT	0.908	0.940	0.924
BioBERT	0.904	0.944	0.923
BioClinical-ModernBERT-large	0.893	0.940	0.916
ModernBERT-large	0.882	0.934	0.907
LitDD BERT (this model)	0.892	0.937	0.914

Comparison with EvAgg (gpt-4-turbo paper classifier) on identical inputs:

	LitDD BERT	EvAgg
Test precision	0.892	0.534
Test recall	0.937	0.995
Random PubMed classified positive	0.51%	3.13%

Full pipeline: on the test set, with an exact match to the curated entries required, precision 0.865, recall 0.806, F1 0.835. A blinded audit of the released map gave precision 0.876 (95% CI 0.841–0.911).

Limitations

• English records published after 1980 only; the classifier was not trained or evaluated outside that range.
• Not disease-specific: it cannot say which G2P entry a paper supports.
• Papers that name no gene in the title or abstract are the main miss mode.
• Trained in fp32; a bf16 build is not interchangeable.
• Trained on developmental-disorder genetics; behaviour on other disease areas is untested.

Citation

Manuscript under review; citation to follow. Please also cite the base model, BioClinical ModernBERT.