model card: revised pipeline is screen -> gene gate -> LLM; cross-encoder off the critical path

9c99efb verified about 2 months ago

7.97 kB

---
license: mit
base_model: ncbi/MedCPT-Cross-Encoder
pipeline_tag: text-classification
library_name: sentence-transformers
tags:
- cross-encoder
- biomedical
- gene-disease
- Gene2Phenotype
- LitDD
language:
- en
---
# LitDD cross-encoder (v2, 2026-08-31)

Fine-tuned [`ncbi/MedCPT-Cross-Encoder`](https://huggingface.co/ncbi/MedCPT-Cross-Encoder)
that scores a PubMed **title + abstract** against a **Gene2Phenotype (G2P) LGMDE thread**
(one gene–disease entry rendered as a 15-field string) and returns a sigmoid relevance score
in [0, 1]. It was the candidate-ranking stage of the LitDD pipeline
([biomedicalinformaticsgroup/LitDD_mining](https://github.com/biomedicalinformaticsgroup/LitDD_mining)).

> ### ⚠️ Not required by the current pipeline (2026-09-02)
>
> The revised pipeline is **screen → gene gate → LLM adjudication**: each abstract is restricted
> to the Gene2Phenotype entries of the genes actually mentioned in its title+abstract (mean 2.4
> candidates, median 2), and **all** of those candidates are shown to the LLM with no score
> threshold. Measured on the held-out annotated test split, that gene gate offers every curated
> entry for 606/606 curated abstracts (retrieval recall 1.000), and removing the cross-encoder
> changes end-to-end performance by less than the noise (P 0.840 / R 0.848 without it vs
> P 0.851 / R 0.850 with it, n = 2,731 abstracts, McNemar p = 0.86).
>
> This model is retained because it is **required to reproduce the published pipeline**, and it
> remains useful as (i) an optional ranker when an abstract yields many candidates and a cap is
> wanted, and (ii) a confidence signal for auditing LLM answers (keeping only answers whose
> entries score ≥ 0.95 raises answer precision 0.927 → 0.936 while discarding 15% of answers).
> It is no longer part of the deployed critical path, and no score gate is applied.

The previous release is preserved at tag [`v1-original`](../../tree/v1-original). It is
superseded: on the current panel rendering it calls essentially every pair positive at 0.5
(test precision 0.25, recall 1.00) and fires on 2.8% of gene-gated random PubMed abstracts at
its 0.9 gate; this release fires on 0.22% with higher in-scope recall (table below).

## Usage

```python
from sentence_transformers import CrossEncoder
from litdd.threads import build_lgmde_map   # renders the thread exactly as in training
model = CrossEncoder("tmy100000001/LitDD_crossencoder")          # fp32 weights
threads = build_lgmde_map("G2P_DD_2026-06-24.csv")               # {g2p_id: thread}
score = model.predict([(title_plus_abstract, threads["G2P00003"])])[0]
passes_gate = score >= 0.9
```

Pair order is **(abstract, thread)** — a cross-encoder is not symmetric. Scoring in fp16 or
bf16 (`model_kwargs={"torch_dtype": torch.float16}`) reproduces the fp32 predictions exactly
on the held-out test set and the frozen external benchmark, so half precision is safe for
corpus-scale inference. Training was fp32 (fp16 training produced an unstable seed).

## Training data

* Base pairs (17,058): clinician-annotated (abstract, thread) pairs — positives and
  gene-sharing negatives — plus curated external positives (premined DDG2P / HPOA / ClinGen,
  gene-fold train split) and a molecular-register augmentation set (gene-level positives kept
  only for single-entry genes). Threads re-rendered from **G2P DD export 2026-06-24**.
* Corpus negatives (4,137 pairs): the gene-gate-passing subset (1,565) of the same 20,000
  random PubMed abstracts used to train the LitDD screen, each paired with the G2P entries of
  the genes it mentions, labelled 0 (silver). This mirrors the screen's "add20k" recipe: the
  in-domain test set is blind to corpus over-prediction, so corpus fire rate is a release gate.
* Hard negatives: `abhinand/MedEmbed-large-v0.1`, 5 per positive, rank range 5–50,
  `max_score` 0.95, `relative_margin` 0.01, mined from **positives only**; the annotated
  negatives enter as labelled negatives. Final labelled-pair set: 62,223 rows (8,712 positive).

## Protocol

* **Leakage control**: train/test split grouped by abstract (TIAB) and verified disjoint at
  PMID level; hard negatives mined inside each CV fold's training half; the held-out test set
  (2,771 pairs, 686 positive) is the same split the screen uses and was scored once.
* **Hyperparameters**: 5-fold `StratifiedGroupKFold` on the training set only, grid
  lr {1e-5, 2e-5, 3e-5} × epochs {1, 2}, batch 16, warmup 0.1 → **lr 3e-5, 1 epoch**
  (grid range of mean fold F1 was 0.004; the choice is non-critical).
* **Thread representation**: three arms (flat / + full gene name / contextualised MONDO+HPO)
  did not separate under CV (Friedman p = 0.074), so the flat deployment rendering ships.
* **Seeds**: three (42/43/44); this checkpoint is **seed 42** by a pre-registered fixed-seed
  rule, and sits inside the seed spread on every axis below.

## Results (seed 42 unless a ± is shown, which is mean ± sd over 3 seeds)

Held-out annotated test set (2,771 pairs; fixed 0.5 threshold):

| model | F1 | precision | recall |
|---|---|---|---|
| **this release** | **0.931 ± 0.002** (seed 42: 0.933) | 0.921 | 0.941 |
| previous release (`v1-original`) | 0.397 | 0.248 | 1.000 |
| untrained MedCPT | 0.417 | 0.266 | 0.971 |

Frozen external benchmark (1,924 leak-free curated truth pairs; 87,600 random PubMed
abstracts as silver negatives), at the deployment gate 0.9:

| model | in-scope recall (663 pairs) | gate-passing recall (1,592) | all (1,924) | corpus fire rate |
|---|---|---|---|---|
| **this release** | **0.932 ± 0.009** | 0.576 ± 0.028 | 0.539 ± 0.027 | **0.227 ± 0.013 %** |
| same recipe without corpus negatives | 0.941 ± 0.009 | 0.632 ± 0.012 | 0.604 ± 0.010 | 0.294 ± 0.012 % |
| previous release | 0.919 | 0.570 | 0.545 | 2.77 % |
| untrained MedCPT | 0.971 | 0.749 | 0.702 | 2.37 % |

"In-scope" = curated pairs whose abstract names the gene; the remainder are reviews,
pre-molecular papers and gene-free abstracts that the pipeline is designed to reject, so the
"all" denominator understates recall by construction. Corpus fire rate = share of the 87,600
abstracts with at least one gate candidate scoring ≥ 0.9.

Why the gate is 0.9 (`revision/crossencoder_gate_analysis_fine.csv`, 0.01-step scan): F1 is
maximised at the lowest threshold on both the test set and the frozen benchmark, but this
stage is a precision filter ahead of the LLM. The precision-weighted benchmark F0.5 rises
from 0.772 (0.5) to 0.782 (0.9) and peaks at 0.789 at 0.95 (on all three seeds); every
threshold from 0.89 to 0.98 is within one seed-sd of that maximum. 0.9 is retained as the
recall-favouring edge of the plateau (0.9 -> 0.95 would trade 1.9 points of in-scope recall
and 6 points of test recall for +1.6 points of benchmark precision and 13% fewer false
fires). Test-set P/R at 0.9: 0.938 / 0.900 (3-seed means).

Top-k coverage of the true entry (held-out test positives, fp32): against the **full 2,861-entry panel** top-1 = 0.818, top-3 = 0.991, top-5 = 1.00 (previous release 0.818 / 0.991 / 1.00; untrained MedCPT 0.713 / 0.924 / 0.953; the submitted manuscript's Table 2 reported 0.76 / 0.97 / 0.99 as 5-fold CV means); against the 279-entry test-set pool top-1 = 0.870, top-3 = 1.00; within each abstract's own gene-gate candidates (the deployment universe, median 2 candidates) top-1 = 0.797 ± 0.013. Full table: the repository's `revision/crossencoder_topk_coverage.csv`.

## Provenance

Trained 2026-08-31 in the LitDD pipeline image
(`ghcr.io/biomedicalinformaticsgroup/litdd_mining:sha-b75f6dfe…`, sentence-transformers 5.7.0,
transformers 5.12.0, torch 2.11 cu130) on one H100. Scripts:
`litdd/training/{build_crossencoder_dataset,build_corpus_negative_pairs,mine_hard_negatives,cv_hp_search_crossencoder,crossencode_finetune}.py`;
evaluation: `litdd/evaluation/{crossencoder_frozen_eval,crossencoder_threshold_sweep,run_cross_encoder_benchmark,compare_models}.py`.

	---
	license: mit
	base_model: ncbi/MedCPT-Cross-Encoder
	pipeline_tag: text-classification
	library_name: sentence-transformers
	tags:
	- cross-encoder
	- biomedical
	- gene-disease
	- Gene2Phenotype
	- LitDD
	language:
	- en
	---

	# LitDD cross-encoder (v2, 2026-08-31)

	Fine-tuned [`ncbi/MedCPT-Cross-Encoder`](https://huggingface.co/ncbi/MedCPT-Cross-Encoder)
	that scores a PubMed title + abstract against a Gene2Phenotype (G2P) LGMDE thread
	(one gene–disease entry rendered as a 15-field string) and returns a sigmoid relevance score
	in [0, 1]. It was the candidate-ranking stage of the LitDD pipeline
	([biomedicalinformaticsgroup/LitDD_mining](https://github.com/biomedicalinformaticsgroup/LitDD_mining)).

	> ### ⚠️ Not required by the current pipeline (2026-09-02)
	>
	> The revised pipeline is screen → gene gate → LLM adjudication: each abstract is restricted
	> to the Gene2Phenotype entries of the genes actually mentioned in its title+abstract (mean 2.4
	> candidates, median 2), and all of those candidates are shown to the LLM with no score
	> threshold. Measured on the held-out annotated test split, that gene gate offers every curated
	> entry for 606/606 curated abstracts (retrieval recall 1.000), and removing the cross-encoder
	> changes end-to-end performance by less than the noise (P 0.840 / R 0.848 without it vs
	> P 0.851 / R 0.850 with it, n = 2,731 abstracts, McNemar p = 0.86).
	>
	> This model is retained because it is required to reproduce the published pipeline, and it
	> remains useful as (i) an optional ranker when an abstract yields many candidates and a cap is
	> wanted, and (ii) a confidence signal for auditing LLM answers (keeping only answers whose
	> entries score ≥ 0.95 raises answer precision 0.927 → 0.936 while discarding 15% of answers).
	> It is no longer part of the deployed critical path, and no score gate is applied.

	The previous release is preserved at tag [`v1-original`](../../tree/v1-original). It is
	superseded: on the current panel rendering it calls essentially every pair positive at 0.5
	(test precision 0.25, recall 1.00) and fires on 2.8% of gene-gated random PubMed abstracts at
	its 0.9 gate; this release fires on 0.22% with higher in-scope recall (table below).

	## Usage

	```python
	from sentence_transformers import CrossEncoder
	from litdd.threads import build_lgmde_map # renders the thread exactly as in training

	model = CrossEncoder("tmy100000001/LitDD_crossencoder") # fp32 weights
	threads = build_lgmde_map("G2P_DD_2026-06-24.csv") # {g2p_id: thread}
	score = model.predict([(title_plus_abstract, threads["G2P00003"])])[0]
	passes_gate = score >= 0.9
	```

	Pair order is (abstract, thread) — a cross-encoder is not symmetric. Scoring in fp16 or
	bf16 (`model_kwargs={"torch_dtype": torch.float16}`) reproduces the fp32 predictions exactly
	on the held-out test set and the frozen external benchmark, so half precision is safe for
	corpus-scale inference. Training was fp32 (fp16 training produced an unstable seed).

	## Training data

	* Base pairs (17,058): clinician-annotated (abstract, thread) pairs — positives and
	gene-sharing negatives — plus curated external positives (premined DDG2P / HPOA / ClinGen,
	gene-fold train split) and a molecular-register augmentation set (gene-level positives kept
	only for single-entry genes). Threads re-rendered from G2P DD export 2026-06-24.
	* Corpus negatives (4,137 pairs): the gene-gate-passing subset (1,565) of the same 20,000
	random PubMed abstracts used to train the LitDD screen, each paired with the G2P entries of
	the genes it mentions, labelled 0 (silver). This mirrors the screen's "add20k" recipe: the
	in-domain test set is blind to corpus over-prediction, so corpus fire rate is a release gate.
	* Hard negatives: `abhinand/MedEmbed-large-v0.1`, 5 per positive, rank range 5–50,
	`max_score` 0.95, `relative_margin` 0.01, mined from positives only; the annotated
	negatives enter as labelled negatives. Final labelled-pair set: 62,223 rows (8,712 positive).

	## Protocol

	* Leakage control: train/test split grouped by abstract (TIAB) and verified disjoint at
	PMID level; hard negatives mined inside each CV fold's training half; the held-out test set
	(2,771 pairs, 686 positive) is the same split the screen uses and was scored once.
	* Hyperparameters: 5-fold `StratifiedGroupKFold` on the training set only, grid
	lr {1e-5, 2e-5, 3e-5} × epochs {1, 2}, batch 16, warmup 0.1 → lr 3e-5, 1 epoch
	(grid range of mean fold F1 was 0.004; the choice is non-critical).
	* Thread representation: three arms (flat / + full gene name / contextualised MONDO+HPO)
	did not separate under CV (Friedman p = 0.074), so the flat deployment rendering ships.
	* Seeds: three (42/43/44); this checkpoint is seed 42 by a pre-registered fixed-seed
	rule, and sits inside the seed spread on every axis below.

	## Results (seed 42 unless a ± is shown, which is mean ± sd over 3 seeds)

	Held-out annotated test set (2,771 pairs; fixed 0.5 threshold):

	\| model \| F1 \| precision \| recall \|
	\|---\|---\|---\|---\|
	\| this release \| 0.931 ± 0.002 (seed 42: 0.933) \| 0.921 \| 0.941 \|
	\| previous release (`v1-original`) \| 0.397 \| 0.248 \| 1.000 \|
	\| untrained MedCPT \| 0.417 \| 0.266 \| 0.971 \|

	Frozen external benchmark (1,924 leak-free curated truth pairs; 87,600 random PubMed
	abstracts as silver negatives), at the deployment gate 0.9:

	\| model \| in-scope recall (663 pairs) \| gate-passing recall (1,592) \| all (1,924) \| corpus fire rate \|
	\|---\|---\|---\|---\|---\|
	\| this release \| 0.932 ± 0.009 \| 0.576 ± 0.028 \| 0.539 ± 0.027 \| 0.227 ± 0.013 % \|
	\| same recipe without corpus negatives \| 0.941 ± 0.009 \| 0.632 ± 0.012 \| 0.604 ± 0.010 \| 0.294 ± 0.012 % \|
	\| previous release \| 0.919 \| 0.570 \| 0.545 \| 2.77 % \|
	\| untrained MedCPT \| 0.971 \| 0.749 \| 0.702 \| 2.37 % \|

	"In-scope" = curated pairs whose abstract names the gene; the remainder are reviews,
	pre-molecular papers and gene-free abstracts that the pipeline is designed to reject, so the
	"all" denominator understates recall by construction. Corpus fire rate = share of the 87,600
	abstracts with at least one gate candidate scoring ≥ 0.9.

	Why the gate is 0.9 (`revision/crossencoder_gate_analysis_fine.csv`, 0.01-step scan): F1 is
	maximised at the lowest threshold on both the test set and the frozen benchmark, but this
	stage is a precision filter ahead of the LLM. The precision-weighted benchmark F0.5 rises
	from 0.772 (0.5) to 0.782 (0.9) and peaks at 0.789 at 0.95 (on all three seeds); every
	threshold from 0.89 to 0.98 is within one seed-sd of that maximum. 0.9 is retained as the
	recall-favouring edge of the plateau (0.9 -> 0.95 would trade 1.9 points of in-scope recall
	and 6 points of test recall for +1.6 points of benchmark precision and 13% fewer false
	fires). Test-set P/R at 0.9: 0.938 / 0.900 (3-seed means).

	Top-k coverage of the true entry (held-out test positives, fp32): against the full 2,861-entry panel top-1 = 0.818, top-3 = 0.991, top-5 = 1.00 (previous release 0.818 / 0.991 / 1.00; untrained MedCPT 0.713 / 0.924 / 0.953; the submitted manuscript's Table 2 reported 0.76 / 0.97 / 0.99 as 5-fold CV means); against the 279-entry test-set pool top-1 = 0.870, top-3 = 1.00; within each abstract's own gene-gate candidates (the deployment universe, median 2 candidates) top-1 = 0.797 ± 0.013. Full table: the repository's `revision/crossencoder_topk_coverage.csv`.

	## Provenance

	Trained 2026-08-31 in the LitDD pipeline image
	(`ghcr.io/biomedicalinformaticsgroup/litdd_mining:sha-b75f6dfe…`, sentence-transformers 5.7.0,
	transformers 5.12.0, torch 2.11 cu130) on one H100. Scripts:
	`litdd/training/{build_crossencoder_dataset,build_corpus_negative_pairs,mine_hard_negatives,cv_hp_search_crossencoder,crossencode_finetune}.py`;
	evaluation: `litdd/evaluation/{crossencoder_frozen_eval,crossencoder_threshold_sweep,run_cross_encoder_benchmark,compare_models}.py`.