re-cep's picture
feat(relations): upgrade to ATLOP context pooling, between-span features, and domain-prior masking
c0fb351 verified
|
Raw History Blame Contribute Delete
6.14 kB
---
license: apache-2.0
language:
- en
base_model: microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext
datasets:
- bigbio/biored
tags:
- biomedical
- relation-extraction
- biored
- pubtator
- vitiligo
metrics:
- f1
inference: false
---
# Vitiligo Relation Extractor
A biomedical relation extraction model built for the
[Vitiligo Initiative](https://github.com/recepsirin/vitiligo-initiative), an
open, non-profit research platform. Given a passage whose chemicals,
diseases, genes and variants are already annotated, it classifies the
relation between every pair of them into BioRED's eight types. It reads the
text PubTator 3.0 does not: full-text paper bodies and abstracts outside
PubMed.
| Benchmark | Pair F1 | Pair + type F1 |
|---|---|---|
| BioRED test (100 abstracts, expert-annotated) | **0.730** | **0.602** |
| Agreement with PubTator 3.0 (591 held-out vitiligo abstracts) | 0.748 | 0.645 |
## Model details
- **Architecture:** BiomedBERT encoder with ATLOP localized context pooling
(joint pair self-attention), closest-mention between-span pooling, token-
distance / co-sentence embeddings, and entity-type domain masking. A passage
is encoded once and every entity pair in it is scored from that single pass.
- **Base model:** [BiomedBERT](https://huggingface.co/microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext)
(base, uncased, PubMed abstracts and full text)
- **Labels:** Association, Positive_Correlation, Negative_Correlation, Bind,
Cotreatment, Comparison, Drug_Interaction, Conversion, or none
- **Entity types:** Chemical, Disease, Gene, Variant
- **Parameters:** 113M · **Language:** English · **License:** Apache-2.0
- **Code:** [`vitiligo.relations`](https://github.com/recepsirin/vitiligo-initiative/tree/master/src/vitiligo/relations)
## Uses
**Intended:** reading relations out of biomedical literature to build a
knowledge graph, where each relation keeps the sentence it came from so it
can be checked. In the Vitiligo Initiative, its relations become graph edges
with lower confidence than curated or registry sources.
**Out of scope:** clinical decisions, or any use where an unchecked relation
is taken as fact. It does not find entities itself; they must be annotated
first, for example by PubTator 3.0.
## How to use
The model runs through the project's own code, not a transformers class.
```bash
pip install "vitiligo[train] @ git+https://github.com/recepsirin/vitiligo-initiative"
```
```python
from vitiligo.relations.train import load
model, bias, settings = load("re-cep/vitiligo-relation-extractor")
```
Append `@<revision>` to the id to pin one version. To reproduce the BioRED
test scores:
```bash
vitiligo relations evaluate --model re-cep/vitiligo-relation-extractor --split Test
```
## Training
1. **Silver:** PubTator 3.0's relations in 11,106 abstracts of the Vitiligo
Initiative corpus, excluding every BioRED abstract and a fixed 1-in-20
held-out set.
2. **Gold:** BioRED's 400 training abstracts, starting from the silver model.
The epoch (5 of 10) and the decision threshold (`bias = 0.4`) were chosen on
BioRED's development split (F1 0.603). Per-epoch development scores are in
`relations.json`.
## Evaluation
BioRED test split, relations between given entities, micro-averaged as in
the BioRED paper. Pair + type precision is 0.624 and recall 0.580 (pair-only
precision is 0.758 and recall is 0.704).
| Training | Dev F1 | Test pair F1 | Test pair + type F1 |
|---|---|---|---|
| DeBERTa-v3-base on BioRED | 0.475 | 0.645 | 0.451 |
| BiomedBERT on BioRED | 0.499 | 0.642 | 0.459 |
| BiomedBERT on PubTator silver | 0.510 | 0.635 | 0.537 |
| BiomedBERT (no context pooling), silver then BioRED | 0.554 | 0.687 | 0.540 |
| **BiomedBERT + ATLOP + Span, silver then BioRED (this model)** | **0.603** | **0.730** | **0.602** |
The BioRED paper's PubMedBERT reaches 0.729 and 0.589 by encoding each pair
separately; this model surpasses both (0.730 and 0.602) while encoding each
passage only once, which is what makes reading 821,413 candidate pairs in full
texts practical.
**In use on full texts.** Of 120 relations read from full-text bodies and
checked by hand against their sentences, 68% were correct. The Vitiligo
Initiative therefore loads only relations with probability ≥ 0.8, outside
tables and figures, excluding co-administration claims, and accepting
"treats" only when the disease is vitiligo. On a fresh hand-checked sample
under that rule, 45 of 60 relations (75%) were correct, and 15 of 17 where
vitiligo is one of the pair.
## Limitations
- About a quarter of the relations it loads in practice are wrong; each
should be read with the sentence it came from.
- Table rows, figure captions and lists of drugs are frequent sources of
error.
- The silver training data come from one disease's literature; accuracy on
other domains is untested beyond BioRED.
## Files
| File | Contents |
|---|---|
| `model.safetensors` | All weights: encoder, entity-type embedding, pair classifier |
| `config.json`, `tokenizer.json`, `tokenizer_config.json` | BiomedBERT's configuration and tokenizer, unchanged |
| `relations.json` | Label order, decision bias, special-token ids, training history |
## License and attribution
Released under Apache-2.0. Fine-tuned from BiomedBERT (© Microsoft, MIT
License). Trained on BioRED, distributed by NCBI as freely available, and on
PubTator 3.0 annotations, a work of the US Government.
## Citation
- Luo L, Lai PT, Wei CH, Arighi CN, Lu Z. BioRED: a rich biomedical relation
extraction dataset. *Briefings in Bioinformatics* 2022;23(5):bbac282.
- Gu Y, Tinn R, Cheng H, et al. Domain-specific language model pretraining for
biomedical natural language processing. *ACM Transactions on Computing for
Healthcare* 2021;3(1).
- Wei CH, Allot A, Lai PT, et al. PubTator 3.0: an AI-powered literature
resource for unlocking biomedical knowledge. *Nucleic Acids Research*
2024;52(W1):W540-W546.
- Zhou W, Huang K, Ma T, Huang J. Document-level relation extraction with
adaptive thresholding and localized context pooling. *AAAI* 2021.