|
Download README.md from re-cep/vitiligo-relation-extractor: direct link, hf CLI and curl.
- Browser
- Download file 6.14 kB
-
https://huggingface.co/re-cep/vitiligo-relation-extractor/resolve/main/README.md
- Command line
-
hf download hf://re-cep/vitiligo-relation-extractor/README.md
-
curl -L -o README.md https://huggingface.co/re-cep/vitiligo-relation-extractor/resolve/main/README.md
6.14 kB
| license: apache-2.0 | |
| language: | |
| - en | |
| base_model: microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext | |
| datasets: | |
| - bigbio/biored | |
| tags: | |
| - biomedical | |
| - relation-extraction | |
| - biored | |
| - pubtator | |
| - vitiligo | |
| metrics: | |
| - f1 | |
| inference: false | |
| # Vitiligo Relation Extractor | |
| A biomedical relation extraction model built for the | |
| [Vitiligo Initiative](https://github.com/recepsirin/vitiligo-initiative), an | |
| open, non-profit research platform. Given a passage whose chemicals, | |
| diseases, genes and variants are already annotated, it classifies the | |
| relation between every pair of them into BioRED's eight types. It reads the | |
| text PubTator 3.0 does not: full-text paper bodies and abstracts outside | |
| PubMed. | |
| | Benchmark | Pair F1 | Pair + type F1 | | |
| |---|---|---| | |
| | BioRED test (100 abstracts, expert-annotated) | **0.730** | **0.602** | | |
| | Agreement with PubTator 3.0 (591 held-out vitiligo abstracts) | 0.748 | 0.645 | | |
| ## Model details | |
| - **Architecture:** BiomedBERT encoder with ATLOP localized context pooling | |
| (joint pair self-attention), closest-mention between-span pooling, token- | |
| distance / co-sentence embeddings, and entity-type domain masking. A passage | |
| is encoded once and every entity pair in it is scored from that single pass. | |
| - **Base model:** [BiomedBERT](https://huggingface.co/microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext) | |
| (base, uncased, PubMed abstracts and full text) | |
| - **Labels:** Association, Positive_Correlation, Negative_Correlation, Bind, | |
| Cotreatment, Comparison, Drug_Interaction, Conversion, or none | |
| - **Entity types:** Chemical, Disease, Gene, Variant | |
| - **Parameters:** 113M · **Language:** English · **License:** Apache-2.0 | |
| - **Code:** [`vitiligo.relations`](https://github.com/recepsirin/vitiligo-initiative/tree/master/src/vitiligo/relations) | |
| ## Uses | |
| **Intended:** reading relations out of biomedical literature to build a | |
| knowledge graph, where each relation keeps the sentence it came from so it | |
| can be checked. In the Vitiligo Initiative, its relations become graph edges | |
| with lower confidence than curated or registry sources. | |
| **Out of scope:** clinical decisions, or any use where an unchecked relation | |
| is taken as fact. It does not find entities itself; they must be annotated | |
| first, for example by PubTator 3.0. | |
| ## How to use | |
| The model runs through the project's own code, not a transformers class. | |
| ```bash | |
| pip install "vitiligo[train] @ git+https://github.com/recepsirin/vitiligo-initiative" | |
| ``` | |
| ```python | |
| from vitiligo.relations.train import load | |
| model, bias, settings = load("re-cep/vitiligo-relation-extractor") | |
| ``` | |
| Append `@<revision>` to the id to pin one version. To reproduce the BioRED | |
| test scores: | |
| ```bash | |
| vitiligo relations evaluate --model re-cep/vitiligo-relation-extractor --split Test | |
| ``` | |
| ## Training | |
| 1. **Silver:** PubTator 3.0's relations in 11,106 abstracts of the Vitiligo | |
| Initiative corpus, excluding every BioRED abstract and a fixed 1-in-20 | |
| held-out set. | |
| 2. **Gold:** BioRED's 400 training abstracts, starting from the silver model. | |
| The epoch (5 of 10) and the decision threshold (`bias = 0.4`) were chosen on | |
| BioRED's development split (F1 0.603). Per-epoch development scores are in | |
| `relations.json`. | |
| ## Evaluation | |
| BioRED test split, relations between given entities, micro-averaged as in | |
| the BioRED paper. Pair + type precision is 0.624 and recall 0.580 (pair-only | |
| precision is 0.758 and recall is 0.704). | |
| | Training | Dev F1 | Test pair F1 | Test pair + type F1 | | |
| |---|---|---|---| | |
| | DeBERTa-v3-base on BioRED | 0.475 | 0.645 | 0.451 | | |
| | BiomedBERT on BioRED | 0.499 | 0.642 | 0.459 | | |
| | BiomedBERT on PubTator silver | 0.510 | 0.635 | 0.537 | | |
| | BiomedBERT (no context pooling), silver then BioRED | 0.554 | 0.687 | 0.540 | | |
| | **BiomedBERT + ATLOP + Span, silver then BioRED (this model)** | **0.603** | **0.730** | **0.602** | | |
| The BioRED paper's PubMedBERT reaches 0.729 and 0.589 by encoding each pair | |
| separately; this model surpasses both (0.730 and 0.602) while encoding each | |
| passage only once, which is what makes reading 821,413 candidate pairs in full | |
| texts practical. | |
| **In use on full texts.** Of 120 relations read from full-text bodies and | |
| checked by hand against their sentences, 68% were correct. The Vitiligo | |
| Initiative therefore loads only relations with probability ≥ 0.8, outside | |
| tables and figures, excluding co-administration claims, and accepting | |
| "treats" only when the disease is vitiligo. On a fresh hand-checked sample | |
| under that rule, 45 of 60 relations (75%) were correct, and 15 of 17 where | |
| vitiligo is one of the pair. | |
| ## Limitations | |
| - About a quarter of the relations it loads in practice are wrong; each | |
| should be read with the sentence it came from. | |
| - Table rows, figure captions and lists of drugs are frequent sources of | |
| error. | |
| - The silver training data come from one disease's literature; accuracy on | |
| other domains is untested beyond BioRED. | |
| ## Files | |
| | File | Contents | | |
| |---|---| | |
| | `model.safetensors` | All weights: encoder, entity-type embedding, pair classifier | | |
| | `config.json`, `tokenizer.json`, `tokenizer_config.json` | BiomedBERT's configuration and tokenizer, unchanged | | |
| | `relations.json` | Label order, decision bias, special-token ids, training history | | |
| ## License and attribution | |
| Released under Apache-2.0. Fine-tuned from BiomedBERT (© Microsoft, MIT | |
| License). Trained on BioRED, distributed by NCBI as freely available, and on | |
| PubTator 3.0 annotations, a work of the US Government. | |
| ## Citation | |
| - Luo L, Lai PT, Wei CH, Arighi CN, Lu Z. BioRED: a rich biomedical relation | |
| extraction dataset. *Briefings in Bioinformatics* 2022;23(5):bbac282. | |
| - Gu Y, Tinn R, Cheng H, et al. Domain-specific language model pretraining for | |
| biomedical natural language processing. *ACM Transactions on Computing for | |
| Healthcare* 2021;3(1). | |
| - Wei CH, Allot A, Lai PT, et al. PubTator 3.0: an AI-powered literature | |
| resource for unlocking biomedical knowledge. *Nucleic Acids Research* | |
| 2024;52(W1):W540-W546. | |
| - Zhou W, Huang K, Ma T, Huang J. Document-level relation extraction with | |
| adaptive thresholding and localized context pooling. *AAAI* 2021. | |