Text Classification
Transformers
Safetensors
English
bert
biomedical
pubmedbert
semrep
semmeddb
relation-extraction
knowledge-graph
Eval Results (legacy)
text-embeddings-inference
Instructions to use xincan/pubmedbert-semrep-accuracy-classifier with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use xincan/pubmedbert-semrep-accuracy-classifier with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-classification", model="xincan/pubmedbert-semrep-accuracy-classifier")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForSequenceClassification tokenizer = AutoTokenizer.from_pretrained("xincan/pubmedbert-semrep-accuracy-classifier") model = AutoModelForSequenceClassification.from_pretrained("xincan/pubmedbert-semrep-accuracy-classifier", device_map="auto") - Notebooks
- Google Colab
- Kaggle
|
Download README.md from xincan/pubmedbert-semrep-accuracy-classifier: direct link, hf CLI and curl.
- Browser
- Download file 7.61 kB
-
https://huggingface.co/xincan/pubmedbert-semrep-accuracy-classifier/resolve/main/README.md
- Command line
-
hf download hf://xincan/pubmedbert-semrep-accuracy-classifier/README.md
-
curl -L -o README.md https://huggingface.co/xincan/pubmedbert-semrep-accuracy-classifier/resolve/main/README.md
7.61 kB
| language: | |
| - en | |
| license: mit | |
| library_name: transformers | |
| pipeline_tag: text-classification | |
| base_model: microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext | |
| tags: | |
| - biomedical | |
| - bert | |
| - pubmedbert | |
| - semrep | |
| - semmeddb | |
| - relation-extraction | |
| - knowledge-graph | |
| metrics: | |
| - precision | |
| - recall | |
| - f1 | |
| widget: | |
| - text: "pulmonary hypertension accompanying ventricular septal defect and patent ductus arteriosus [SEP] pulmonary hypertension coexists with ventricular septal defect" | |
| example_title: "Correct predication" | |
| - text: "control of brain tryptophan concentration in rats on a high fat diet [SEP] tryptophan treats rats" | |
| example_title: "Incorrect predication" | |
| model-index: | |
| - name: pubmedbert-semrep-accuracy-classifier | |
| results: | |
| - task: | |
| type: text-classification | |
| name: SemRep predication accuracy | |
| dataset: | |
| type: semrep-predication-annotations | |
| name: SemRep predication annotations (held-out test set, 703 predications) | |
| split: test | |
| metrics: | |
| - type: precision | |
| value: 0.871 | |
| name: Test precision | |
| - type: recall | |
| value: 0.815 | |
| name: Test recall | |
| - type: f1 | |
| value: 0.842 | |
| name: Test F1 | |
| # PubMedBERT SemRep accuracy classifier | |
| > **This model is a reproduction of the PubMedBERT accuracy classifier described in** Zhang R, Hristovski D, Schutte D, | |
| > Kastrin A, Fiszman M, Kilicoglu H. Drug repurposing for COVID-19 via knowledge graph completion. *Journal of Biomedical | |
| > Informatics*. 2021;115:103696. [doi:10.1016/j.jbi.2021.103696](https://doi.org/10.1016/j.jbi.2021.103696); | |
| > preprint [arXiv:2010.09600](https://arxiv.org/abs/2010.09600) (methods in Section 3.2.1; results in Section 4.1 and | |
| > Table 1). | |
| > The model was trained on the annotated predications used in the paper, with the same pretrained model and the authors' | |
| > training code. Averaged over 10 independently trained models, it achieved a test F1 score of 0.859, compared with | |
| > 0.854 reported in the paper. | |
| ## Model description | |
| The model determines whether a semantic predication (subject, predicate, object) extracted by | |
| [SemRep](https://lhncbc.nlm.nih.gov/ii/tools/SemRep_SemMedDB_SKR.html) is correct, that is, whether it is stated in its | |
| source sentence. It is intended for removing incorrect predications from | |
| [SemMedDB](https://lhncbc.nlm.nih.gov/ii/tools/SemRep_SemMedDB_SKR/SemMedDB_download.html) before a knowledge graph is | |
| constructed. | |
| - **Architecture:** BERT-base with a sequence classification head (`BertForSequenceClassification`) | |
| - **Pretrained model:** [BiomedBERT (formerly PubMedBERT), uncased, pretrained on PubMed abstracts and full-text articles](https://huggingface.co/microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext) | |
| - **Labels:** `CORRECT` (1) and `INCORRECT` (0) | |
| - **Language:** English | |
| - **License:** MIT, consistent with the pretrained model | |
| ## Usage | |
| Each input is a single text consisting of the source sentence, the `[SEP]` token, and the predication written as | |
| `subject predicate object`. As in training, the text should be converted to lowercase, and every character other than | |
| a letter or digit (including the underscore) should be replaced with a space. | |
| ```python | |
| import re | |
| from transformers import pipeline | |
| clf = pipeline("text-classification", model="xincan/pubmedbert-semrep-accuracy-classifier") | |
| def norm(s): | |
| return " ".join(re.sub(r"[\W_]+", " ", s).split()).lower() | |
| def predication_input(sentence, subject, predicate, obj): | |
| return f"{norm(sentence)} [SEP] {norm(f'{subject} {predicate} {obj}')}" | |
| print(clf(predication_input( | |
| "Pulmonary hypertension accompanying ventricular septal defect and patent ductus arteriosus.", | |
| "Pulmonary hypertension", "COEXISTS_WITH", "ventricular septal defect"))) | |
| # [{'label': 'CORRECT', 'score': 0.978}] | |
| print(clf(predication_input( | |
| "Control of brain tryptophan concentration in rats on a high fat diet.", | |
| "tryptophan", "TREATS", "rats"))) | |
| # [{'label': 'INCORRECT', 'score': 0.914}] (the sentence does not state that tryptophan treats rats) | |
| ``` | |
| Multiple predications can be classified at once by passing a list, for example | |
| `clf(texts, batch_size=64, truncation=True)`. | |
| ## Training data | |
| The training data comprise 7,038 SemRep predications from PubMed abstracts, each annotated as correct or incorrect | |
| with respect to its source sentence; 67.0% are correct. They cover 20 predicate types, the most frequent being | |
| INTERACTS_WITH (1,898), PROCESS_OF (775), STIMULATES (654), TREATS (649), INHIBITS (638), USES (454), COEXISTS_WITH | |
| (449), ADMINISTERED_TO (213), DIAGNOSES (191), and PRODUCES (173). The data combine the 6,492 annotations described in | |
| the paper (6,000 annotations from Vasilakes et al., *JAMIA Open*, 2018, and 492 newly annotated predications) and 546 | |
| additional annotations. The publicly available portion (6,492 predications) is provided in the | |
| [kilicogluh/lbd-covid](https://github.com/kilicogluh/lbd-covid/tree/master/filtering) repository. | |
| ## Training procedure | |
| Training followed the authors' notebook | |
| ([`COVID_CLF.ipynb`](https://github.com/kilicogluh/lbd-covid/blob/master/filtering/COVID_CLF.ipynb)). | |
| - **Data split:** A test set of 10% was sampled with the original label distribution. The remaining data were balanced | |
| 1:1 by randomly removing correct predications and then divided into training (90%) and validation (10%) sets, | |
| yielding 3,762 training, 416 validation, and 703 test predications. | |
| - **Optimization:** AdamW with a learning rate of 1e-5, weight decay of 0.01, and epsilon of 1e-6; a linear learning | |
| rate schedule with 150 warm-up steps; a batch size of 16; and cross-entropy loss. Training ran for at most 10 epochs, | |
| retaining the weights with the highest validation F1 score and stopping early after 3 epochs without improvement. | |
| The maximum sequence length was 512 tokens; the longest input contained 190 tokens. | |
| - **Model selection:** Ten models were trained with the random seeds used in the authors' notebook. The released model | |
| is the one with the highest validation F1 score (seed 1386, epoch 4, validation F1 = 0.823). | |
| - **Hardware:** A single NVIDIA A40 GPU. | |
| ## Evaluation | |
| Ten models were trained with the paper's method, each with a different random seed and a different random split of the | |
| data. Each model was evaluated on its own held-out test set of 703 predications (67% correct). | |
| | Test set | Precision | Recall | F1 | | |
| |---|---|---|---| | |
| | Zhang et al. (2021), Table 1 | 0.816 | 0.895 | 0.854 | | |
| | Mean of the 10 models (range) | 0.850 (0.798–0.871) | 0.869 (0.815–0.904) | **0.859** (0.839–0.875) | | |
| | Released model | 0.871 | 0.815 | 0.842 | | |
| The mean performance of the 10 models is consistent with that reported in the paper. The released model is one of the | |
| 10 models; following the authors' code, it was selected by its F1 score on the validation set (0.823), a separate set | |
| of 416 predications used only for model selection. | |
| ## Limitations | |
| - The model was trained on SemRep predications from PubMed abstracts and may not generalize to other text sources, | |
| relation extraction systems, or predicate types. | |
| - The model assesses whether a predication is stated in its source sentence, not whether the statement is factually | |
| true. | |
| ## Citation | |
| ```bibtex | |
| @article{zhang2021drug, | |
| title = {Drug repurposing for {COVID-19} via knowledge graph completion}, | |
| author = {Zhang, Rui and Hristovski, Dimitar and Schutte, Dalton and Kastrin, Andrej and Fiszman, Marcelo and Kilicoglu, Halil}, | |
| journal = {Journal of Biomedical Informatics}, | |
| volume = {115}, | |
| pages = {103696}, | |
| year = {2021}, | |
| doi = {10.1016/j.jbi.2021.103696} | |
| } | |
| ``` | |