How to use from the
Use from the
Transformers library
# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-classification", model="xincan/pubmedbert-semrep-accuracy-classifier")
# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForSequenceClassification

tokenizer = AutoTokenizer.from_pretrained("xincan/pubmedbert-semrep-accuracy-classifier")
model = AutoModelForSequenceClassification.from_pretrained("xincan/pubmedbert-semrep-accuracy-classifier", device_map="auto")
Quick Links

PubMedBERT SemRep accuracy classifier

This model is a reproduction of the PubMedBERT accuracy classifier described in Zhang R, Hristovski D, Schutte D, Kastrin A, Fiszman M, Kilicoglu H. Drug repurposing for COVID-19 via knowledge graph completion. Journal of Biomedical Informatics. 2021;115:103696. doi:10.1016/j.jbi.2021.103696; preprint arXiv:2010.09600 (methods in Section 3.2.1; results in Section 4.1 and Table 1). The model was trained on the annotated predications used in the paper, with the same pretrained model and the authors' training code. Averaged over 10 independently trained models, it achieved a test F1 score of 0.859, compared with 0.854 reported in the paper.

Model description

The model determines whether a semantic predication (subject, predicate, object) extracted by SemRep is correct, that is, whether it is stated in its source sentence. It is intended for removing incorrect predications from SemMedDB before a knowledge graph is constructed.

Usage

Each input is a single text consisting of the source sentence, the [SEP] token, and the predication written as subject predicate object. As in training, the text should be converted to lowercase, and every character other than a letter or digit (including the underscore) should be replaced with a space.

import re
from transformers import pipeline

clf = pipeline("text-classification", model="xincan/pubmedbert-semrep-accuracy-classifier")

def norm(s):
    return " ".join(re.sub(r"[\W_]+", " ", s).split()).lower()

def predication_input(sentence, subject, predicate, obj):
    return f"{norm(sentence)} [SEP] {norm(f'{subject} {predicate} {obj}')}"

print(clf(predication_input(
    "Pulmonary hypertension accompanying ventricular septal defect and patent ductus arteriosus.",
    "Pulmonary hypertension", "COEXISTS_WITH", "ventricular septal defect")))
# [{'label': 'CORRECT', 'score': 0.978}]
print(clf(predication_input(
    "Control of brain tryptophan concentration in rats on a high fat diet.",
    "tryptophan", "TREATS", "rats")))
# [{'label': 'INCORRECT', 'score': 0.914}]  (the sentence does not state that tryptophan treats rats)

Multiple predications can be classified at once by passing a list, for example clf(texts, batch_size=64, truncation=True).

Training data

The training data comprise 7,038 SemRep predications from PubMed abstracts, each annotated as correct or incorrect with respect to its source sentence; 67.0% are correct. They cover 20 predicate types, the most frequent being INTERACTS_WITH (1,898), PROCESS_OF (775), STIMULATES (654), TREATS (649), INHIBITS (638), USES (454), COEXISTS_WITH (449), ADMINISTERED_TO (213), DIAGNOSES (191), and PRODUCES (173). The data combine the 6,492 annotations described in the paper (6,000 annotations from Vasilakes et al., JAMIA Open, 2018, and 492 newly annotated predications) and 546 additional annotations. The publicly available portion (6,492 predications) is provided in the kilicogluh/lbd-covid repository.

Training procedure

Training followed the authors' notebook (COVID_CLF.ipynb).

  • Data split: A test set of 10% was sampled with the original label distribution. The remaining data were balanced 1:1 by randomly removing correct predications and then divided into training (90%) and validation (10%) sets, yielding 3,762 training, 416 validation, and 703 test predications.
  • Optimization: AdamW with a learning rate of 1e-5, weight decay of 0.01, and epsilon of 1e-6; a linear learning rate schedule with 150 warm-up steps; a batch size of 16; and cross-entropy loss. Training ran for at most 10 epochs, retaining the weights with the highest validation F1 score and stopping early after 3 epochs without improvement. The maximum sequence length was 512 tokens; the longest input contained 190 tokens.
  • Model selection: Ten models were trained with the random seeds used in the authors' notebook. The released model is the one with the highest validation F1 score (seed 1386, epoch 4, validation F1 = 0.823).
  • Hardware: A single NVIDIA A40 GPU.

Evaluation

Ten models were trained with the paper's method, each with a different random seed and a different random split of the data. Each model was evaluated on its own held-out test set of 703 predications (67% correct).

Test set Precision Recall F1
Zhang et al. (2021), Table 1 0.816 0.895 0.854
Mean of the 10 models (range) 0.850 (0.798โ€“0.871) 0.869 (0.815โ€“0.904) 0.859 (0.839โ€“0.875)
Released model 0.871 0.815 0.842

The mean performance of the 10 models is consistent with that reported in the paper. The released model is one of the 10 models; following the authors' code, it was selected by its F1 score on the validation set (0.823), a separate set of 416 predications used only for model selection.

Limitations

  • The model was trained on SemRep predications from PubMed abstracts and may not generalize to other text sources, relation extraction systems, or predicate types.
  • The model assesses whether a predication is stated in its source sentence, not whether the statement is factually true.

Citation

@article{zhang2021drug,
  title   = {Drug repurposing for {COVID-19} via knowledge graph completion},
  author  = {Zhang, Rui and Hristovski, Dimitar and Schutte, Dalton and Kastrin, Andrej and Fiszman, Marcelo and Kilicoglu, Halil},
  journal = {Journal of Biomedical Informatics},
  volume  = {115},
  pages   = {103696},
  year    = {2021},
  doi     = {10.1016/j.jbi.2021.103696}
}
Downloads last month
66
Safetensors
Model size
0.1B params
Tensor type
F32
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for xincan/pubmedbert-semrep-accuracy-classifier

Paper for xincan/pubmedbert-semrep-accuracy-classifier

Evaluation results

  • Test precision on SemRep predication annotations (held-out test set, 703 predications)
    test set self-reported
    0.871
  • Test recall on SemRep predication annotations (held-out test set, 703 predications)
    test set self-reported
    0.815
  • Test F1 on SemRep predication annotations (held-out test set, 703 predications)
    test set self-reported
    0.842