PubMedBERT/BiomedBERT fine-tuned for CHIA clinical trial eligibility NER

Fine-tuned microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext for named entity recognition on clinical trial eligibility criteria (CHIA corpus), as part of a course NLP project comparing fine-tuned biomedical transformers vs. GPT-4 prompting.

  • Code / full writeup: https://github.com/jatinpsingh/NER_Clinical_Trial_Eligibility/tree/main/pmb
  • Base model: PubMedBERT-base (renamed BiomedBERT), Gu et al. 2021
  • Task: token classification, 31 labels (O + 15 entity types x B/I)
  • Data: CHIA eligibility criteria, team's shared fixed split (10,006 train / 1,240 val / 1,163 test sentences -- fold 0 of a 10-fold CV scheme, see below)

Hyperparameters (matching Li et al. 2022, Table 3)

Learning rate 5e-5
Batch size 8
Epochs 10
Max sequence length 256
Adam epsilon 1e-8

Results

This checkpoint's own fold (entity-level, test set):

Precision Recall F1
Strict (exact span match) 0.639 0.674 0.656
Relaxed (type + overlap match) 0.750 0.791 0.770

Full 10-fold cross-validation (mean +/- std across all 10 folds -- the number directly comparable to Li et al. 2022's own 10-fold-averaged reporting; the weights hosted in this repo are from one of these 10 folds, not a checkpoint averaged across them):

Precision Recall F1
Strict 0.657 +/- 0.013 0.682 +/- 0.021 0.669 +/- 0.013
Relaxed 0.758 +/- 0.014 0.787 +/- 0.024 0.772 +/- 0.015

Both this checkpoint's own score and the 10-fold mean exceed Li et al. 2022's published PubMedBERT numbers on Chia (0.622 strict / 0.744 relaxed).

Usage

from transformers import AutoModelForTokenClassification, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("ptanwar/pubmedbert-chia-ner")
model = AutoModelForTokenClassification.from_pretrained("ptanwar/pubmedbert-chia-ner")

Known limitations

  • The hosted weights are one fold's model, not a checkpoint averaged/ensembled across the 10-fold CV run -- treat the single-fold numbers above as this specific checkpoint's performance, and the 10-fold mean +/- std as the more rigorous estimate of the approach's true performance.
  • Weakest on rare/ambiguous types (Mood, Observation, Reference_point) -- see the full writeup for error analysis.
Downloads last month
23
Safetensors
Model size
0.1B params
Tensor type
F32
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for ptanwar/pubmedbert-chia-ner