DOKTERBERT

DOKTERBERT (Dutch Ontology-aligned Knowledge-injected Text Encoder for Representations using BERT) is a Dutch clinical language model trained across the full breadth of Dutch clinical text: GP consultation notes, medical literature, clinical and pharmacological guidelines, and patient-facing health information, on top of MedRoBERTa.nl's hospital-note foundation. Pretraining uses a SNOMED CT-grounded contrastive objective that aligns contextual span representations to SNOMED concept anchors, organising clinical concept representations against the ontology rather than treating terms in isolation.

Where MedRoBERTa.nl was trained on hospital notes alone, DOKTERBERT extends coverage to the wider Dutch clinical language ecosystem. Contextual span representations are aligned to SNOMED CT concept anchors, with contrastive pressure between negatives weighted by graph distance in the SNOMED IS-A hierarchy, concentrating discriminative pressure on semantically adjacent concepts. The result is a representation space whose geometry reflects clinical concept structure rather than only linguistic co-occurrence.

• Developed by: Gijs Danoe, Matthijs S. Berends, Andreas Voss, Axel Hamprecht
• Model type: RoBERTa-based clinical text encoder (continued pretraining, ~125M parameters)
• Language: Dutch (nl)
• License: MIT
• Base model: CLTL/MedRoBERTa.nl
• Repository: https://github.com/gijsdanoe/DOKTERBERT
• Paper: Danoe et al. (2026), DOKTERBERT at #SMM4H–HeaRD 2026: Ontology-Grounded Contextual Representations for Dutch Clinical NLP

Intended Use

DOKTERBERT is designed for tasks that operate directly on the representation space, without task-specific fine-tuning: similarity-based retrieval, clustering, and anomaly detection over clinical text. It is most useful where labelled data is scarce and downstream systems depend on embedding geometry. Span representations should be obtained by mean-pooling the final-layer hidden states over the tokens of a span in its sentence context, as used in training.

It can also be fine-tuned for supervised tasks such as NER, where it performs comparably to its baselines; its advantage is in representation quality rather than fine-tuned task performance.

DOKTERBERT is a research artifact, not a medical device, and must not be used to inform clinical decisions about individual patients.

Usage

from transformers import AutoTokenizer, AutoModel
import torch

tokenizer = AutoTokenizer.from_pretrained("gijsdanoe/DOKTERBERT")
model = AutoModel.from_pretrained("gijsdanoe/DOKTERBERT")

text = "Patiënt presenteert met koorts en hoesten."
enc = tokenizer(text, return_tensors="pt")
with torch.no_grad():
    out = model(**enc)

emb = out.last_hidden_state.mean(dim=1)  # mean-pooled embedding

Training

Data: 2.34 GB of Dutch clinical text spanning the full spectrum of Dutch clinical language: GP consultation notes, medical journal articles, clinical and pharmacological guidelines, and patient-facing health information. Combined with the hospital notes underlying MedRoBERTa.nl, this makes DOKTERBERT the Dutch clinical encoder with the broadest register coverage, spanning specialist hospital documentation, primary care, and the guideline and patient-information ecosystem.

Preprocessing: Candidate medical spans are extracted with spaCy dependency parsing, then linked to SNOMED CT concepts by exact string match against the Dutch SNOMED term set, with a SapBERT similarity fallback (cosine threshold 0.85). This yields 11.4M linked spans covering 30,408 unique SNOMED concepts.

Procedure:

• Initialized from MedRoBERTa.nl; 1 epoch of continued pretraining
• Objective: masked language modelling + distance-weighted InfoNCE contrastive loss
• Contrastive weight α = 0.2, temperature τ = 0.07, graph-distance decay σ = 15
• Optimizer AdamW, learning rate 2e-5, 1,000 warmup steps, weight decay 0.01

Evaluation

Evaluated on MultiClinNER-nl, the Dutch subtask of the MultiClinAI shared task at the SMM4H/HeaRD workshop.

On supervised NER, DOKTERBERT, RobBERT, MedRoBERTa.nl, and MedRoBERTa.nl-SapBERT all fall within a narrow band (macro F1 ≈ 0.69–0.70). On the unsupervised representational analysis, DOKTERBERT separates clearly from every baseline, with a concept discrimination gap of +0.592 versus +0.170 for the next-best model, and leads on retrieval, clustering, and intra/inter-concept separation.

Standard fine-tuning evaluation obscures pretraining-level differences in representation quality that representation analysis exposes; DOKTERBERT's ontology grounding produces a measurably more clinically structured embedding space.

Citation

@inproceedings{danoe2026dokterbert,
  title     = {DOKTERBERT at \#SMM4H--HeaRD 2026: Ontology-Grounded Contextual Representations for Dutch Clinical NLP},
  author    = {Danoe, Gijs and Berends, Matthijs S. and Voss, Andreas and Hamprecht, Axel},
  booktitle = {Proceedings of the SMM4H/HeaRD Workshop},
  year      = {2026}
}

Contact

g.danoe@umcg.nl