dekmATtud/mhd-xlmr-base

MHD XLM-RoBERTa base

FacebookAI/xlm-roberta-base adapted to medieval and early modern German (Middle High German, Early New High German, Middle Low German) by continued masked-language-model pretraining (domain-adaptive pretraining, Gururangan et al. 2020). Use it as an encoder for token-level tasks such as named entity recognition or part-of-speech tagging, or as a starting point for sentence embeddings.

Usage

from transformers import pipeline

fill = pipeline("fill-mask", model="dekmATtud/mhd-xlmr-base")
fill("Diz iſt ein <mask> rede")

As word embeddings in Flair:

from flair.embeddings import TransformerWordEmbeddings

emb = TransformerWordEmbeddings("dekmATtud/mhd-xlmr-base", layers="all", layer_mean=True, subtoken_pooling="mean")

Model

Architecture of FacebookAI/xlm-roberta-base, 278M parameters. The tokenizer adds 62 medieval characters to the original vocabulary. It normalises NFKC, so the model sees ſ as s.

Training

Masked language modelling, 15% of the tokens masked, sequences of 512 tokens, 101,020 training sequences. AdamW, learning rate 0.0001, beta2 0.98, weight decay 0.01, warmup 0.06 with linear decay, effective batch size 256, bf16, 20 epochs, seed 42. The published weights are the checkpoint with the lowest validation loss.

Validation with a fixed mask:

step	epoch	train loss (recent)	valid MLM loss
197	0.50	4.2021	3.1282
1379	3.49	2.1477	1.8223
2364	5.98	1.9072	1.6522
3546	8.98	1.7641	1.5392
4531	11.47	1.6859	1.4870
5713	14.46	1.6259	1.4420
6698	16.96	1.5882	1.4170
7880	19.95	1.5667	1.4029

On the held-out test texts (1,233 sequences of 512 tokens, 15 % masked with a fixed seed): MLM loss 1.5359, pseudo-perplexity 4.65.

Training logs in this repository: training_history.json. Task evaluations will follow with the accompanying paper.

Training corpora

13,478 texts, 154.4M characters, mostly diplomatic transcriptions with a focus before 1500. Text is kept as transcribed (Unicode NFC only), with all special characters. About 1 % of the texts of each corpus is held out for validation and 1 % for test. The training texts are not distributed with this model.

• Referenzkorpus Mittelhochdeutsch (ReM), CC BY-SA 4.0
• Referenzkorpus Frühneuhochdeutsch (ReF.MLU), CC BY-SA 4.0
• Referenzkorpus Frühneuhochdeutsch (ReF.RUB), CC BY-SA 4.0
• Referenzkorpus Frühneuhochdeutsch (ReF.UP), CC BY-SA 4.0
• Referenzkorpus Mittelniederdeutsch/Niederrheinisch (ReN), annotated texts, CC BY 4.0
• Referenzkorpus Mittelniederdeutsch/Niederrheinisch (ReN), transcriptions, CC BY 4.0
• Mittelhochdeutsche Begriffsdatenbank (MHDBDB), CC BY-NC-SA 4.0
• Source texts of the Mittelhochdeutsches Wörterbuch (MWB)
• Corpus der altdeutschen Originalurkunden (CAO), CC BY 4.0
• Königsfelden edition (Zürich / Staatsarchiv Aargau), CC BY 4.0
• Sammlung Schweizerischer Rechtsquellen (SSRQ), German pieces, CC BY-NC-SA 4.0
• Nuremberg letter books, vols. 2-5, CC BY-NC 4.0 / CC BY 4.0
• Nuremberg letter books, vols. 6-14, uncorrected HTR, CC BY 4.0
• Lüne letters (HAB Wolfenbüttel), CC BY-SA 3.0 DE
• Deutsches Textarchiv, texts up to 1500, mostly CC BY-NC 3.0
• HTR/OCR transcriptions by Stefan Tomasek

In this build every SSRQ text was read twice, so these texts carry double weight in training.

Context

Collaboration between Prof. Dr. Michael Schonhardt (Technische Universität Darmstadt / AdWL Mainz) and Dr. Stefan Tomasek (Universität Würzburg). The models were trained at the Fachgebiet Digitale Editorik und Kulturgeschichte des Mittelalters, TU Darmstadt, as part of the LOEWE Exploration project "Embedding the Past", funded by the HMWK Hessen. Training was conducted by Michael Schonhardt.

Licence

CC BY-NC-SA 4.0, as the NonCommercial and ShareAlike terms of several corpora apply.

Citation

@misc{mhd_xlmr_base_2026,
  author       = {Schonhardt, Michael and Tomasek, Stefan},
  title        = {MHD XLM-RoBERTa base},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/dekmATtud/mhd-xlmr-base}},
  note         = {LOEWE Exploration Embedding the Past, TU Darmstadt. Training: Michael Schonhardt}
}