dekmATtud/mhd-xlmr-large

MHD XLM-RoBERTa large

FacebookAI/xlm-roberta-large adapted to medieval and early modern German (Middle High German, Early New High German, Middle Low German) by continued masked-language-model pretraining (domain-adaptive pretraining, Gururangan et al. 2020). Use it as an encoder for token-level tasks such as named entity recognition or part-of-speech tagging, or as a starting point for sentence embeddings.

Usage

from transformers import pipeline

fill = pipeline("fill-mask", model="dekmATtud/mhd-xlmr-large")
fill("Diz iſt ein <mask> rede")

As word embeddings in Flair:

from flair.embeddings import TransformerWordEmbeddings

emb = TransformerWordEmbeddings("dekmATtud/mhd-xlmr-large", layers="all", layer_mean=True, subtoken_pooling="mean")

Model

Architecture of FacebookAI/xlm-roberta-large, 560M parameters. The tokenizer adds 62 medieval characters to the original vocabulary. It normalises NFKC, so the model sees ſ as s.

Training

Masked language modelling, 15% of the tokens masked, sequences of 512 tokens, 101,020 training sequences. AdamW, learning rate 5e-05, beta2 0.98, weight decay 0.01, warmup 0.06 with linear decay, effective batch size 256, bf16, 10 epochs, seed 42. The published weights are the checkpoint with the lowest validation loss.

Validation with a fixed mask:

step	epoch	train loss (recent)	valid MLM loss
197	0.50	3.7693	2.6280
788	2.00	2.1582	1.7559
1182	2.99	1.9328	1.6153
1773	4.49	1.7605	1.4976
2364	5.99	1.6613	1.4227
2955	7.48	1.5997	1.3857
3349	8.48	1.5762	1.3621
3940	9.98	1.5503	1.3501

On the held-out test texts (1,233 sequences of 512 tokens, 15 % masked with a fixed seed): MLM loss 1.4779, pseudo-perplexity 4.38.

Training logs in this repository: training_history.json. Task evaluations will follow with the accompanying paper.

Training corpora

13,478 texts, 154.4M characters, mostly diplomatic transcriptions with a focus before 1500. Text is kept as transcribed (Unicode NFC only), with all special characters. About 1 % of the texts of each corpus is held out for validation and 1 % for test. The training texts are not distributed with this model.

• Referenzkorpus Mittelhochdeutsch (ReM), CC BY-SA 4.0
• Referenzkorpus Frühneuhochdeutsch (ReF.MLU), CC BY-SA 4.0
• Referenzkorpus Frühneuhochdeutsch (ReF.RUB), CC BY-SA 4.0
• Referenzkorpus Frühneuhochdeutsch (ReF.UP), CC BY-SA 4.0
• Referenzkorpus Mittelniederdeutsch/Niederrheinisch (ReN), annotated texts, CC BY 4.0
• Referenzkorpus Mittelniederdeutsch/Niederrheinisch (ReN), transcriptions, CC BY 4.0
• Mittelhochdeutsche Begriffsdatenbank (MHDBDB), CC BY-NC-SA 4.0
• Source texts of the Mittelhochdeutsches Wörterbuch (MWB)
• Corpus der altdeutschen Originalurkunden (CAO), CC BY 4.0
• Königsfelden edition (Zürich / Staatsarchiv Aargau), CC BY 4.0
• Sammlung Schweizerischer Rechtsquellen (SSRQ), German pieces, CC BY-NC-SA 4.0
• Nuremberg letter books, vols. 2-5, CC BY-NC 4.0 / CC BY 4.0
• Nuremberg letter books, vols. 6-14, uncorrected HTR, CC BY 4.0
• Lüne letters (HAB Wolfenbüttel), CC BY-SA 3.0 DE
• Deutsches Textarchiv, texts up to 1500, mostly CC BY-NC 3.0
• HTR/OCR transcriptions by Stefan Tomasek

In this build every SSRQ text was read twice, so these texts carry double weight in training.

Context

Collaboration between Prof. Dr. Michael Schonhardt (Technische Universität Darmstadt / AdWL Mainz) and Dr. Stefan Tomasek (Universität Würzburg). The models were trained at the Fachgebiet Digitale Editorik und Kulturgeschichte des Mittelalters, TU Darmstadt, as part of the LOEWE Exploration project "Embedding the Past", funded by the HMWK Hessen. Training was conducted by Michael Schonhardt.

Licence

CC BY-NC-SA 4.0, as the NonCommercial and ShareAlike terms of several corpora apply.

Citation

@misc{mhd_xlmr_large_2026,
  author       = {Schonhardt, Michael and Tomasek, Stefan},
  title        = {MHD XLM-RoBERTa large},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/dekmATtud/mhd-xlmr-large}},
  note         = {LOEWE Exploration Embedding the Past, TU Darmstadt. Training: Michael Schonhardt}
}