dekmATtud/mhd-flair-raw-forward

MHD Flair character language model (raw, forward)

A forward character language model for medieval and early modern German (Middle High German, Early New High German, Middle Low German). Use it as contextual string embeddings (Akbik et al. 2018), together with dekmATtud/mhd-flair-raw-backward.

Usage

from huggingface_hub import hf_hub_download
from flair.data import Sentence
from flair.embeddings import FlairEmbeddings, StackedEmbeddings

fw = FlairEmbeddings(hf_hub_download("dekmATtud/mhd-flair-raw-forward", "best-lm.pt"))
bw = FlairEmbeddings(hf_hub_download("dekmATtud/mhd-flair-raw-backward", "best-lm.pt"))
emb = StackedEmbeddings([fw, bw])
s = Sentence("Diz iſt ein ſchoͤne rede")
emb.embed(s)          # s[i].embedding: 2 x 2048 dimensions

Model

LSTM over characters: embedding 100, 1 layer of 2048 units, softmax over 310 characters.

Training

Flair 0.15.1 LanguageModelTrainer. SGD, learning rate 20.0, annealed by 0.25 after 31 training splits (5M characters each) without improvement, gradient clipping 0.25, dropout 0.1, sequences of 250 characters, batch size 100, mixed precision, 30 epochs, seed 42. best-lm.pt is the state with the lowest validation loss.

Validation in bits per character:

epoch	valid bpc (end of epoch)	best valid bpc in epoch	learning rate
1	2.0283	2.0283	20
5	1.7543	1.7539	20
9	1.7157	1.7157	20
13	1.6843	1.6843	5
17	1.6839	1.6825	0.3125
18	1.6833	1.6828	0.0781
22	1.6839	1.6839	0.0003
26	1.6839	1.6839	0
30	1.6839	1.6839	0

On the held-out test texts (1,847,023 characters): 1.7778 bits per character, perplexity 3.429 per character.

Training logs in this repository: loss.txt, training.log. Task evaluations will follow with the accompanying paper.

Training corpora

13,478 texts, 154.4M characters, mostly diplomatic transcriptions with a focus before 1500. Text is kept as transcribed (Unicode NFC only), with all special characters. About 1 % of the texts of each corpus is held out for validation and 1 % for test. The training texts are not distributed with this model.

• Referenzkorpus Mittelhochdeutsch (ReM), CC BY-SA 4.0
• Referenzkorpus Frühneuhochdeutsch (ReF.MLU), CC BY-SA 4.0
• Referenzkorpus Frühneuhochdeutsch (ReF.RUB), CC BY-SA 4.0
• Referenzkorpus Frühneuhochdeutsch (ReF.UP), CC BY-SA 4.0
• Referenzkorpus Mittelniederdeutsch/Niederrheinisch (ReN), annotated texts, CC BY 4.0
• Referenzkorpus Mittelniederdeutsch/Niederrheinisch (ReN), transcriptions, CC BY 4.0
• Mittelhochdeutsche Begriffsdatenbank (MHDBDB), CC BY-NC-SA 4.0
• Source texts of the Mittelhochdeutsches Wörterbuch (MWB)
• Corpus der altdeutschen Originalurkunden (CAO), CC BY 4.0
• Königsfelden edition (Zürich / Staatsarchiv Aargau), CC BY 4.0
• Sammlung Schweizerischer Rechtsquellen (SSRQ), German pieces, CC BY-NC-SA 4.0
• Nuremberg letter books, vols. 2-5, CC BY-NC 4.0 / CC BY 4.0
• Nuremberg letter books, vols. 6-14, uncorrected HTR, CC BY 4.0
• Lüne letters (HAB Wolfenbüttel), CC BY-SA 3.0 DE
• Deutsches Textarchiv, texts up to 1500, mostly CC BY-NC 3.0
• HTR/OCR transcriptions by Stefan Tomasek

In this build every SSRQ text was read twice, so these texts carry double weight in training.

Context

Collaboration between Prof. Dr. Michael Schonhardt (Technische Universität Darmstadt / AdWL Mainz) and Dr. Stefan Tomasek (Universität Würzburg). The models were trained at the Fachgebiet Digitale Editorik und Kulturgeschichte des Mittelalters, TU Darmstadt, as part of the LOEWE Exploration project "Embedding the Past", funded by the HMWK Hessen. Training was conducted by Michael Schonhardt.

Licence

CC BY-NC-SA 4.0, as the NonCommercial and ShareAlike terms of several corpora apply.

Citation

@misc{mhd_flair_raw_forward_2026,
  author       = {Schonhardt, Michael and Tomasek, Stefan},
  title        = {MHD Flair character language model (raw, forward)},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/dekmATtud/mhd-flair-raw-forward}},
  note         = {LOEWE Exploration Embedding the Past, TU Darmstadt. Training: Michael Schonhardt}
}