MHD CharLSTM character language model (raw, forward)
A small forward character language model for medieval and early modern German (Middle High German, Early New High German, Middle Low German). Use it together with dekmATtud/mhd-charlm-raw-backward.
Usage
import sys
from pathlib import Path
from huggingface_hub import hf_hub_download
sys.path.insert(0, str(Path(hf_hub_download("dekmATtud/mhd-charlm-raw-forward", "charlm.py")).parent))
from charlm import load, token_vectors
fw = load(hf_hub_download("dekmATtud/mhd-charlm-raw-forward", "charlm.pt"))
bw = load(hf_hub_download("dekmATtud/mhd-charlm-raw-backward", "charlm.pt"))
vecs = token_vectors(fw, bw, "Diz iſt ein ſchoͤne rede") # [tokens, 1024]
charlm.py needs only torch and numpy.
Model
Character embedding 128, one LSTM layer of 512 units, linear decoder over 311 symbols.
Training
PyTorch 2.11.0. Adam, learning rate 0.001, reduced by 0.25 after 1 epoch without improvement, gradient clipping 1.0, windows of 256 characters, batch size 64, seed 42. Stopped at epoch 42 (no improvement for 3 epochs (best: epoch 39)). charlm.pt is the best epoch by validation loss.
Training and validation in bits per character:
| epoch | train bpc | valid bpc | learning rate |
|---|---|---|---|
| 1 | 2.4232 | 2.2062 | 0.001 |
| 7 | 2.0431 | 2.0620 | 0.001 |
| 13 | 2.0111 | 2.0441 | 0.001 |
| 19 | 1.9964 | 2.0330 | 0.001 |
| 24 | 1.9890 | 2.0278 | 0.001 |
| 30 | 1.9527 | 2.0022 | 0.00025 |
| 36 | 1.9404 | 1.9957 | 6.25e-05 |
| 39 | 1.9376 | 1.9944 | 1.5625e-05 |
| 42 | 1.9368 | 1.9944 | 3.90625e-06 |
On the held-out test texts (1,847,023 characters): 2.0914 bits per character, perplexity 4.262 per character.
Training logs in this repository: train_log.jsonl. Task evaluations will follow with the accompanying paper.
Training corpora
13,478 texts, 154.4M characters, mostly diplomatic transcriptions with a focus before 1500. Text is kept as transcribed (Unicode NFC only), with all special characters. About 1 % of the texts of each corpus is held out for validation and 1 % for test. The training texts are not distributed with this model.
- Referenzkorpus Mittelhochdeutsch (ReM), CC BY-SA 4.0
- Referenzkorpus Frühneuhochdeutsch (ReF.MLU), CC BY-SA 4.0
- Referenzkorpus Frühneuhochdeutsch (ReF.RUB), CC BY-SA 4.0
- Referenzkorpus Frühneuhochdeutsch (ReF.UP), CC BY-SA 4.0
- Referenzkorpus Mittelniederdeutsch/Niederrheinisch (ReN), annotated texts, CC BY 4.0
- Referenzkorpus Mittelniederdeutsch/Niederrheinisch (ReN), transcriptions, CC BY 4.0
- Mittelhochdeutsche Begriffsdatenbank (MHDBDB), CC BY-NC-SA 4.0
- Source texts of the Mittelhochdeutsches Wörterbuch (MWB)
- Corpus der altdeutschen Originalurkunden (CAO), CC BY 4.0
- Königsfelden edition (Zürich / Staatsarchiv Aargau), CC BY 4.0
- Sammlung Schweizerischer Rechtsquellen (SSRQ), German pieces, CC BY-NC-SA 4.0
- Nuremberg letter books, vols. 2-5, CC BY-NC 4.0 / CC BY 4.0
- Nuremberg letter books, vols. 6-14, uncorrected HTR, CC BY 4.0
- Lüne letters (HAB Wolfenbüttel), CC BY-SA 3.0 DE
- Deutsches Textarchiv, texts up to 1500, mostly CC BY-NC 3.0
- HTR/OCR transcriptions by Stefan Tomasek
In this build every SSRQ text was read twice, so these texts carry double weight in training.
Context
Collaboration between Prof. Dr. Michael Schonhardt (Technische Universität Darmstadt / AdWL Mainz) and Dr. Stefan Tomasek (Universität Würzburg). The models were trained at the Fachgebiet Digitale Editorik und Kulturgeschichte des Mittelalters, TU Darmstadt, as part of the LOEWE Exploration project "Embedding the Past", funded by the HMWK Hessen. Training was conducted by Michael Schonhardt.
Licence
CC BY-NC-SA 4.0, as the NonCommercial and ShareAlike terms of several corpora apply.
Citation
@misc{mhd_charlm_raw_forward_2026,
author = {Schonhardt, Michael and Tomasek, Stefan},
title = {MHD CharLSTM character language model (raw, forward)},
year = {2026},
publisher = {Hugging Face},
howpublished = {\url{https://huggingface.co/dekmATtud/mhd-charlm-raw-forward}},
note = {LOEWE Exploration Embedding the Past, TU Darmstadt. Training: Michael Schonhardt}
}