MHD CharLSTM character language model (raw, forward)

A small forward character language model for medieval and early modern German (Middle High German, Early New High German, Middle Low German). Use it together with dekmATtud/mhd-charlm-raw-backward.

Usage

import sys
from pathlib import Path
from huggingface_hub import hf_hub_download

sys.path.insert(0, str(Path(hf_hub_download("dekmATtud/mhd-charlm-raw-forward", "charlm.py")).parent))
from charlm import load, token_vectors

fw = load(hf_hub_download("dekmATtud/mhd-charlm-raw-forward", "charlm.pt"))
bw = load(hf_hub_download("dekmATtud/mhd-charlm-raw-backward", "charlm.pt"))
vecs = token_vectors(fw, bw, "Diz iſt ein ſchoͤne rede")   # [tokens, 1024]

charlm.py needs only torch and numpy.

Model

Character embedding 128, one LSTM layer of 512 units, linear decoder over 311 symbols.

Training

PyTorch 2.11.0. Adam, learning rate 0.001, reduced by 0.25 after 1 epoch without improvement, gradient clipping 1.0, windows of 256 characters, batch size 64, seed 42. Stopped at epoch 42 (no improvement for 3 epochs (best: epoch 39)). charlm.pt is the best epoch by validation loss.

Training and validation in bits per character:

epoch train bpc valid bpc learning rate
1 2.4232 2.2062 0.001
7 2.0431 2.0620 0.001
13 2.0111 2.0441 0.001
19 1.9964 2.0330 0.001
24 1.9890 2.0278 0.001
30 1.9527 2.0022 0.00025
36 1.9404 1.9957 6.25e-05
39 1.9376 1.9944 1.5625e-05
42 1.9368 1.9944 3.90625e-06

On the held-out test texts (1,847,023 characters): 2.0914 bits per character, perplexity 4.262 per character.

Training logs in this repository: train_log.jsonl. Task evaluations will follow with the accompanying paper.

Training corpora

13,478 texts, 154.4M characters, mostly diplomatic transcriptions with a focus before 1500. Text is kept as transcribed (Unicode NFC only), with all special characters. About 1 % of the texts of each corpus is held out for validation and 1 % for test. The training texts are not distributed with this model.

  • Referenzkorpus Mittelhochdeutsch (ReM), CC BY-SA 4.0
  • Referenzkorpus Frühneuhochdeutsch (ReF.MLU), CC BY-SA 4.0
  • Referenzkorpus Frühneuhochdeutsch (ReF.RUB), CC BY-SA 4.0
  • Referenzkorpus Frühneuhochdeutsch (ReF.UP), CC BY-SA 4.0
  • Referenzkorpus Mittelniederdeutsch/Niederrheinisch (ReN), annotated texts, CC BY 4.0
  • Referenzkorpus Mittelniederdeutsch/Niederrheinisch (ReN), transcriptions, CC BY 4.0
  • Mittelhochdeutsche Begriffsdatenbank (MHDBDB), CC BY-NC-SA 4.0
  • Source texts of the Mittelhochdeutsches Wörterbuch (MWB)
  • Corpus der altdeutschen Originalurkunden (CAO), CC BY 4.0
  • Königsfelden edition (Zürich / Staatsarchiv Aargau), CC BY 4.0
  • Sammlung Schweizerischer Rechtsquellen (SSRQ), German pieces, CC BY-NC-SA 4.0
  • Nuremberg letter books, vols. 2-5, CC BY-NC 4.0 / CC BY 4.0
  • Nuremberg letter books, vols. 6-14, uncorrected HTR, CC BY 4.0
  • Lüne letters (HAB Wolfenbüttel), CC BY-SA 3.0 DE
  • Deutsches Textarchiv, texts up to 1500, mostly CC BY-NC 3.0
  • HTR/OCR transcriptions by Stefan Tomasek

In this build every SSRQ text was read twice, so these texts carry double weight in training.

Context

Collaboration between Prof. Dr. Michael Schonhardt (Technische Universität Darmstadt / AdWL Mainz) and Dr. Stefan Tomasek (Universität Würzburg). The models were trained at the Fachgebiet Digitale Editorik und Kulturgeschichte des Mittelalters, TU Darmstadt, as part of the LOEWE Exploration project "Embedding the Past", funded by the HMWK Hessen. Training was conducted by Michael Schonhardt.

Licence

CC BY-NC-SA 4.0, as the NonCommercial and ShareAlike terms of several corpora apply.

Citation

@misc{mhd_charlm_raw_forward_2026,
  author       = {Schonhardt, Michael and Tomasek, Stefan},
  title        = {MHD CharLSTM character language model (raw, forward)},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/dekmATtud/mhd-charlm-raw-forward}},
  note         = {LOEWE Exploration Embedding the Past, TU Darmstadt. Training: Michael Schonhardt}
}
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support