diffu-minicpm-ocr-htr

A full fine-tune of MiniCPM-OCR-1.0 that reads Swedish handwritten archive records: one text line, or a whole text region (its lines in reading order, one to a line). It is the reader of diffu, trained to be compared with Riksarkivet's Swedish Lion TrOCR on the same data and the same test lines.

Its proof-reader is diffu-minicpm-ocr-htr-jev and its DSpark draft for faster decoding diffu-minicpm-ocr-htr-dspark, where they have been published.

Training

• Data: Riksarkivet's GT-Lejonet HTR ground truth (February 2026), the Swedish Lion's training side (its test split test_0050 held out): 1,481,877 text lines, each crop white outside its line's polygon, and 83,072 text regions, each crop masked the same way. Lines whose text marks an uncertain reading (??, (?), [, ]) and lines under 10 pixels are left out, as the baseline left them out. The text is trained as transcribed, not normalised.
• Recipe: OpenBMB's full fine-tune (vision encoder and language model trained, the aligner frozen), learning rate 5e-6 on a cosine schedule planned over 3 epochs, 128 samples a step, bfloat16, on two RTX PRO 6000 Blackwell GPUs. Stopped after 28,000 of its 36,681 steps (2.3 epochs), where the held-out lines' error had all but stopped falling.
• Checkpoint: checkpoint-28000, the one of lowest corpus CER on 1,000 held-out dev lines.

Use

Needs vLLM at commit fa3b9d13 or later (MiniCPM-OCR support, vllm#64105), as OpenBMB's installer builds it (OpenBMB/MiniCPM-OCR). Crop as in training: a line's or region's bounding box, white outside its polygon, scaled down to at most 448 x 448 pixels' worth (200,704; a region up to 9 times that).

vllm serve Gabriel/diffu-minicpm-ocr-htr --served-model-name htr --max-model-len 8192 \
  --limit-mm-per-prompt '{"image": 4, "video": 0}' \
  --mm-processor-kwargs '{"downsample_mode": "4x", "max_slice_nums": 9}' --trust-request-mm-kwargs \
  --default-chat-template-kwargs '{"enable_thinking": false}'

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="-")
reply = client.chat.completions.create(
    model="htr",
    messages=[{"role": "user", "content": [
        {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,..."}},
        {"type": "text", "text": "Text Recognition:"},
    ]}],
    max_tokens=512,  # a line; a region takes up to 4096
    temperature=0.01, top_p=0.001, extra_body={"top_k": 1, "repetition_penalty": 1.0},
)
print(reply.choices[0].message.content)

Only Text Recognition: was trained. A reading that ends with finish_reason == "length" has looped (dotted leader lines do); cut its repeated tail, as the loops cut columns below do.

Evaluation

This checkpoint read through vLLM as above, beside the Swedish Lion TrOCR's greedy readings of the same lines: 5,000 lines of the baseline's own test split (test_0050, test_* rows) and all 6,358 lines of Riksarkivet/eval_htr_out_of_domain_lines (ood_* rows). Each table is one reader on one set (trocr the baseline, minicpm this model).

• mean CER: the per-line mean, whitespace normalised (the baseline's own scoring).
• CER, WER (OpenBMB): corpus rates, all edits over all reference characters or words (NFC, ends trimmed).
• mean CER, norm.: the baseline's training normalisation applied to both sides.
• cut off: readings that reached the token limit.
• loops cut: each reading scored again with a repeating tail cut to one copy.

                                                         runs/htr/eval/ood_trocr_greedy.jsonl                                                         
┏━━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ subset ┃ lines ┃ mean CER ┃ CER (OpenBMB) ┃ WER (OpenBMB) ┃ mean CER, norm. ┃ cut off ┃ mean CER, loops cut ┃ CER (OpenBMB), loops cut ┃ loops cut ┃
┡━━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ all    │  6358 │    6.67% │         5.30% │        21.12% │           6.62% │       0 │               6.67% │                    5.30% │         0 │
└────────┴───────┴──────────┴───────────────┴───────────────┴─────────────────┴─────────┴─────────────────────┴──────────────────────────┴───────────┘
                                                           runs/htr/final/ood_minicpm.jsonl                                                           
┏━━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ subset ┃ lines ┃ mean CER ┃ CER (OpenBMB) ┃ WER (OpenBMB) ┃ mean CER, norm. ┃ cut off ┃ mean CER, loops cut ┃ CER (OpenBMB), loops cut ┃ loops cut ┃
┡━━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ all    │  6358 │    7.21% │         5.55% │        21.76% │           7.18% │       0 │               7.21% │                    5.55% │         0 │
└────────┴───────┴──────────┴───────────────┴───────────────┴─────────────────┴─────────┴─────────────────────┴──────────────────────────┴───────────┘
                                                         runs/htr/ckpt_eval/test_trocr.jsonl                                                          
┏━━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ subset ┃ lines ┃ mean CER ┃ CER (OpenBMB) ┃ WER (OpenBMB) ┃ mean CER, norm. ┃ cut off ┃ mean CER, loops cut ┃ CER (OpenBMB), loops cut ┃ loops cut ┃
┡━━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ all    │  5000 │    4.17% │         3.58% │        14.61% │           3.87% │       0 │               4.17% │                    3.58% │         0 │
└────────┴───────┴──────────┴───────────────┴───────────────┴─────────────────┴─────────┴─────────────────────┴──────────────────────────┴───────────┘
                                                          runs/htr/final/test_minicpm.jsonl                                                           
┏━━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ subset ┃ lines ┃ mean CER ┃ CER (OpenBMB) ┃ WER (OpenBMB) ┃ mean CER, norm. ┃ cut off ┃ mean CER, loops cut ┃ CER (OpenBMB), loops cut ┃ loops cut ┃
┡━━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ all    │  5000 │    5.22% │         5.64% │        20.06% │           4.42% │       3 │               4.46% │                    3.92% │         3 │
└────────┴───────┴──────────┴───────────────┴───────────────┴─────────────────┴─────────┴─────────────────────┴──────────────────────────┴───────────┘

This model read the 5,000 test lines at 68.55 lines a second on one RTX PRO 6000 Blackwell (vLLM, 64 requests at a time); the baseline read them at 7.91 (transformers, float32, batches of 32, greedy).

Limitations

• Trained and tested on GT-Lejonet's collections. Hands, periods and kinds of record it has not seen read worse (the out-of-domain rows).
• Like other vision-language readers, it can write a familiar modern spelling for an old one, or drop a letter, where a dedicated line recogniser copies the strokes.
• Near-greedy decoding can repeat one unit to the token limit; the loops cut columns show what cutting such a tail changes.

Licence

Apache-2.0, as its base model.