Image-Text-to-Text
Transformers
TensorBoard
Safetensors
MiniCPM-OCR
Swedish
minicpm_ocr
handwriting
historical-documents
archives
htr
ocr
conversational
Instructions to use Gabriel/diffu-minicpm-ocr-htr with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Gabriel/diffu-minicpm-ocr-htr with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="Gabriel/diffu-minicpm-ocr-htr") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("Gabriel/diffu-minicpm-ocr-htr") model = AutoModelForMultimodalLM.from_pretrained("Gabriel/diffu-minicpm-ocr-htr", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - MiniCPM-OCR
How to use Gabriel/diffu-minicpm-ocr-htr with MiniCPM-OCR:
pip install "minicpm-ocr[all]" huggingface_hub # Deploy Gabriel/diffu-minicpm-ocr-htr as a recognition service before running the Python example: # https://github.com/OpenBMB/MiniCPM-OCR/blob/main/docs/vllm_deploy_en.md # vLLM: load the model above; use --port 8080, # --served-model-name minicpm-ocr and --max-model-len 40960. # Layout detection uses cuda:0; recognition runs on the deployed service.
from importlib.resources import files from pathlib import Path from minicpmvocr import load_config, pipelined_run from huggingface_hub import snapshot_download layout_model = snapshot_download("PaddlePaddle/PP-DocLayoutV3") cfg = load_config( str(files("minicpmvocr").joinpath("configs/minicpmvocr.yaml")), overrides={ "input": {"file_path": str(Path("demo.png").resolve())}, "engine": { "type": "vllm", "server_url": "http://127.0.0.1:8080", "model_name": "minicpm-ocr", "max_model_len": 40960, }, "layout": {"model_dir": layout_model, "device": "cuda:0"}, # Render PDFs in-process so this example also works without a main guard. "pipeline": {"overlap_layout": True, "render_workers": 1}, }, ) pipelined_run(cfg) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use Gabriel/diffu-minicpm-ocr-htr with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "Gabriel/diffu-minicpm-ocr-htr" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Gabriel/diffu-minicpm-ocr-htr", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/Gabriel/diffu-minicpm-ocr-htr
- SGLang
How to use Gabriel/diffu-minicpm-ocr-htr with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "Gabriel/diffu-minicpm-ocr-htr" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Gabriel/diffu-minicpm-ocr-htr", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "Gabriel/diffu-minicpm-ocr-htr" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Gabriel/diffu-minicpm-ocr-htr", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use Gabriel/diffu-minicpm-ocr-htr with Docker Model Runner:
docker model run hf.co/Gabriel/diffu-minicpm-ocr-htr
File size: 12,367 Bytes
7735d9e | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 | ---
license: apache-2.0
base_model: openbmb/MiniCPM-OCR-1.0
base_model_relation: finetune
pipeline_tag: image-text-to-text
library_name: transformers
language:
- sv
tags:
- minicpm-ocr
- handwriting
- historical-documents
- archives
- htr
- ocr
---
# diffu-minicpm-ocr-htr
A full fine-tune of [MiniCPM-OCR-1.0](https://huggingface.co/openbmb/MiniCPM-OCR-1.0) that reads Swedish handwritten
archive records: one text line, or a whole text region (its lines in reading order, one to a line). It is the reader
of [diffu](https://github.com/Borg93/diffu2), trained to be compared with Riksarkivet's
[Swedish Lion TrOCR](https://huggingface.co/Riksarkivet/swedish-lion-trocr-large-handwritten-char) on the same data and
the same test lines.
Its proof-reader is [diffu-minicpm-ocr-htr-jev](https://huggingface.co/Gabriel/diffu-minicpm-ocr-htr-jev) and its DSpark draft for faster decoding
[diffu-minicpm-ocr-htr-dspark](https://huggingface.co/Gabriel/diffu-minicpm-ocr-htr-dspark), where they have been published.
## Training
- **Data:** Riksarkivet's GT-Lejonet HTR ground truth (February 2026), the Swedish Lion's training side (its test split
`test_0050` held out): 1,481,877 text lines, each crop white outside its line's polygon, and 83,072 text regions,
each crop masked the same way. Lines whose text marks an uncertain reading (`??`, `(?)`, `[`, `]`) and lines under
10 pixels are left out, as the baseline left them out. The text is trained as transcribed, not normalised.
- **Recipe:** OpenBMB's full fine-tune (vision encoder and language model trained, the aligner frozen), learning rate
5e-6 on a cosine schedule planned over 3 epochs, 128 samples a step, bfloat16, on two RTX PRO 6000 Blackwell GPUs.
Stopped after 28,000 of its 36,681 steps (2.3 epochs), where the held-out lines' error had all but stopped falling.
- **Checkpoint:** checkpoint-28000, the one of lowest corpus CER on 1,000 held-out dev lines.
## Use
Needs vLLM at commit `fa3b9d13` or later (MiniCPM-OCR support, [vllm#64105](https://github.com/vllm-project/vllm/pull/64105)),
as OpenBMB's installer builds it ([OpenBMB/MiniCPM-OCR](https://github.com/OpenBMB/MiniCPM-OCR)). Crop as in training:
a line's or region's bounding box, white outside its polygon, scaled down to at most 448 x 448 pixels' worth (200,704;
a region up to 9 times that).
```bash
vllm serve Gabriel/diffu-minicpm-ocr-htr --served-model-name htr --max-model-len 8192 \
--limit-mm-per-prompt '{"image": 4, "video": 0}' \
--mm-processor-kwargs '{"downsample_mode": "4x", "max_slice_nums": 9}' --trust-request-mm-kwargs \
--default-chat-template-kwargs '{"enable_thinking": false}'
```
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="-")
reply = client.chat.completions.create(
model="htr",
messages=[{"role": "user", "content": [
{"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,..."}},
{"type": "text", "text": "Text Recognition:"},
]}],
max_tokens=512, # a line; a region takes up to 4096
temperature=0.01, top_p=0.001, extra_body={"top_k": 1, "repetition_penalty": 1.0},
)
print(reply.choices[0].message.content)
```
Only `Text Recognition:` was trained. A reading that ends with `finish_reason == "length"` has looped (dotted leader
lines do); cut its repeated tail, as the `loops cut` columns below do.
## Evaluation
This checkpoint read through vLLM as above, beside the Swedish Lion TrOCR's greedy readings of the same lines: 5,000
lines of the baseline's own test split (`test_0050`, `test_*` rows) and all 6,358 lines of
[`Riksarkivet/eval_htr_out_of_domain_lines`](https://huggingface.co/datasets/Riksarkivet/eval_htr_out_of_domain_lines)
(`ood_*` rows). Each table is one reader on one set (`trocr` the baseline, `minicpm` this model).
- **mean CER:** the per-line mean, whitespace normalised (the baseline's own scoring).
- **CER, WER (OpenBMB):** corpus rates, all edits over all reference characters or words (NFC, ends trimmed).
- **mean CER, norm.:** the baseline's training normalisation applied to both sides.
- **cut off:** readings that reached the token limit.
- **loops cut:** each reading scored again with a repeating tail cut to one copy.
```
runs/htr/eval/ood_trocr_greedy.jsonl
ββββββββββ³ββββββββ³βββββββββββ³ββββββββββββββββ³ββββββββββββββββ³ββββββββββββββββββ³ββββββββββ³ββββββββββββββββββββββ³βββββββββββββββββββββββββββ³ββββββββββββ
β subset β lines β mean CER β CER (OpenBMB) β WER (OpenBMB) β mean CER, norm. β cut off β mean CER, loops cut β CER (OpenBMB), loops cut β loops cut β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β all β 6358 β 6.67% β 5.30% β 21.12% β 6.62% β 0 β 6.67% β 5.30% β 0 β
ββββββββββ΄ββββββββ΄βββββββββββ΄ββββββββββββββββ΄ββββββββββββββββ΄ββββββββββββββββββ΄ββββββββββ΄ββββββββββββββββββββββ΄βββββββββββββββββββββββββββ΄ββββββββββββ
runs/htr/final/ood_minicpm.jsonl
ββββββββββ³ββββββββ³βββββββββββ³ββββββββββββββββ³ββββββββββββββββ³ββββββββββββββββββ³ββββββββββ³ββββββββββββββββββββββ³βββββββββββββββββββββββββββ³ββββββββββββ
β subset β lines β mean CER β CER (OpenBMB) β WER (OpenBMB) β mean CER, norm. β cut off β mean CER, loops cut β CER (OpenBMB), loops cut β loops cut β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β all β 6358 β 7.21% β 5.55% β 21.76% β 7.18% β 0 β 7.21% β 5.55% β 0 β
ββββββββββ΄ββββββββ΄βββββββββββ΄ββββββββββββββββ΄ββββββββββββββββ΄ββββββββββββββββββ΄ββββββββββ΄ββββββββββββββββββββββ΄βββββββββββββββββββββββββββ΄ββββββββββββ
runs/htr/ckpt_eval/test_trocr.jsonl
ββββββββββ³ββββββββ³βββββββββββ³ββββββββββββββββ³ββββββββββββββββ³ββββββββββββββββββ³ββββββββββ³ββββββββββββββββββββββ³βββββββββββββββββββββββββββ³ββββββββββββ
β subset β lines β mean CER β CER (OpenBMB) β WER (OpenBMB) β mean CER, norm. β cut off β mean CER, loops cut β CER (OpenBMB), loops cut β loops cut β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β all β 5000 β 4.17% β 3.58% β 14.61% β 3.87% β 0 β 4.17% β 3.58% β 0 β
ββββββββββ΄ββββββββ΄βββββββββββ΄ββββββββββββββββ΄ββββββββββββββββ΄ββββββββββββββββββ΄ββββββββββ΄ββββββββββββββββββββββ΄βββββββββββββββββββββββββββ΄ββββββββββββ
runs/htr/final/test_minicpm.jsonl
ββββββββββ³ββββββββ³βββββββββββ³ββββββββββββββββ³ββββββββββββββββ³ββββββββββββββββββ³ββββββββββ³ββββββββββββββββββββββ³βββββββββββββββββββββββββββ³ββββββββββββ
β subset β lines β mean CER β CER (OpenBMB) β WER (OpenBMB) β mean CER, norm. β cut off β mean CER, loops cut β CER (OpenBMB), loops cut β loops cut β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β all β 5000 β 5.22% β 5.64% β 20.06% β 4.42% β 3 β 4.46% β 3.92% β 3 β
ββββββββββ΄ββββββββ΄βββββββββββ΄ββββββββββββββββ΄ββββββββββββββββ΄ββββββββββββββββββ΄ββββββββββ΄ββββββββββββββββββββββ΄βββββββββββββββββββββββββββ΄ββββββββββββ
```
This model read the 5,000 test lines at 68.55 lines a second on one RTX PRO 6000 Blackwell (vLLM, 64 requests at a
time); the baseline read them at 7.91 (transformers, float32, batches of 32, greedy).
## Limitations
- Trained and tested on GT-Lejonet's collections. Hands, periods and kinds of record it has not seen read worse (the
out-of-domain rows).
- Like other vision-language readers, it can write a familiar modern spelling for an old one, or drop a letter, where
a dedicated line recogniser copies the strokes.
- Near-greedy decoding can repeat one unit to the token limit; the `loops cut` columns show what cutting such a tail
changes.
## Licence
Apache-2.0, as its base model.
|