Equal CER, Unequal Damage: Severity-Weighted Character Error Rate for OCR
Abstract
A new metric called semCER uses an LLM judge to weight grapheme edits by severity, improving alignment with human ratings over standard character error rate for OCR and vision-language models.
Character Error Rate (CER), the standard metric for OCR and vision-language models (VLMs), charges a typo as much as a wrong name. We introduce semCER, which keeps CER's alignment and denominator and charges each grapheme edit a severity from an LLM judge, with no word segmentation. Normalized CER keeps the score of a whole transcription, where it tracks readers as well as semCER or better, and semCER breaks that score down into named edits. The charges track readers' ratings edit by edit in English, Vietnamese, and Chinese and put the edits readers rate most severe near the top of a sorted list. On pairs drawn where CER and semCER disagreed, readers side with semCER more often than with normalized CER in English and Chinese, and equally often in Vietnamese. Across 16 languages, the judge charges a wrong diacritic or tone mark almost nothing in Spanish but nearly a wrong letter's severity in Telugu. Where we can count real words, a general VLM's wrong letter forms one far above chance while a classical pipeline's stays near it. We release the 61 million judged edits, the aggregated reader ratings, and the distilled scorer at https://github.com/nguyenx2t/semcer.
Community
Hi all, author here. Thanks for taking a look.
CER charges every character error 1, so a typo costs the same as a wrong digit in a price or a wrong name. semCER keeps CER's alignment and denominator, and gives each edit a severity from 0 (harmless) to 1 (meaning lost).
What we found:
- Edit by edit, the severities track human readers' ratings in English, Vietnamese and Chinese.
- For whole lines, normalized CER tracks readers as well as semCER or better. So we see semCER as a breakdown that sits next to CER, not a replacement: it tells you which errors hurt.
- Across 16 languages, the same error type gets very different severities. A wrong diacritic costs almost nothing in Spanish but nearly a wrong letter's cost in Telugu.
Limitations: the severities come from an LLM judge (DeepSeek V4 Flash), distilled into a 0.8B model that runs locally. Reader studies cover only English, Vietnamese and Chinese, and differences below about 0.1 aren't reliable.
Everything is open:
- Code:
pip install semcer, https://github.com/nguyenx2t/semcer - Model: https://huggingface.co/nguyenx2t/semcer-qwen3.5-0.8b
- 61M judged edits + reader ratings: https://huggingface.co/datasets/nguyenx2t/semcer-judged-edits
- Colab: https://colab.research.google.com/github/nguyenx2t/semcer/blob/main/notebooks/quickstart.ipynb
Feedback and bug reports are very welcome, especially from people working on languages we haven't checked against readers.
Models citing this paper 1
Datasets citing this paper 1
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper