ocr-projects-archive / ocr-tech /docs /EVAL_HARNESS_RESEARCH_GOAL.md
prosperious's picture
Add 1000 files (OCR_INDRA/.gitignore ...)
f9bc0c1 verified
|
Raw History Blame Contribute Delete
7.48 kB
# Research-Backed Eval Harness Goal
Updated: 2026-06-20
## Sources Read
- Web/paper search performed on 2026-06-20 before drafting this goal.
- OCR-D quality assurance guidance: representative ground truth, realistic images,
manual transcription, page/region localization, CER/WER definitions, reading-order
and layout caveats, IoU/object-detection evaluation, and NFC normalization guidance:
<https://ocr-d.de/en/spec/ocrd_eval.html>
- Neudecker et al., "A survey of OCR evaluation tools and metrics" (HIP 2021):
OCR evaluation is tool- and normalization-sensitive, and text accuracy depends on
alignment, reading order, layout, and the exact edit-distance metric:
<https://dl.acm.org/doi/10.1145/3476887.3476888>
- Santos, "OCR Evaluation Tools for the 21st Century" (ComputEL 2019): OCR evaluation
for modern and endangered-language work needs Unicode-capable character/word
reporting rather than ASCII-era assumptions:
<https://doi.org/10.33011/computel.v1i.345>
- Unicode Standard Annex #29: extended grapheme clusters are the default
programmatic approximation of user-perceived characters:
<https://www.unicode.org/reports/tr29/>
- PaddleOCR `RecMetric`: `norm_edit_dis` is one minus a per-sample average of
normalized Levenshtein distance, not corpus micro CER:
<https://raw.githubusercontent.com/PaddlePaddle/PaddleOCR/main/ppocr/metrics/rec_metric.py>
- Leifert et al., "End-to-End Measure for Text Recognition" (ICDAR 2019): end-to-end
text recognition evaluation must account for recognition plus segmentation/alignment,
not just isolated text strings:
<https://doi.org/10.1109/ICDAR.2019.00-16>
- Beyene and Dancy, "A Survey of OCR Evaluation Methods and Metrics and the
Invisibility of Historical Documents" (arXiv 2026 preprint): modern OCR and document
understanding evaluations overfit to modern/institutional benchmark genres, often
miss historical/community archive layout failures, and can make marginalized
materials invisible even when headline CER/WER looks acceptable:
<https://arxiv.org/html/2603.25761v1>
- Springmann et al., "Automatic quality evaluation and (semi-) automatic improvement
of OCR models for historical printings": GT is scarce, manual lines can change
results materially, and confidence/proxy measures are not replacements for true GT:
<https://arxiv.org/abs/1606.05157>
- Gebru et al., "Datasheets for Datasets": datasets need explicit motivation,
composition, collection, processing, intended use, and limitations:
<https://arxiv.org/abs/1803.09010>
- Mitchell et al., "Model Cards for Model Reporting": model releases need benchmarked
performance, intended use, limitations, evaluation procedure, and per-condition
reporting:
<https://arxiv.org/abs/1810.03993>
## Dense Goal Paragraph
Build a claim-grade OCR evaluation harness for the Nepal/diaspora recognizer suite
whose default output makes an unsafe paper claim obvious before anyone can quote a
number: every supported target, including the shared Devanagari languages
Majhi, Magar, Sunuwar-Devanagari, Magahi, Bhojpuri, Thami, Wambule, and Darai plus
native-script targets such as Limbu/Sirijonga, Sunuwar/Jenticha, Newa/Prachalit,
Tirhuta, Lepcha, Kirat Rai, Gurung Khema, and any later Tamang, Bantawa/Rai, Kulung,
Yakkha, Tharu/Rana Tharu, or English scope, must be evaluated as its own
language/script/domain slice from a frozen, versioned, real-image eval pack with
image bytes, labels, manifest rows, source-document/page/bbox provenance, split role,
domain, ground-truth source and trust evidence, native-reader calibration when GT is
machine-converted, row claim eligibility, script inventory coverage, and leakage
declarations; every run must record exact model/checkpoint/config/dictionary or model
card, prediction artifact, normalization profile, Unicode and grapheme-segmentation
behavior, command, timestamp, machine, tool versions, and artifact hashes; every
reported metric must be recomputable from canonical row artifacts as corpus micro
edit-distance CER over both Unicode codepoints and UAX #29 extended grapheme clusters,
not PaddleOCR `norm_edit_dis` or any per-line macro surrogate, with exact-line
accuracy, matched and unmatched rows, missing-prediction accounting, confusion
summaries, worst examples, reviewer artifacts, deterministic confidence intervals,
and paired same-pack tests for model comparisons; recognizer oracle-crop results must
remain separate from page-level OCR results, where page claims additionally require
detection/crop lineage, reading-order and layout alignment, line-bbox/IoU or explicit
reference matching, table/figure/text reconstruction metrics where relevant,
historical/newspaper degradation and multi-column failure visibility, and visible
coverage of missing documents or references; the harness must fail or hard-warn on
missing or mutated files, duplicate IDs or image paths, stale hashes, unsupported or
asymmetric normalization, low match rate, unrun or positive train/dev/test leakage,
checkpoint selection on test data, synthetic or rendered data presented as real
evidence, transliterated or machine-converted labels presented as human-verified
evidence, human GT without review/agreement evidence, too-small or script-incomplete
packs, aggregate scores that hide per-language/script/domain/GT failures, modern-
benchmark averages that do not exercise the target community archives, and any
hand-edited report whose JSON/TSV/CSV artifacts no longer recompute; a reviewer
opening one output directory must be able to answer, without trusting prose, exactly
what was scored, against which GT, how trustworthy that GT is, whether the test set was
frozen and unseen for selection, which Unicode, script, layout, provenance, and archive
coverage assumptions moved the number, how uncertain the number is, what failed, whose
material is still unsupported, and whether the result is safe to cite in a paper.
## Non-Negotiable Requirements
- Real held-out eval packs are evidence objects, not folders of images. They need
immutable manifests, hashes, provenance, split status, GT trust, domain labels,
claim eligibility, script inventory coverage, and a datasheet-style summary.
- Recognizer and page OCR are different tasks. Oracle-crop CER may support a recognizer
claim only; page OCR claims need detector output, crop lineage, reading order,
layout alignment, and missing-page/reference accounting.
- Metrics must be recomputed from emitted row artifacts. Aggregate JSON is not a
source of truth unless `predictions.tsv`, `page-metrics.csv`, alignment rows, and
references recompute to the same values.
- Unicode is part of the metric contract. The harness must record normalization,
Unicode/grapheme implementation behavior, codepoint CER, grapheme CER, and missing
required script inventory items.
- Claim-facing reports need uncertainty and comparison discipline. Point CER alone is
not enough; small packs need hard warnings, and model-vs-model claims need paired
same-pack analysis.
- Ground truth quality must be visible. Machine conversion and transliteration are
scoreable, but they are not human-verified evidence unless a native-reader
calibration artifact supports that upgrade.
- The harness must be hostile to silent omissions. Missing predictions, duplicate
rows, leaked train/dev examples, stale hashes, skipped pages, unsupported
normalization, and stale Markdown should fail audit or block claim safety.