ocr-projects-archive / ocr-tech /docs /EVAL_HARNESS_RESEARCH_GOAL.md
prosperious's picture
Add 1000 files (OCR_INDRA/.gitignore ...)
f9bc0c1 verified
|
Raw History Blame Contribute Delete
7.48 kB

Research-Backed Eval Harness Goal

Updated: 2026-06-20

Sources Read

  • Web/paper search performed on 2026-06-20 before drafting this goal.
  • OCR-D quality assurance guidance: representative ground truth, realistic images, manual transcription, page/region localization, CER/WER definitions, reading-order and layout caveats, IoU/object-detection evaluation, and NFC normalization guidance: https://ocr-d.de/en/spec/ocrd_eval.html
  • Neudecker et al., "A survey of OCR evaluation tools and metrics" (HIP 2021): OCR evaluation is tool- and normalization-sensitive, and text accuracy depends on alignment, reading order, layout, and the exact edit-distance metric: https://dl.acm.org/doi/10.1145/3476887.3476888
  • Santos, "OCR Evaluation Tools for the 21st Century" (ComputEL 2019): OCR evaluation for modern and endangered-language work needs Unicode-capable character/word reporting rather than ASCII-era assumptions: https://doi.org/10.33011/computel.v1i.345
  • Unicode Standard Annex #29: extended grapheme clusters are the default programmatic approximation of user-perceived characters: https://www.unicode.org/reports/tr29/
  • PaddleOCR RecMetric: norm_edit_dis is one minus a per-sample average of normalized Levenshtein distance, not corpus micro CER: https://raw.githubusercontent.com/PaddlePaddle/PaddleOCR/main/ppocr/metrics/rec_metric.py
  • Leifert et al., "End-to-End Measure for Text Recognition" (ICDAR 2019): end-to-end text recognition evaluation must account for recognition plus segmentation/alignment, not just isolated text strings: https://doi.org/10.1109/ICDAR.2019.00-16
  • Beyene and Dancy, "A Survey of OCR Evaluation Methods and Metrics and the Invisibility of Historical Documents" (arXiv 2026 preprint): modern OCR and document understanding evaluations overfit to modern/institutional benchmark genres, often miss historical/community archive layout failures, and can make marginalized materials invisible even when headline CER/WER looks acceptable: https://arxiv.org/html/2603.25761v1
  • Springmann et al., "Automatic quality evaluation and (semi-) automatic improvement of OCR models for historical printings": GT is scarce, manual lines can change results materially, and confidence/proxy measures are not replacements for true GT: https://arxiv.org/abs/1606.05157
  • Gebru et al., "Datasheets for Datasets": datasets need explicit motivation, composition, collection, processing, intended use, and limitations: https://arxiv.org/abs/1803.09010
  • Mitchell et al., "Model Cards for Model Reporting": model releases need benchmarked performance, intended use, limitations, evaluation procedure, and per-condition reporting: https://arxiv.org/abs/1810.03993

Dense Goal Paragraph

Build a claim-grade OCR evaluation harness for the Nepal/diaspora recognizer suite whose default output makes an unsafe paper claim obvious before anyone can quote a number: every supported target, including the shared Devanagari languages Majhi, Magar, Sunuwar-Devanagari, Magahi, Bhojpuri, Thami, Wambule, and Darai plus native-script targets such as Limbu/Sirijonga, Sunuwar/Jenticha, Newa/Prachalit, Tirhuta, Lepcha, Kirat Rai, Gurung Khema, and any later Tamang, Bantawa/Rai, Kulung, Yakkha, Tharu/Rana Tharu, or English scope, must be evaluated as its own language/script/domain slice from a frozen, versioned, real-image eval pack with image bytes, labels, manifest rows, source-document/page/bbox provenance, split role, domain, ground-truth source and trust evidence, native-reader calibration when GT is machine-converted, row claim eligibility, script inventory coverage, and leakage declarations; every run must record exact model/checkpoint/config/dictionary or model card, prediction artifact, normalization profile, Unicode and grapheme-segmentation behavior, command, timestamp, machine, tool versions, and artifact hashes; every reported metric must be recomputable from canonical row artifacts as corpus micro edit-distance CER over both Unicode codepoints and UAX #29 extended grapheme clusters, not PaddleOCR norm_edit_dis or any per-line macro surrogate, with exact-line accuracy, matched and unmatched rows, missing-prediction accounting, confusion summaries, worst examples, reviewer artifacts, deterministic confidence intervals, and paired same-pack tests for model comparisons; recognizer oracle-crop results must remain separate from page-level OCR results, where page claims additionally require detection/crop lineage, reading-order and layout alignment, line-bbox/IoU or explicit reference matching, table/figure/text reconstruction metrics where relevant, historical/newspaper degradation and multi-column failure visibility, and visible coverage of missing documents or references; the harness must fail or hard-warn on missing or mutated files, duplicate IDs or image paths, stale hashes, unsupported or asymmetric normalization, low match rate, unrun or positive train/dev/test leakage, checkpoint selection on test data, synthetic or rendered data presented as real evidence, transliterated or machine-converted labels presented as human-verified evidence, human GT without review/agreement evidence, too-small or script-incomplete packs, aggregate scores that hide per-language/script/domain/GT failures, modern- benchmark averages that do not exercise the target community archives, and any hand-edited report whose JSON/TSV/CSV artifacts no longer recompute; a reviewer opening one output directory must be able to answer, without trusting prose, exactly what was scored, against which GT, how trustworthy that GT is, whether the test set was frozen and unseen for selection, which Unicode, script, layout, provenance, and archive coverage assumptions moved the number, how uncertain the number is, what failed, whose material is still unsupported, and whether the result is safe to cite in a paper.

Non-Negotiable Requirements

  • Real held-out eval packs are evidence objects, not folders of images. They need immutable manifests, hashes, provenance, split status, GT trust, domain labels, claim eligibility, script inventory coverage, and a datasheet-style summary.
  • Recognizer and page OCR are different tasks. Oracle-crop CER may support a recognizer claim only; page OCR claims need detector output, crop lineage, reading order, layout alignment, and missing-page/reference accounting.
  • Metrics must be recomputed from emitted row artifacts. Aggregate JSON is not a source of truth unless predictions.tsv, page-metrics.csv, alignment rows, and references recompute to the same values.
  • Unicode is part of the metric contract. The harness must record normalization, Unicode/grapheme implementation behavior, codepoint CER, grapheme CER, and missing required script inventory items.
  • Claim-facing reports need uncertainty and comparison discipline. Point CER alone is not enough; small packs need hard warnings, and model-vs-model claims need paired same-pack analysis.
  • Ground truth quality must be visible. Machine conversion and transliteration are scoreable, but they are not human-verified evidence unless a native-reader calibration artifact supports that upgrade.
  • The harness must be hostile to silent omissions. Missing predictions, duplicate rows, leaked train/dev examples, stale hashes, skipped pages, unsupported normalization, and stale Markdown should fail audit or block claim safety.