|
Download ocr-tech/docs/EVAL_HARNESS_RESEARCH_GOAL.md from prosperious/ocr-projects-archive: direct link, hf CLI and curl.
- Browser
- Download file 7.48 kB
-
https://huggingface.co/prosperious/ocr-projects-archive/resolve/main/ocr-tech/docs/EVAL_HARNESS_RESEARCH_GOAL.md
- Command line
-
hf download hf://prosperious/ocr-projects-archive/ocr-tech/docs/EVAL_HARNESS_RESEARCH_GOAL.md
-
curl -L -o EVAL_HARNESS_RESEARCH_GOAL.md https://huggingface.co/prosperious/ocr-projects-archive/resolve/main/ocr-tech/docs/EVAL_HARNESS_RESEARCH_GOAL.md
7.48 kB
| # Research-Backed Eval Harness Goal | |
| Updated: 2026-06-20 | |
| ## Sources Read | |
| - Web/paper search performed on 2026-06-20 before drafting this goal. | |
| - OCR-D quality assurance guidance: representative ground truth, realistic images, | |
| manual transcription, page/region localization, CER/WER definitions, reading-order | |
| and layout caveats, IoU/object-detection evaluation, and NFC normalization guidance: | |
| <https://ocr-d.de/en/spec/ocrd_eval.html> | |
| - Neudecker et al., "A survey of OCR evaluation tools and metrics" (HIP 2021): | |
| OCR evaluation is tool- and normalization-sensitive, and text accuracy depends on | |
| alignment, reading order, layout, and the exact edit-distance metric: | |
| <https://dl.acm.org/doi/10.1145/3476887.3476888> | |
| - Santos, "OCR Evaluation Tools for the 21st Century" (ComputEL 2019): OCR evaluation | |
| for modern and endangered-language work needs Unicode-capable character/word | |
| reporting rather than ASCII-era assumptions: | |
| <https://doi.org/10.33011/computel.v1i.345> | |
| - Unicode Standard Annex #29: extended grapheme clusters are the default | |
| programmatic approximation of user-perceived characters: | |
| <https://www.unicode.org/reports/tr29/> | |
| - PaddleOCR `RecMetric`: `norm_edit_dis` is one minus a per-sample average of | |
| normalized Levenshtein distance, not corpus micro CER: | |
| <https://raw.githubusercontent.com/PaddlePaddle/PaddleOCR/main/ppocr/metrics/rec_metric.py> | |
| - Leifert et al., "End-to-End Measure for Text Recognition" (ICDAR 2019): end-to-end | |
| text recognition evaluation must account for recognition plus segmentation/alignment, | |
| not just isolated text strings: | |
| <https://doi.org/10.1109/ICDAR.2019.00-16> | |
| - Beyene and Dancy, "A Survey of OCR Evaluation Methods and Metrics and the | |
| Invisibility of Historical Documents" (arXiv 2026 preprint): modern OCR and document | |
| understanding evaluations overfit to modern/institutional benchmark genres, often | |
| miss historical/community archive layout failures, and can make marginalized | |
| materials invisible even when headline CER/WER looks acceptable: | |
| <https://arxiv.org/html/2603.25761v1> | |
| - Springmann et al., "Automatic quality evaluation and (semi-) automatic improvement | |
| of OCR models for historical printings": GT is scarce, manual lines can change | |
| results materially, and confidence/proxy measures are not replacements for true GT: | |
| <https://arxiv.org/abs/1606.05157> | |
| - Gebru et al., "Datasheets for Datasets": datasets need explicit motivation, | |
| composition, collection, processing, intended use, and limitations: | |
| <https://arxiv.org/abs/1803.09010> | |
| - Mitchell et al., "Model Cards for Model Reporting": model releases need benchmarked | |
| performance, intended use, limitations, evaluation procedure, and per-condition | |
| reporting: | |
| <https://arxiv.org/abs/1810.03993> | |
| ## Dense Goal Paragraph | |
| Build a claim-grade OCR evaluation harness for the Nepal/diaspora recognizer suite | |
| whose default output makes an unsafe paper claim obvious before anyone can quote a | |
| number: every supported target, including the shared Devanagari languages | |
| Majhi, Magar, Sunuwar-Devanagari, Magahi, Bhojpuri, Thami, Wambule, and Darai plus | |
| native-script targets such as Limbu/Sirijonga, Sunuwar/Jenticha, Newa/Prachalit, | |
| Tirhuta, Lepcha, Kirat Rai, Gurung Khema, and any later Tamang, Bantawa/Rai, Kulung, | |
| Yakkha, Tharu/Rana Tharu, or English scope, must be evaluated as its own | |
| language/script/domain slice from a frozen, versioned, real-image eval pack with | |
| image bytes, labels, manifest rows, source-document/page/bbox provenance, split role, | |
| domain, ground-truth source and trust evidence, native-reader calibration when GT is | |
| machine-converted, row claim eligibility, script inventory coverage, and leakage | |
| declarations; every run must record exact model/checkpoint/config/dictionary or model | |
| card, prediction artifact, normalization profile, Unicode and grapheme-segmentation | |
| behavior, command, timestamp, machine, tool versions, and artifact hashes; every | |
| reported metric must be recomputable from canonical row artifacts as corpus micro | |
| edit-distance CER over both Unicode codepoints and UAX #29 extended grapheme clusters, | |
| not PaddleOCR `norm_edit_dis` or any per-line macro surrogate, with exact-line | |
| accuracy, matched and unmatched rows, missing-prediction accounting, confusion | |
| summaries, worst examples, reviewer artifacts, deterministic confidence intervals, | |
| and paired same-pack tests for model comparisons; recognizer oracle-crop results must | |
| remain separate from page-level OCR results, where page claims additionally require | |
| detection/crop lineage, reading-order and layout alignment, line-bbox/IoU or explicit | |
| reference matching, table/figure/text reconstruction metrics where relevant, | |
| historical/newspaper degradation and multi-column failure visibility, and visible | |
| coverage of missing documents or references; the harness must fail or hard-warn on | |
| missing or mutated files, duplicate IDs or image paths, stale hashes, unsupported or | |
| asymmetric normalization, low match rate, unrun or positive train/dev/test leakage, | |
| checkpoint selection on test data, synthetic or rendered data presented as real | |
| evidence, transliterated or machine-converted labels presented as human-verified | |
| evidence, human GT without review/agreement evidence, too-small or script-incomplete | |
| packs, aggregate scores that hide per-language/script/domain/GT failures, modern- | |
| benchmark averages that do not exercise the target community archives, and any | |
| hand-edited report whose JSON/TSV/CSV artifacts no longer recompute; a reviewer | |
| opening one output directory must be able to answer, without trusting prose, exactly | |
| what was scored, against which GT, how trustworthy that GT is, whether the test set was | |
| frozen and unseen for selection, which Unicode, script, layout, provenance, and archive | |
| coverage assumptions moved the number, how uncertain the number is, what failed, whose | |
| material is still unsupported, and whether the result is safe to cite in a paper. | |
| ## Non-Negotiable Requirements | |
| - Real held-out eval packs are evidence objects, not folders of images. They need | |
| immutable manifests, hashes, provenance, split status, GT trust, domain labels, | |
| claim eligibility, script inventory coverage, and a datasheet-style summary. | |
| - Recognizer and page OCR are different tasks. Oracle-crop CER may support a recognizer | |
| claim only; page OCR claims need detector output, crop lineage, reading order, | |
| layout alignment, and missing-page/reference accounting. | |
| - Metrics must be recomputed from emitted row artifacts. Aggregate JSON is not a | |
| source of truth unless `predictions.tsv`, `page-metrics.csv`, alignment rows, and | |
| references recompute to the same values. | |
| - Unicode is part of the metric contract. The harness must record normalization, | |
| Unicode/grapheme implementation behavior, codepoint CER, grapheme CER, and missing | |
| required script inventory items. | |
| - Claim-facing reports need uncertainty and comparison discipline. Point CER alone is | |
| not enough; small packs need hard warnings, and model-vs-model claims need paired | |
| same-pack analysis. | |
| - Ground truth quality must be visible. Machine conversion and transliteration are | |
| scoreable, but they are not human-verified evidence unless a native-reader | |
| calibration artifact supports that upgrade. | |
| - The harness must be hostile to silent omissions. Missing predictions, duplicate | |
| rows, leaked train/dev examples, stale hashes, skipped pages, unsupported | |
| normalization, and stale Markdown should fail audit or block claim safety. | |