Add 1000 files (ocr-tech/tests/test_audit_limbu_book_missed_line_review_coverage.py ...)
d82db76 verified |
Download ocr-tech-mps-integration/RESEARCH.md from prosperious/ocr-projects-archive: direct link, hf CLI and curl.
- Browser
- Download file 23.8 kB
-
https://huggingface.co/prosperious/ocr-projects-archive/resolve/main/ocr-tech-mps-integration/RESEARCH.md
- Command line
-
hf download hf://prosperious/ocr-projects-archive/ocr-tech-mps-integration/RESEARCH.md
-
curl -L -o RESEARCH.md https://huggingface.co/prosperious/ocr-projects-archive/resolve/main/ocr-tech-mps-integration/RESEARCH.md
23.8 kB
| # Research State: Nepal Language Pack OCR | |
| Updated: 2026-08-09 | |
| ## Current Goal | |
| The end goal is an OCR system covering the multiple languages of the Nepal pack, | |
| built out of Paddle, **with Limbu and Newar (Nepal Bhasa) as the first two focus | |
| languages** (user steer, 2026-07-04). The system should accept real book/page images | |
| or PDFs for any language in the pack and output editable Unicode text with page, | |
| line, crop, confidence, and provenance evidence preserved. Limbu stays the proven | |
| template — it has the strongest recognizer and the deepest harvest. Newar is the | |
| deliberate second: it has its own native script (Prachalit), real cultural demand, | |
| and is currently the weakest trained recognizer, so it exercises every part of the | |
| method that Limbu already proved. Every stage that was built Limbu-specific must | |
| generalize: harvest, silver-label extraction, recognizer training, script routing, | |
| page/book OCR, and eval. The product is the full pipeline: page | |
| detection, script-aware routing to per-script recognizers, target-region and | |
| furniture filtering, reading order, assembly, Unicode/JSON/Markdown export, and | |
| review artifacts for hard lines. YOLO line detection stays a diagnostic/probe; | |
| Paddle-first is the production story. | |
| ## The Language Pack | |
| Two inventories exist and they are not the same thing: | |
| - The registry `languages/nepal-v1.json` tracks 18 target languages (17 Nepal + English) | |
| with scripts, fonts, dictionaries, and corpus status. | |
| - The Gorkhapatra Naya Nepal archive publishes far more: the 2026-07-04 harvest found | |
| **50 normalized languages (53 raw labels) in 400 articles over ~7 months** (see the | |
| harvest section). Labels include all registry languages plus Chamling Rai, Bahing, | |
| Jirel, Chepang, Baram, Tajpuriya, Kumal, Doteli, Bajhangi, Dandeldhureli, Hyolmo, | |
| Mugal, Marwadi, Urdu, Sanskrit, Awadhi, Kisan, Achhami, Bote, Hayu, Meche, Lhomi, | |
| Uraon, Bhote, Danuwar, Majhi, Magahi, Bajjika, Khash Jumli, and more. | |
| Label variants must be normalized before counting (थारु/थारू, याक्ख/याक्खा, | |
| ह्योलमो/ह्योल्मो, लिम्बु/लिम्बू, चाम्लिङ vs चाम्लिङ राई) — | |
| `tools/build_gorkhapatra_language_pack_inventory.py` owns that normalization and the | |
| label→registry mapping. | |
| Per-language recognizer state (grapheme CER, honest micro metric, from the re-grounded | |
| 2026-06-18 scorecard in `docs/RECOGNIZER_GOAL_PROGRESS.md`): | |
| - **Limbu/Sirijonga 0.041 real-print** (champion: | |
| `runs/limbu-real-heavy-gork-cdc-replay-v1-lr1e6-8ep-normedit-g6-l4-ppocrv5-2026-06-23`, | |
| cp CER 0.011 on the frozen 255-line Gorkhapatra silver eval). | |
| - **8 Devanagari languages 0.075–0.129 real-print** (Majhi, Magar, Sunuwar-Deva, Magahi, | |
| Bhojpuri, Thami, Wambule, Darai) from one shared Devanagari recognizer. | |
| - **Sunuwar (Jenticha) 0.121 real-crop** after the same-font retrain (0.342 synth-only). | |
| - **Tirhuta 0.343 and Newa 0.613 synthetic-only** — weak, transliteration-fed, honest | |
| fails; Newa needs real printed-register Nepal Lipi, not more pixels. | |
| - **Kirat Rai (70 real crops) and Lepcha (132 real crops)** blocked on legacy-font | |
| converter coverage outside their eval editions; **Gurung Khema** has no real print | |
| anywhere (revival script) — synthetic-only by nature, claim-scoped accordingly. | |
| The near-term pack claim should therefore be script-family shaped: one Sirijonga | |
| recognizer, one shared Devanagari recognizer covering the many Devanagari-written | |
| languages, plus per-script experts (Jenticha, Kirat Rai, Lepcha, Prachalit/Newa, | |
| Tirhuta, Gurung Khema, Ol Chiki, Tibetan-script) admitted one by one as real eval | |
| crops exist. | |
| **Focus order (2026-07-04 steer): Limbu first, Newar second.** Newar state: the | |
| Prachalit recognizer is the weakest in the pack (grapheme CER 0.613 on its own | |
| synthetic eval; transliteration-fed training failed; real printed-register Nepal | |
| Lipi is the known lever, and only 2 Unicode Newa fonts exist). The Devanagari-Newar | |
| route moved fast on 2026-07-04: 6,800 real silver lines harvested from 20 | |
| Gorkhapatra pages; `newar` added to the registry (Devanagari primary + Prachalit | |
| secondary; audit passes); a frozen by-date pack built at | |
| `outputs/newar-devanagari-silver-pack-v1/` (train 4,096 / dev 1,528 / test 1,139 | |
| lines; test dates 2082-07-25 and 2083-01-05; zero date/text overlap independently | |
| verified; pack sha256 | |
| `1fa7b23b6c18701857aea686fd55f3075e104d224730896f6ee35ba25e7ccb85`); and the | |
| **first real-print Devanagari-Newar baseline** was scored locally (CPU export | |
| route, `tools/score_recognizer_pack.py`, per `docs/LOCAL_RECOGNIZER_SCORING.md`): | |
| the shared deva-v2 recognizer, never trained on Newar, reads the frozen test at | |
| **codepoint CER 0.0622, grapheme CER 0.1312, exact-line 0.158** (n=1,139 silver | |
| lines — machine-converted labels, diagnostic, not claim-grade). The same-day | |
| warm-start fine-tune (**newar-ft-v1**: deva-v2 base, LR 2e-5 cosine, 60 epochs, | |
| batch 64, selection on the all-batch2 dev split, best_epoch 46; G6 L4 instance | |
| `ashish-experiments`, ~68 min; artifacts + train log at | |
| `outputs/aws-newar-deva-ft-v1/`) scores the same frozen test at **codepoint CER | |
| 0.0465, grapheme CER 0.0955, exact-line 0.245** — a 25/27% relative CER drop and | |
| +55% exact-line over the deva-v2 baseline, scored through the identical local | |
| CPU path. **Newar's grapheme CER now sits inside deva-v2's own-language range | |
| (0.075–0.129): the second focus language reached Devanagari-pack parity from | |
| zero in one day.** Same claim boundary: silver machine-converted labels, | |
| diagnostic, not claim-grade until native review. Next Newar lever: native | |
| Prachalit (real printed-register data), and page-level Newar eval for the | |
| region-filtering track. | |
| ## Source Rule | |
| For Gorkhapatra/Naya Nepal language pages, the Naya Nepal page is discovery metadata | |
| only. Use `https://gorkhapatraonline.com/publications/naya-nepal` to find the language, | |
| date, and article, but do not use the blog/card image for OCR, training, or evaluation | |
| when the full paper exists. The authoritative source is the matching Gorkhapatra epaper | |
| PDF from the date-range endpoint. The correct pair is the rendered PDF page image plus | |
| the text layer extracted from that same PDF page, with PDF URL/hash, page number, | |
| rendered image hash, extraction command/version, font evidence, and converter/map hash | |
| recorded. For Limbu/Sirijonga the text layer is Namdhinggo-style legacy converted with | |
| `src/ocrtech/limbu_legacy.py` + `data/mappings/limbu-legacy/Limbu.map`. The same | |
| pattern — classify the text layer, then build an explicit converter only when the font | |
| requires it — replicates to every other language. Manual transcription is the fallback | |
| for no usable text layer, not the default. | |
| ## Getting More Gorkhapatra Pages (the specified harvest recipe) | |
| All numbers below were verified live on 2026-07-04. | |
| **Archive size.** The Naya Nepal publication archive paginates at | |
| `https://gorkhapatraonline.com/publications/naya-nepal?page=N` with 5 articles per | |
| page and runs to page ~352 before drying up: **~1,750 language-page articles total** | |
| (about 2 per day going back years). The 60-page crawl already yielded 300 items across | |
| 53 labels spanning BS 2082-10-18 → 2083-03-19; deeper pages reach correspondingly | |
| older dates. | |
| **Epaper endpoint behavior (the critical constraint).** The date-range endpoint | |
| `https://epaper.gorkhapatraonline.com/epaper-list?pdfdaterange1=A&pdfdaterange2=B&slug=gorkhapatra` | |
| **caps results at 30 PDFs per query and ignores `page=N` pagination**. A wide window | |
| silently drops most dates — the 2026-07-04 six-window discovery pass aligned only | |
| 111/300 items for exactly this reason, with unmatched dates interleaved among matched | |
| ones. **Seven-day windows return complete daily coverage (7/7 verified).** Always | |
| harvest with weekly windows. | |
| **The recipe** (one command; add one `--epaper-url` per week to cover the publication | |
| date span being harvested): | |
| ```bash | |
| env PYTHONPATH=src uv run --python 3.11 --with pymupdf \ | |
| ocrtech audit-gorkhapatra-language-pages \ | |
| --out outputs/gorkhapatra-language-pages-full-<date> \ | |
| --publication-url https://gorkhapatraonline.com/publications/naya-nepal \ | |
| --publication-pages 80 \ | |
| --epaper-url 'https://epaper.gorkhapatraonline.com/epaper-list?pdfdaterange1=2026-06-28&pdfdaterange2=2026-07-04&slug=gorkhapatra' \ | |
| --epaper-url '...one per week, going back...' \ | |
| --download-pdfs \ | |
| --render-pages \ | |
| --timeout-seconds 180 | |
| ``` | |
| The command aligns every publication card (all languages at once — alignment is | |
| language-agnostic) to its same-date PDF, downloads each PDF once (cached by URL), | |
| searches pages for the language-page markers (`भाषा पृष्ठ` and its legacy-encoding | |
| form `efiff k[i7`), and renders hit pages at 2×. The 2026-07-04 run with 23 weekly | |
| windows (2026-01-25 → 2026-07-04, publication-pages 80) completed with | |
| **400 publication items, 195 aligned dates, 394 rendered language-page hits, zero | |
| warnings** at `outputs/gorkhapatra-language-pages-full-2026-07-04/`. Batch 2 | |
| (2025-08-03 → 2026-01-24, 25 weekly windows, publication-pages 160) added 398 more | |
| items: **798 items, 294 aligned dates, 589 further page hits**, at | |
| `outputs/gorkhapatra-language-pages-full-batch2-2026-07-04/`. The merged normalized | |
| inventory (`tools/build_gorkhapatra_language_pack_inventory.py`) shows **56 | |
| languages, 983 rendered language-page hits, most languages with ~20–27 pages | |
| spanning 14 months**, at `outputs/language-pack-inventory-merged-2026-07-04/`. For | |
| comparison, the entire previous state of the project had 8 usable pages for one | |
| language. Batches cost ~2–2.5 GB each; the remaining archive depth (back to at least | |
| March 2024 — verified the epaper endpoint serves complete weeks there) is another | |
| ~2× and stays month-batchable with the same recipe. | |
| **Scaling to the full archive.** Full depth is ~700+ unique dates ≈ 35 GB of PDFs. | |
| Do not do it in one run: batch by month (4–5 weekly windows + the matching | |
| publication-page depth), run `extract-gorkhapatra-language-page-pdf-text` per batch, | |
| then keep rendered pages, extracted text, and hashes; PDFs can be pruned after | |
| extraction because URL+SHA-256 make them re-fetchable. The remote 2 TB volume | |
| (`cdjk@100.110.148.43:/Volumes/TRANSCEND`) is the right home for a full-depth mirror. | |
| **Per-language downstream.** After harvest, run per language: | |
| ```bash | |
| env PYTHONPATH=src uv run --python 3.11 --with pymupdf \ | |
| ocrtech extract-gorkhapatra-language-page-pdf-text \ | |
| <pack-manifest.jsonl> \ | |
| --candidate-language <label> \ | |
| --target-font <font> \ | |
| --out outputs/<language>-pdf-text-<date> | |
| ``` | |
| then classify each page's text layer (native Unicode / legacy font / Devanagari | |
| transcription / Latin / none) and only then decide converter vs manual transcription. | |
| **Font/text-layer facts already established** (30-page, 15-language extraction at | |
| `outputs/gorkhapatra-multilingual-language-pages-current-2026-06-24-pdf-text/`): | |
| - The body font of non-Limbu language pages is **`nayanepal` — a legacy 8-bit | |
| Preeti-family encoding, not Unicode** (8,845 of 11,791 spans; the text layer reads | |
| `k[i7 ;+of]hs` where the page shows पृष्ठ संयोजक). **The converter already | |
| exists**: `nepal_ttf2utf.convert_devanagari(font="nayanepal")` in the sibling repo | |
| `/Users/cdjk/github/llm/nepal-ttf2utf` (Preeti base map + nayanepal extension | |
| glyphs + NFC, ~97–99% clean on real Gorkhapatra pages), already used by | |
| `tools/build_gorkhapatra_devanagari_real.py` for the 79k-row real Devanagari | |
| corpus. The remaining leverage is wiring it **per language**: run it over each | |
| harvested language page and emit per-language silver line packs with the same | |
| faithfulness gate (clean-Unicode allow-set, `conversion_gap` accounting, leakage | |
| checks) that the Devanagari pack builder already implements. The same sibling repo | |
| also hosts the limbu, kiratrai, lepcha, and sunuwar legacy converters. | |
| - Tibetan-script fonts (`TibetanMachine`, `MonlamUniOuChan5`, `Qomolangma-*`) appear on | |
| Mugal and related pages, but **every such span has empty extracted text (verified | |
| 2026-07-04 across all 2026-06-24 samples)** — Tibetan-script pages are visual-only, | |
| like the known Limbu `2082-11-12` date. They need OCR bootstrap or manual | |
| transcription; no converter is possible. | |
| - `Gorkhapatra-*`, `Nagarik-*`, `TimesNewRoman*`, `MyriadPro-*` are masthead/furniture | |
| fonts — useful as non-target filters for target-region selection. | |
| - `NamdhinggoSILL` appears only on Limbu dates (the existing Limbu path). | |
| ## What We Built (Limbu template, still true) | |
| The Gorkhapatra alignment path: publication-page discovery, epaper-date matching, PDF | |
| download/rendering, page-marker search, PDF-native span/font extraction, review | |
| sheets, reference templates, and dashboard visualization. The expanded Limbu harvest | |
| has 9 discovered dates, 8 usable PDF-text-layer pages, 2,441 line rows, 2,430 usable | |
| silver labels (`outputs/limbu-gorkhapatra-pdf-backed-silver-2026-07-03-v1/`). | |
| The Limbu recognizer training/eval path: 48,507-row combined pack; real-heavy | |
| fine-tune champion at codepoint CER 0.011336 / grapheme 0.024736 / exact-line 0.729 on | |
| the frozen 255-line silver eval. Synthetic hardening lessons: targeted+Unicode packs | |
| regressed; v3 dictionary warm-start catastrophically failed (~0.98 CER); keep the | |
| real-heavy v2 champion; future gains come from real/book-domain harvesting or a | |
| deliberate rehead, not blind synth rows. | |
| The page/book OCR loop: local + HF Space (`spaces/limbu-ocr/`, deployed private at | |
| `https://huggingface.co/spaces/ampixa/limbu-ocr`) with line OCR, Paddle Page OCR, | |
| Paddle Book OCR, image-line fallback, runtime checks, annotated pages, and crops. | |
| Page-level diagnostics on the 8-page silver pack: Rows LTR + y/conf70 filter scores | |
| page cp CER 0.414330, line F1 0.963009, matched-line cp CER 0.026039, order accuracy | |
| 0.993176. Recognition on matched lines is strong; **full-page CER is dominated by page | |
| reconstruction, target-region selection, and furniture filtering** — that is the | |
| pipeline problem to solve, and it is language-independent. | |
| Supervised line-detector infrastructure from CDC PDF geometry (287 pages, 26,160 | |
| annotations, mAP50 ≈ 0.921 on PDF-rendered val) — evidence the detector path works, | |
| not a solved product. | |
| A claim-grade eval harness (versioned packs, hashes, provenance, normalization | |
| profiles, cp/grapheme CER, CIs, paired comparisons, leakage manifests) — parked as a | |
| guardrail; do not let it eat work blocks that should go to the pipeline. | |
| ## Current Working State | |
| The strongest component is the Sirijonga line recognizer. The weakest component is | |
| end-to-end page/book reconstruction: target-region selection, furniture filtering, | |
| line preservation, and ordering. Of the three Limbu-locked code spots, two were | |
| generalized on 2026-07-04 (Codex worker, tests green): `_script_ratio` in | |
| `src/ocrtech/engines.py` is now driven by the shared `SCRIPT_CODEPOINT_RANGES` table | |
| covering all pack scripts (devanagari, limbu/sirijonga, latin, sunuwar/jenticha, | |
| lepcha, kirat_rai, newa/prachalit, tirhuta, gurung_khema, ol_chiki, tibetan, with | |
| aliases), and the image-line filter in `src/ocrtech/image_lines.py` accepts any | |
| script key from that same table. The third spot fell the same day: the | |
| `src/ocrtech/limbu_runtime.py` preflight now takes `required_script` (default | |
| `limbu`, validated against the shared table; CLI `--required-script`), so a router | |
| card for any pack script can be preflighted. The engine classes themselves (`CandidateEngine`, `PaddleOcrEngine`, | |
| `ScriptSelectCompositeEngine`, `build_image_line_document`, `structure.py` reading | |
| order) are already parameterized — the rest of the generalization is registry-driven | |
| wiring, not a rewrite. | |
| ## Apple MPS Recognizer Training Evidence (2026-08-09) | |
| Stock PaddlePaddle is not the immediate Apple-GPU route. A direct PaddlePaddle | |
| 3.3.1 probe on `k2` exposed no custom devices and rejected `mps`, `metal`, and | |
| `custom_device:metal`. Paddle's official `PaddleCustomDevice` MPS backend exists, | |
| but is currently a version-0.0.1 prototype with too small an operator surface to | |
| establish PP-OCRv5 training. | |
| The practical path is an Apache-2.0 vendored subset of | |
| `frotms/PaddleOCR2Pytorch` pinned at | |
| `c05307fbd61e575d55f0fef0334026c7536aec76`: PPLCNetV3 scale 0.95 plus the | |
| SVTR/CTC branch of the PP-OCRv5 mobile recognizer. Dynamic pooling preserves the | |
| recognition width (`T = input_width / 4`), and a localized contiguity fix before | |
| the SVTR encoder's `conv3` makes the model safe for the tested MPS runtimes. The | |
| model and log-softmax run on MPS; `CTCLoss` runs on CPU and autograd returns | |
| finite gradients to MPS for the optimizer update. The NRTR auxiliary branch is | |
| frozen and not part of this CTC loss, so this is not a complete reproduction of | |
| PaddleOCR's multi-loss training recipe. | |
| One-step evidence is green on both Macs. The end-to-end `mbp2` run used Python | |
| 3.11.15 and Torch 2.13.0: `[4,3,48,160] -> [4,40,7]`, loss 29.883989, 582 finite | |
| MPS gradient tensors, and a non-zero optimizer update across synthetic Latin, | |
| Devanagari, and Limbu samples. Direct width-320 forward/backward/update checks | |
| also passed on `mbp2` and on `k2` with Torch 2.12. These cold one-step timings | |
| are diagnostics, not benchmarks. | |
| The checkpoint boundary is strict: this backend writes PyTorch checkpoints, | |
| not Paddle `.pdparams`; conversion requires explicit tensor mapping and frozen | |
| inference-parity tests. `mbp2` and `k2` should run independent experiments with | |
| distinct seeds and run directories, not multi-host MPS DDP. The remaining proof | |
| is a multi-epoch real-manifest run with resume/reload, memory observation, and | |
| canonical held-out CER. See `docs/PADDLEOCR_MPS.md` for exact smoke artifacts, | |
| hashes, compatibility limits, and host guidance. | |
| ## The Paddle Pipeline (whole pipeline, not just recognizers) | |
| Target architecture, all stages Paddle-first and language-pack-parameterized by the | |
| per-language pipeline config **`configs/nepal-language-pack-pipeline-v1.json`** | |
| (50 languages: label, registry id, scripts, recognizer artifact, dictionary, | |
| converter, text-layer class, harvest counts — generated 2026-07-04 from the | |
| inventory): | |
| 1. **Capture/prep** — existing `prepare-limbu-capture` path, already | |
| language-agnostic in behavior; rename/rescope. | |
| 2. **Detection** — Paddle text detection as production default; YOLO line detector and | |
| morphology image-line boxes stay as diagnostics/rescue passes. | |
| 3. **Script identification & routing** — generalize `_script_ratio` to a | |
| Unicode-block-driven scorer over the pack's script list (Devanagari, Limbu, | |
| Jenticha/Sunuwar, Lepcha, Kirat Rai, Newa, Tirhuta, Gurung Khema, Ol Chiki, | |
| Tibetan, Latin), then route each line to the script's recognizer from the registry. | |
| 4. **Recognition** — per-script PP-OCRv5 recognizers (existing runs) with per-script | |
| dictionaries and input-width profiles (Sirijonga width-480 lesson). | |
| 5. **Target-region & furniture filtering** — non-oracle region selection using layout | |
| plus font/language signals; furniture fonts and masthead geometry as negative | |
| evidence. This is the main open research item; score against the Rows LTR + | |
| y/conf70 baseline. | |
| 6. **Reading order & assembly** — `auto_layout` selection (wide-column RTL vs rows) | |
| per page, conservative assembly stays diagnostic. | |
| 7. **Export** — Unicode text, JSON with provenance, Markdown; review queues + | |
| dashboards per page (existing Limbu artifact shapes, renamed pack-wide). | |
| 8. **Eval guardrail** — frozen real-page packs per language: page CER/WER, line | |
| P/R/F1, reading order, blank/extra/missed lines, review burden. | |
| PP-StructureV3 and PaddleOCR-VL remain the documented fallback/teacher direction for | |
| hard layouts, per `docs/NEPAL_MULTILINGUAL_OCR_GOAL.md`. | |
| ## Next Work | |
| 1. ~~Finish the 2026-07-04 full harvest and inventory~~ **Done 2026-07-04**: 394 | |
| language-page hits, 50 languages, inventory manifests at | |
| `outputs/language-pack-inventory-full-2026-07-04/`. | |
| 2. ~~Per-language silver packs via the existing `nayanepal` converter~~ **Done | |
| 2026-07-04** (`tools/build_language_pack_silver_line_packs.py`, tests green): the | |
| batch-1 run produced **116,287 kept silver lines across 50 languages, zero | |
| errors**, with per-line provenance (PDF hash, page, bbox, raw legacy bytes, | |
| converted text, image hash) at `outputs/language-pack-silver-lines-2026-07-04/`; | |
| spot-checked conversions read as clean, language-plausible Devanagari. The | |
| batch-2 run over the older 589 page hits kept **183,324 more lines (49 languages, | |
| zero errors)** at `outputs/language-pack-silver-lines-batch2-2026-07-04/` — | |
| **299,611 per-language-attributed silver lines total** across the two batches | |
| (~2.4 GB of crops). Run the tool with | |
| `uv run --python 3.11 --with pymupdf --with npttf2utf` — `nepal_ttf2utf` needs the | |
| `npttf2utf` base-map package at runtime, and without it every span fails | |
| (cleanly, with per-span error records, but kept=0). **Honest caveat**: lines | |
| are page-attributed, not language-verified — a language page also carries Nepali | |
| furniture/boxes, and pages whose body is not nayanepal Devanagari (Urdu, the | |
| Tibetan-script pages) yield mostly furniture lines. A language-ID/lexicon filter | |
| before per-language recognizer training is the follow-up gate. | |
| 3. **Newar track (second focus language)**: add the Newar registry entry | |
| (Devanagari + Prachalit rows); build the Devanagari-written Newar recognizer | |
| eval/train split from the 6,800 harvested silver lines (by-date split, frozen | |
| test); keep native Prachalit as the separate longer arc — real printed-register | |
| Nepal Lipi harvesting, not more transliteration, is the lever. | |
| 4. **Classify Tibetan-script pages** (Hyolmo/Mugal/Bhote) and decide converter vs | |
| native Unicode. | |
| 5. ~~Generalize the pipeline wiring~~ **Done 2026-07-04**: all three Limbu-locked | |
| spots generalized (shared script table, filter keys, `--required-script` | |
| preflight), tests green; `configs/nepal-language-pack-pipeline-v1.json` holds the | |
| per-language wiring. | |
| 6. **Target-region/furniture filtering stage** for Paddle page/book OCR, scored | |
| against the current baseline on the 8-page Limbu pack and at least one | |
| Devanagari-language pack (Newar is the natural candidate given the focus order). | |
| 7. Only after page reconstruction improves, export high-quality real/silver line | |
| crops back into recognizer training, per language. | |
| 8. Month-batch the archive back-harvest toward the full ~1,750-article depth, storing | |
| the mirror on the remote 2 TB volume. | |
| ## Mistakes To Avoid | |
| Do not use Naya Nepal blog images as training/eval evidence when the PDF exists. Do | |
| not use wide epaper date windows — the endpoint silently caps at 30 PDFs per query; | |
| weekly windows only. Do not treat unnormalized Gorkhapatra labels as distinct | |
| languages (थारु vs थारू). Do not treat the `nayanepal` text layer as Unicode — it is | |
| legacy Preeti-family and needs the converter before any silver label is real. Do not | |
| report Paddle `norm_edit_dis` as CER; real CER comes from `tools/eval_metrics.py` | |
| micro CER over canonical predictions. Do not treat synthetic data as claim evidence, | |
| and label Gurung Khema results as synthetic-by-nature. Do not promote the | |
| targeted+Unicode checkpoint or v3 dictionary warm-starts. Do not spend long blocks on | |
| generic eval-harness work unless it directly improves the book/page OCR loop. Do not | |
| frame video-to-page work as OCR scope; reduce video to stills first. Do not present | |
| page OCR as solved: matched-line recognition is strong, but full-page CER is still | |
| dominated by reconstruction and region selection. | |