prosperious's picture
Add 1000 files (ocr-tech/tests/test_audit_limbu_book_missed_line_review_coverage.py ...)
d82db76 verified
|
Raw History Blame Contribute Delete
23.8 kB
# Research State: Nepal Language Pack OCR
Updated: 2026-08-09
## Current Goal
The end goal is an OCR system covering the multiple languages of the Nepal pack,
built out of Paddle, **with Limbu and Newar (Nepal Bhasa) as the first two focus
languages** (user steer, 2026-07-04). The system should accept real book/page images
or PDFs for any language in the pack and output editable Unicode text with page,
line, crop, confidence, and provenance evidence preserved. Limbu stays the proven
template — it has the strongest recognizer and the deepest harvest. Newar is the
deliberate second: it has its own native script (Prachalit), real cultural demand,
and is currently the weakest trained recognizer, so it exercises every part of the
method that Limbu already proved. Every stage that was built Limbu-specific must
generalize: harvest, silver-label extraction, recognizer training, script routing,
page/book OCR, and eval. The product is the full pipeline: page
detection, script-aware routing to per-script recognizers, target-region and
furniture filtering, reading order, assembly, Unicode/JSON/Markdown export, and
review artifacts for hard lines. YOLO line detection stays a diagnostic/probe;
Paddle-first is the production story.
## The Language Pack
Two inventories exist and they are not the same thing:
- The registry `languages/nepal-v1.json` tracks 18 target languages (17 Nepal + English)
with scripts, fonts, dictionaries, and corpus status.
- The Gorkhapatra Naya Nepal archive publishes far more: the 2026-07-04 harvest found
**50 normalized languages (53 raw labels) in 400 articles over ~7 months** (see the
harvest section). Labels include all registry languages plus Chamling Rai, Bahing,
Jirel, Chepang, Baram, Tajpuriya, Kumal, Doteli, Bajhangi, Dandeldhureli, Hyolmo,
Mugal, Marwadi, Urdu, Sanskrit, Awadhi, Kisan, Achhami, Bote, Hayu, Meche, Lhomi,
Uraon, Bhote, Danuwar, Majhi, Magahi, Bajjika, Khash Jumli, and more.
Label variants must be normalized before counting (थारु/थारू, याक्ख/याक्खा,
ह्योलमो/ह्योल्मो, लिम्बु/लिम्बू, चाम्लिङ vs चाम्लिङ राई) —
`tools/build_gorkhapatra_language_pack_inventory.py` owns that normalization and the
label→registry mapping.
Per-language recognizer state (grapheme CER, honest micro metric, from the re-grounded
2026-06-18 scorecard in `docs/RECOGNIZER_GOAL_PROGRESS.md`):
- **Limbu/Sirijonga 0.041 real-print** (champion:
`runs/limbu-real-heavy-gork-cdc-replay-v1-lr1e6-8ep-normedit-g6-l4-ppocrv5-2026-06-23`,
cp CER 0.011 on the frozen 255-line Gorkhapatra silver eval).
- **8 Devanagari languages 0.075–0.129 real-print** (Majhi, Magar, Sunuwar-Deva, Magahi,
Bhojpuri, Thami, Wambule, Darai) from one shared Devanagari recognizer.
- **Sunuwar (Jenticha) 0.121 real-crop** after the same-font retrain (0.342 synth-only).
- **Tirhuta 0.343 and Newa 0.613 synthetic-only** — weak, transliteration-fed, honest
fails; Newa needs real printed-register Nepal Lipi, not more pixels.
- **Kirat Rai (70 real crops) and Lepcha (132 real crops)** blocked on legacy-font
converter coverage outside their eval editions; **Gurung Khema** has no real print
anywhere (revival script) — synthetic-only by nature, claim-scoped accordingly.
The near-term pack claim should therefore be script-family shaped: one Sirijonga
recognizer, one shared Devanagari recognizer covering the many Devanagari-written
languages, plus per-script experts (Jenticha, Kirat Rai, Lepcha, Prachalit/Newa,
Tirhuta, Gurung Khema, Ol Chiki, Tibetan-script) admitted one by one as real eval
crops exist.
**Focus order (2026-07-04 steer): Limbu first, Newar second.** Newar state: the
Prachalit recognizer is the weakest in the pack (grapheme CER 0.613 on its own
synthetic eval; transliteration-fed training failed; real printed-register Nepal
Lipi is the known lever, and only 2 Unicode Newa fonts exist). The Devanagari-Newar
route moved fast on 2026-07-04: 6,800 real silver lines harvested from 20
Gorkhapatra pages; `newar` added to the registry (Devanagari primary + Prachalit
secondary; audit passes); a frozen by-date pack built at
`outputs/newar-devanagari-silver-pack-v1/` (train 4,096 / dev 1,528 / test 1,139
lines; test dates 2082-07-25 and 2083-01-05; zero date/text overlap independently
verified; pack sha256
`1fa7b23b6c18701857aea686fd55f3075e104d224730896f6ee35ba25e7ccb85`); and the
**first real-print Devanagari-Newar baseline** was scored locally (CPU export
route, `tools/score_recognizer_pack.py`, per `docs/LOCAL_RECOGNIZER_SCORING.md`):
the shared deva-v2 recognizer, never trained on Newar, reads the frozen test at
**codepoint CER 0.0622, grapheme CER 0.1312, exact-line 0.158** (n=1,139 silver
lines — machine-converted labels, diagnostic, not claim-grade). The same-day
warm-start fine-tune (**newar-ft-v1**: deva-v2 base, LR 2e-5 cosine, 60 epochs,
batch 64, selection on the all-batch2 dev split, best_epoch 46; G6 L4 instance
`ashish-experiments`, ~68 min; artifacts + train log at
`outputs/aws-newar-deva-ft-v1/`) scores the same frozen test at **codepoint CER
0.0465, grapheme CER 0.0955, exact-line 0.245** — a 25/27% relative CER drop and
+55% exact-line over the deva-v2 baseline, scored through the identical local
CPU path. **Newar's grapheme CER now sits inside deva-v2's own-language range
(0.075–0.129): the second focus language reached Devanagari-pack parity from
zero in one day.** Same claim boundary: silver machine-converted labels,
diagnostic, not claim-grade until native review. Next Newar lever: native
Prachalit (real printed-register data), and page-level Newar eval for the
region-filtering track.
## Source Rule
For Gorkhapatra/Naya Nepal language pages, the Naya Nepal page is discovery metadata
only. Use `https://gorkhapatraonline.com/publications/naya-nepal` to find the language,
date, and article, but do not use the blog/card image for OCR, training, or evaluation
when the full paper exists. The authoritative source is the matching Gorkhapatra epaper
PDF from the date-range endpoint. The correct pair is the rendered PDF page image plus
the text layer extracted from that same PDF page, with PDF URL/hash, page number,
rendered image hash, extraction command/version, font evidence, and converter/map hash
recorded. For Limbu/Sirijonga the text layer is Namdhinggo-style legacy converted with
`src/ocrtech/limbu_legacy.py` + `data/mappings/limbu-legacy/Limbu.map`. The same
pattern — classify the text layer, then build an explicit converter only when the font
requires it — replicates to every other language. Manual transcription is the fallback
for no usable text layer, not the default.
## Getting More Gorkhapatra Pages (the specified harvest recipe)
All numbers below were verified live on 2026-07-04.
**Archive size.** The Naya Nepal publication archive paginates at
`https://gorkhapatraonline.com/publications/naya-nepal?page=N` with 5 articles per
page and runs to page ~352 before drying up: **~1,750 language-page articles total**
(about 2 per day going back years). The 60-page crawl already yielded 300 items across
53 labels spanning BS 2082-10-18 → 2083-03-19; deeper pages reach correspondingly
older dates.
**Epaper endpoint behavior (the critical constraint).** The date-range endpoint
`https://epaper.gorkhapatraonline.com/epaper-list?pdfdaterange1=A&pdfdaterange2=B&slug=gorkhapatra`
**caps results at 30 PDFs per query and ignores `page=N` pagination**. A wide window
silently drops most dates — the 2026-07-04 six-window discovery pass aligned only
111/300 items for exactly this reason, with unmatched dates interleaved among matched
ones. **Seven-day windows return complete daily coverage (7/7 verified).** Always
harvest with weekly windows.
**The recipe** (one command; add one `--epaper-url` per week to cover the publication
date span being harvested):
```bash
env PYTHONPATH=src uv run --python 3.11 --with pymupdf \
ocrtech audit-gorkhapatra-language-pages \
--out outputs/gorkhapatra-language-pages-full-<date> \
--publication-url https://gorkhapatraonline.com/publications/naya-nepal \
--publication-pages 80 \
--epaper-url 'https://epaper.gorkhapatraonline.com/epaper-list?pdfdaterange1=2026-06-28&pdfdaterange2=2026-07-04&slug=gorkhapatra' \
--epaper-url '...one per week, going back...' \
--download-pdfs \
--render-pages \
--timeout-seconds 180
```
The command aligns every publication card (all languages at once — alignment is
language-agnostic) to its same-date PDF, downloads each PDF once (cached by URL),
searches pages for the language-page markers (`भाषा पृष्ठ` and its legacy-encoding
form `efiff k[i7`), and renders hit pages at 2×. The 2026-07-04 run with 23 weekly
windows (2026-01-25 → 2026-07-04, publication-pages 80) completed with
**400 publication items, 195 aligned dates, 394 rendered language-page hits, zero
warnings** at `outputs/gorkhapatra-language-pages-full-2026-07-04/`. Batch 2
(2025-08-03 → 2026-01-24, 25 weekly windows, publication-pages 160) added 398 more
items: **798 items, 294 aligned dates, 589 further page hits**, at
`outputs/gorkhapatra-language-pages-full-batch2-2026-07-04/`. The merged normalized
inventory (`tools/build_gorkhapatra_language_pack_inventory.py`) shows **56
languages, 983 rendered language-page hits, most languages with ~20–27 pages
spanning 14 months**, at `outputs/language-pack-inventory-merged-2026-07-04/`. For
comparison, the entire previous state of the project had 8 usable pages for one
language. Batches cost ~2–2.5 GB each; the remaining archive depth (back to at least
March 2024 — verified the epaper endpoint serves complete weeks there) is another
~2× and stays month-batchable with the same recipe.
**Scaling to the full archive.** Full depth is ~700+ unique dates ≈ 35 GB of PDFs.
Do not do it in one run: batch by month (4–5 weekly windows + the matching
publication-page depth), run `extract-gorkhapatra-language-page-pdf-text` per batch,
then keep rendered pages, extracted text, and hashes; PDFs can be pruned after
extraction because URL+SHA-256 make them re-fetchable. The remote 2 TB volume
(`cdjk@100.110.148.43:/Volumes/TRANSCEND`) is the right home for a full-depth mirror.
**Per-language downstream.** After harvest, run per language:
```bash
env PYTHONPATH=src uv run --python 3.11 --with pymupdf \
ocrtech extract-gorkhapatra-language-page-pdf-text \
<pack-manifest.jsonl> \
--candidate-language <label> \
--target-font <font> \
--out outputs/<language>-pdf-text-<date>
```
then classify each page's text layer (native Unicode / legacy font / Devanagari
transcription / Latin / none) and only then decide converter vs manual transcription.
**Font/text-layer facts already established** (30-page, 15-language extraction at
`outputs/gorkhapatra-multilingual-language-pages-current-2026-06-24-pdf-text/`):
- The body font of non-Limbu language pages is **`nayanepal` — a legacy 8-bit
Preeti-family encoding, not Unicode** (8,845 of 11,791 spans; the text layer reads
`k[i7 ;+of]hs` where the page shows पृष्ठ संयोजक). **The converter already
exists**: `nepal_ttf2utf.convert_devanagari(font="nayanepal")` in the sibling repo
`/Users/cdjk/github/llm/nepal-ttf2utf` (Preeti base map + nayanepal extension
glyphs + NFC, ~97–99% clean on real Gorkhapatra pages), already used by
`tools/build_gorkhapatra_devanagari_real.py` for the 79k-row real Devanagari
corpus. The remaining leverage is wiring it **per language**: run it over each
harvested language page and emit per-language silver line packs with the same
faithfulness gate (clean-Unicode allow-set, `conversion_gap` accounting, leakage
checks) that the Devanagari pack builder already implements. The same sibling repo
also hosts the limbu, kiratrai, lepcha, and sunuwar legacy converters.
- Tibetan-script fonts (`TibetanMachine`, `MonlamUniOuChan5`, `Qomolangma-*`) appear on
Mugal and related pages, but **every such span has empty extracted text (verified
2026-07-04 across all 2026-06-24 samples)** — Tibetan-script pages are visual-only,
like the known Limbu `2082-11-12` date. They need OCR bootstrap or manual
transcription; no converter is possible.
- `Gorkhapatra-*`, `Nagarik-*`, `TimesNewRoman*`, `MyriadPro-*` are masthead/furniture
fonts — useful as non-target filters for target-region selection.
- `NamdhinggoSILL` appears only on Limbu dates (the existing Limbu path).
## What We Built (Limbu template, still true)
The Gorkhapatra alignment path: publication-page discovery, epaper-date matching, PDF
download/rendering, page-marker search, PDF-native span/font extraction, review
sheets, reference templates, and dashboard visualization. The expanded Limbu harvest
has 9 discovered dates, 8 usable PDF-text-layer pages, 2,441 line rows, 2,430 usable
silver labels (`outputs/limbu-gorkhapatra-pdf-backed-silver-2026-07-03-v1/`).
The Limbu recognizer training/eval path: 48,507-row combined pack; real-heavy
fine-tune champion at codepoint CER 0.011336 / grapheme 0.024736 / exact-line 0.729 on
the frozen 255-line silver eval. Synthetic hardening lessons: targeted+Unicode packs
regressed; v3 dictionary warm-start catastrophically failed (~0.98 CER); keep the
real-heavy v2 champion; future gains come from real/book-domain harvesting or a
deliberate rehead, not blind synth rows.
The page/book OCR loop: local + HF Space (`spaces/limbu-ocr/`, deployed private at
`https://huggingface.co/spaces/ampixa/limbu-ocr`) with line OCR, Paddle Page OCR,
Paddle Book OCR, image-line fallback, runtime checks, annotated pages, and crops.
Page-level diagnostics on the 8-page silver pack: Rows LTR + y/conf70 filter scores
page cp CER 0.414330, line F1 0.963009, matched-line cp CER 0.026039, order accuracy
0.993176. Recognition on matched lines is strong; **full-page CER is dominated by page
reconstruction, target-region selection, and furniture filtering** — that is the
pipeline problem to solve, and it is language-independent.
Supervised line-detector infrastructure from CDC PDF geometry (287 pages, 26,160
annotations, mAP50 ≈ 0.921 on PDF-rendered val) — evidence the detector path works,
not a solved product.
A claim-grade eval harness (versioned packs, hashes, provenance, normalization
profiles, cp/grapheme CER, CIs, paired comparisons, leakage manifests) — parked as a
guardrail; do not let it eat work blocks that should go to the pipeline.
## Current Working State
The strongest component is the Sirijonga line recognizer. The weakest component is
end-to-end page/book reconstruction: target-region selection, furniture filtering,
line preservation, and ordering. Of the three Limbu-locked code spots, two were
generalized on 2026-07-04 (Codex worker, tests green): `_script_ratio` in
`src/ocrtech/engines.py` is now driven by the shared `SCRIPT_CODEPOINT_RANGES` table
covering all pack scripts (devanagari, limbu/sirijonga, latin, sunuwar/jenticha,
lepcha, kirat_rai, newa/prachalit, tirhuta, gurung_khema, ol_chiki, tibetan, with
aliases), and the image-line filter in `src/ocrtech/image_lines.py` accepts any
script key from that same table. The third spot fell the same day: the
`src/ocrtech/limbu_runtime.py` preflight now takes `required_script` (default
`limbu`, validated against the shared table; CLI `--required-script`), so a router
card for any pack script can be preflighted. The engine classes themselves (`CandidateEngine`, `PaddleOcrEngine`,
`ScriptSelectCompositeEngine`, `build_image_line_document`, `structure.py` reading
order) are already parameterized — the rest of the generalization is registry-driven
wiring, not a rewrite.
## Apple MPS Recognizer Training Evidence (2026-08-09)
Stock PaddlePaddle is not the immediate Apple-GPU route. A direct PaddlePaddle
3.3.1 probe on `k2` exposed no custom devices and rejected `mps`, `metal`, and
`custom_device:metal`. Paddle's official `PaddleCustomDevice` MPS backend exists,
but is currently a version-0.0.1 prototype with too small an operator surface to
establish PP-OCRv5 training.
The practical path is an Apache-2.0 vendored subset of
`frotms/PaddleOCR2Pytorch` pinned at
`c05307fbd61e575d55f0fef0334026c7536aec76`: PPLCNetV3 scale 0.95 plus the
SVTR/CTC branch of the PP-OCRv5 mobile recognizer. Dynamic pooling preserves the
recognition width (`T = input_width / 4`), and a localized contiguity fix before
the SVTR encoder's `conv3` makes the model safe for the tested MPS runtimes. The
model and log-softmax run on MPS; `CTCLoss` runs on CPU and autograd returns
finite gradients to MPS for the optimizer update. The NRTR auxiliary branch is
frozen and not part of this CTC loss, so this is not a complete reproduction of
PaddleOCR's multi-loss training recipe.
One-step evidence is green on both Macs. The end-to-end `mbp2` run used Python
3.11.15 and Torch 2.13.0: `[4,3,48,160] -> [4,40,7]`, loss 29.883989, 582 finite
MPS gradient tensors, and a non-zero optimizer update across synthetic Latin,
Devanagari, and Limbu samples. Direct width-320 forward/backward/update checks
also passed on `mbp2` and on `k2` with Torch 2.12. These cold one-step timings
are diagnostics, not benchmarks.
The checkpoint boundary is strict: this backend writes PyTorch checkpoints,
not Paddle `.pdparams`; conversion requires explicit tensor mapping and frozen
inference-parity tests. `mbp2` and `k2` should run independent experiments with
distinct seeds and run directories, not multi-host MPS DDP. The remaining proof
is a multi-epoch real-manifest run with resume/reload, memory observation, and
canonical held-out CER. See `docs/PADDLEOCR_MPS.md` for exact smoke artifacts,
hashes, compatibility limits, and host guidance.
## The Paddle Pipeline (whole pipeline, not just recognizers)
Target architecture, all stages Paddle-first and language-pack-parameterized by the
per-language pipeline config **`configs/nepal-language-pack-pipeline-v1.json`**
(50 languages: label, registry id, scripts, recognizer artifact, dictionary,
converter, text-layer class, harvest counts — generated 2026-07-04 from the
inventory):
1. **Capture/prep** — existing `prepare-limbu-capture` path, already
language-agnostic in behavior; rename/rescope.
2. **Detection** — Paddle text detection as production default; YOLO line detector and
morphology image-line boxes stay as diagnostics/rescue passes.
3. **Script identification & routing** — generalize `_script_ratio` to a
Unicode-block-driven scorer over the pack's script list (Devanagari, Limbu,
Jenticha/Sunuwar, Lepcha, Kirat Rai, Newa, Tirhuta, Gurung Khema, Ol Chiki,
Tibetan, Latin), then route each line to the script's recognizer from the registry.
4. **Recognition** — per-script PP-OCRv5 recognizers (existing runs) with per-script
dictionaries and input-width profiles (Sirijonga width-480 lesson).
5. **Target-region & furniture filtering** — non-oracle region selection using layout
plus font/language signals; furniture fonts and masthead geometry as negative
evidence. This is the main open research item; score against the Rows LTR +
y/conf70 baseline.
6. **Reading order & assembly** — `auto_layout` selection (wide-column RTL vs rows)
per page, conservative assembly stays diagnostic.
7. **Export** — Unicode text, JSON with provenance, Markdown; review queues +
dashboards per page (existing Limbu artifact shapes, renamed pack-wide).
8. **Eval guardrail** — frozen real-page packs per language: page CER/WER, line
P/R/F1, reading order, blank/extra/missed lines, review burden.
PP-StructureV3 and PaddleOCR-VL remain the documented fallback/teacher direction for
hard layouts, per `docs/NEPAL_MULTILINGUAL_OCR_GOAL.md`.
## Next Work
1. ~~Finish the 2026-07-04 full harvest and inventory~~ **Done 2026-07-04**: 394
language-page hits, 50 languages, inventory manifests at
`outputs/language-pack-inventory-full-2026-07-04/`.
2. ~~Per-language silver packs via the existing `nayanepal` converter~~ **Done
2026-07-04** (`tools/build_language_pack_silver_line_packs.py`, tests green): the
batch-1 run produced **116,287 kept silver lines across 50 languages, zero
errors**, with per-line provenance (PDF hash, page, bbox, raw legacy bytes,
converted text, image hash) at `outputs/language-pack-silver-lines-2026-07-04/`;
spot-checked conversions read as clean, language-plausible Devanagari. The
batch-2 run over the older 589 page hits kept **183,324 more lines (49 languages,
zero errors)** at `outputs/language-pack-silver-lines-batch2-2026-07-04/` —
**299,611 per-language-attributed silver lines total** across the two batches
(~2.4 GB of crops). Run the tool with
`uv run --python 3.11 --with pymupdf --with npttf2utf` — `nepal_ttf2utf` needs the
`npttf2utf` base-map package at runtime, and without it every span fails
(cleanly, with per-span error records, but kept=0). **Honest caveat**: lines
are page-attributed, not language-verified — a language page also carries Nepali
furniture/boxes, and pages whose body is not nayanepal Devanagari (Urdu, the
Tibetan-script pages) yield mostly furniture lines. A language-ID/lexicon filter
before per-language recognizer training is the follow-up gate.
3. **Newar track (second focus language)**: add the Newar registry entry
(Devanagari + Prachalit rows); build the Devanagari-written Newar recognizer
eval/train split from the 6,800 harvested silver lines (by-date split, frozen
test); keep native Prachalit as the separate longer arc — real printed-register
Nepal Lipi harvesting, not more transliteration, is the lever.
4. **Classify Tibetan-script pages** (Hyolmo/Mugal/Bhote) and decide converter vs
native Unicode.
5. ~~Generalize the pipeline wiring~~ **Done 2026-07-04**: all three Limbu-locked
spots generalized (shared script table, filter keys, `--required-script`
preflight), tests green; `configs/nepal-language-pack-pipeline-v1.json` holds the
per-language wiring.
6. **Target-region/furniture filtering stage** for Paddle page/book OCR, scored
against the current baseline on the 8-page Limbu pack and at least one
Devanagari-language pack (Newar is the natural candidate given the focus order).
7. Only after page reconstruction improves, export high-quality real/silver line
crops back into recognizer training, per language.
8. Month-batch the archive back-harvest toward the full ~1,750-article depth, storing
the mirror on the remote 2 TB volume.
## Mistakes To Avoid
Do not use Naya Nepal blog images as training/eval evidence when the PDF exists. Do
not use wide epaper date windows — the endpoint silently caps at 30 PDFs per query;
weekly windows only. Do not treat unnormalized Gorkhapatra labels as distinct
languages (थारु vs थारू). Do not treat the `nayanepal` text layer as Unicode — it is
legacy Preeti-family and needs the converter before any silver label is real. Do not
report Paddle `norm_edit_dis` as CER; real CER comes from `tools/eval_metrics.py`
micro CER over canonical predictions. Do not treat synthetic data as claim evidence,
and label Gurung Khema results as synthetic-by-nature. Do not promote the
targeted+Unicode checkpoint or v3 dictionary warm-starts. Do not spend long blocks on
generic eval-harness work unless it directly improves the book/page OCR loop. Do not
frame video-to-page work as OCR scope; reduce video to stills first. Do not present
page OCR as solved: matched-line recognition is strong, but full-page CER is still
dominated by reconstruction and region selection.