|
Download OCR_INDRA/docs/INDRA_EXPERIMENTS.md from prosperious/ocr-projects-archive: direct link, hf CLI and curl.
- Browser
- Download file 14.9 kB
-
https://huggingface.co/prosperious/ocr-projects-archive/resolve/main/OCR_INDRA/docs/INDRA_EXPERIMENTS.md
- Command line
-
hf download hf://prosperious/ocr-projects-archive/OCR_INDRA/docs/INDRA_EXPERIMENTS.md
-
curl -L -o INDRA_EXPERIMENTS.md https://huggingface.co/prosperious/ocr-projects-archive/resolve/main/OCR_INDRA/docs/INDRA_EXPERIMENTS.md
14.9 kB
| # Indra × Nepali — Experiment Plan (A–E) | |
| Companion to `docs/INDRA_INVESTIGATION.md`. Everything runnable is designed | |
| around `tools/indra_eval/eval_indra.py` (works on CPU; the remote T4 simply | |
| makes batches fast). Remote-dependent sections are marked **[B]** and are | |
| blocked until SSH access to `khalti-t4` (202.51.0.7) is restored; every | |
| command to run there is written out so the set can be executed as-is. | |
| Fairness principle: **recognizer comparisons use identical line crops** | |
| (rec-lines mode); **pipeline comparisons use page-level text**. Both modes are | |
| built and smoke-tested (`results/smoke/`: CER 3.49% rec-only / 2.62% full on a | |
| synthetic page — indicational only). | |
| --- | |
| ## Experiment A — zero-shot Indra vs deployed PP-OCRv5 (do FIRST) [B with A.1 runnable] | |
| **Goal:** close "does the ~1.24 M-param Indra recognizer beat the deployed | |
| PP-OCRv5 Mobile Devanagari recognizer?" without training anything. | |
| **Materials (all remote):** frozen eval sets `~/ocr/ocr-eval-20260924*/` + | |
| `docs/DATA_LICENSING.md`, the deployed paddle pipeline | |
| (`~/ocr/ocr-deva-ft`, `tools/deva_rec_torch`, `tools/eval_metrics.py`), the | |
| CLONE of this repo + `models/indra-ocr-v3.1.1/` (copy `indra/` and `models/` | |
| up: ~46 MB — no dataset leaves the machine). | |
| **A.1 runnable from here already indep. of remote sets:** | |
| - `make_synth_page.py` renders pages at **matching font metrics** for a sanity | |
| envelope (no migration of datasets needed: it was built here, scored here). | |
| **A.2 (remote) recognizer head-to-head on identical crops:** | |
| 1. Export the deployed pipeline's GT line boxes for an eval set once: | |
| PP-OCRv5 det results on the set → boxes jsonl | |
| (`{"file", lines: [{i, text: GT, box}]}`). | |
| 2. Run our adapter on the same crops: | |
| ``` | |
| PYTHONPATH=$HOME/ocr/OCR_INDRA/indra:$HOME/ocr/dep python3 \ | |
| tools/indra_eval/eval_indra.py predict <page.jpg> \ | |
| --boxes <gt_boxes.jsonl> --jsonl results/indra_rec/<set>.jsonl | |
| ``` | |
| and score with the identical-crop baseline (PP-OCRv5 rec on the same cropped | |
| images saved as png), then `cer` both against GT. | |
| 3. Report: micro/macro CER + WER + per-line detail + conf histogram;(any | |
| `MIN_CONF`-dropped lines are reported separately (they hit the full-pipeline | |
| number but not rec-lines). | |
| 4. Also submit **full-pipeline** Indra (`predict` without `--boxes`) on the | |
| same set for the det+rec+order end-to-end number. | |
| **Pass/fail:** Indra rec-lines CER < deployed rec CER on ≥ 2 of 3 eval sets, | |
| with per-set mean confidence reported. Also record det recall (lines found / | |
| GT lines) for the full-pipeline run. | |
| ## Experiment B — fine-tune the recognizer (the supported path) | |
| **Goal:** Adapt `rec.onnx`'s weights to Nepali-specific distribution | |
| (conjunct-heavy printing, nukta, ₹, Devanagari digits in running text), retrain | |
| with `model.safetensors` as init, export an ONNX with the *identical* | |
| interface, drop-in replace. | |
| **Architecture requirements to reproduce byte-compatible behavior (from | |
| Investigation §1):** | |
| - module: stem Conv(1→128, k32×4, s32×4) + BN + HardSwish → mask Mul → | |
| 10 × [dw 9-k1 g128 pad4 → expand 1×1 128→256 → HS → SE(masked-mean→256→64→256 | |
| sigmoid gate) → project 1×1 256→128 → +residual → mask Mul] → head 1×1 | |
| 128→1678 → softmax → argmax. | |
| - Padding mask `(arange(W/4) < lengths).float()` with `lengths = | |
| ceil(widths/4)` (clip W/4); apply post-stem, at SE pools (mean over VALID | |
| steps only), and after each block's residual add. **The mask is functional, | |
| not cosmetic: it keeps SE statistics from being diluted by padding.** | |
| - Parameter mapping table: `model.safetensors` keys `rec/<name>` — | |
| `stem.weight` [128,1,32,4], `stem.bias` [128], `stem_bn.{weight,bias, | |
| running_mean,running_var}` [128], `blocks.<i>.dw.weight` [128,1,9] / | |
| `dw.bias` [128] (ONNX names `onnx::Conv_{369,370};372,373,…396,397`), | |
| `blocks.<i>.expand.{weight,bias}` [256,128], `blocks.<i>.se.fc1/fc2`, | |
| `blocks.<i>.project.{weight,bias}` [128,256], `head.weight` [1678,128,1], | |
| `head.bias` [1678]. | |
| - Loss: CTC (log_softmax over [B,T,1678]; blank id 0; input_lengths = lengths, | |
| target = `vocab.encode(canonical(text))` IDs list; NFC-normalize exactly as | |
| `vocab.py:42`). | |
| - Optimizer: AdamW, lr 3e-4 cosine, ~10–20k steps, batch 32 lines by | |
| width-bucket, crop 32px — the pipeline above runs on 1× A100/40 GB in | |
| minutes per epoch for our line counts ([B] exact data volume from | |
| `~/ocr/ocr-deva-ft/data`). | |
| **Unique risk (verified):** the trainer's crop augmentation recipe is closed; | |
| starting too aggressively (`rotate`, `tone`) would shift the input | |
| distribution away from `crop_line`'s exact ops. Mitigation: augmentations | |
| limited to elastic/scale jitter of ±5% consistent with `PAD_Y/PAD_X` bend, | |
| train/eval identical crop code (shared module imported from `rec_preproc.py` | |
| — the DOC that matters). | |
| **Export/eval:** torch → ONNX with same outputs (`ids,conf,lengths`) or a | |
| logits-only export + a 15-line Python decode; either way validate against | |
| `rec.onnx` outputs on 512 in-crops before scoring (**exactness test**: | |
| max |Δconf| < 1e-3 on identical inputs). | |
| **Verified offline 2026-10-20** (torch 2.14.1+cpu / onnxruntime 1.31.0, | |
| `tools/exp/verify_bundle.sh`): `model.safetensors` → IndraRec (1,236,366 | |
| tensors) → ONNX export matches the shipped `rec.onnx` with **0 id mismatches, | |
| 0 lengths mismatches, max |Δconf| 7.1e-6** on the 1,590-step probe battery; | |
| the post-train gate (torch .pt vs its own export) passes at the same tolerance; | |
| `run_rec.py` reproduces the recorded smoke CER (3.49 %) bit-for-bit. The | |
| exporter forces the legacy TorchScript path (`dynamo=False`) because torch ≥ 2.6 | |
| defaults to the torch.export exporter, which needs the `onnxscript` package. | |
| **Acceptance:** fine-tuned rec-lines CER < zero-shot rec-lines CER on all | |
| 3 frozen sets by ≥ 2 points relative, on Devanagari digit-heavy and | |
| conjunct-heavy slices specifically (those are where the smoke run showed | |
| weakness: `सङ्केत→सड्लेत`, `₹→र`, `ऋतु→तऋहतु`). | |
| ## Experiment C — fine-tune the line detector / order model | |
| **Verdict from Investigation §3: blocked by shipped-artifact reality** — | |
| `det.onnx`/`order.onnx` come with NO state dict and NO recipe. Two paths: | |
| 1. **Torch re-implementation from scratch** using the target specs that ARE | |
| in the repo (`det_post.encode_targets` → line core logit + top/bottom | |
| distance regression; `order_post.label_order` → 10 block classes + | |
| pairwise salience). Cost: reimplementing the backbone (which is unknown — | |
| the ONNX graphs would need the same node-by-node reverse archaeology as | |
| rec.onnx: feasible, we did it, but heavy) + a data pipeline + synth label | |
| generator. **Not recommended as the first move** — zero-shot + rec-only | |
| fine-tuning captures most value: the det model already detects 10/10 | |
| synthetic lines and machine-printed Devanagari well. | |
| 2. **Skip det/order entirely**; if detection quality on real scan material | |
| turns out to be the bottleneck in A-full, locally replace with our known | |
| corresponding PP-OCR det model and keep only the Indra rec — that is | |
| Experiment D. | |
| ## Experiment D — hybrid: keep PP-OCR det, swap rec → Indra | |
| **Goal:** combine a known-good Devanagari line finder (PP-OCRv5 det) with the | |
| Indra recognizer under identical crops, and vice-versa. Outputs line boxes | |
| jsonl exactly like A.2-step 1. This is a 1-day experiment **with zero training** | |
| and makes the fair-comparison claim airtight: if `PP-det + Indra-rec` beats | |
| `PP-det + PP-rec` on the same crops, the recognizer itself is the win; if not, | |
| Indra's advantage (if any) lives elsewhere (orientation/order/layout). | |
| ## Experiment E — productionization / distill for WSL CPU box | |
| **Goal:** a deployable artifact for the 8-core/3 GB (or T4) runtime: | |
| - Indra package supports `rec.int8.onnx` automatically (`pipeline.py:195–200`) | |
| — quantize the rec model once (no retraining), then re-run Experiment A with | |
| the int8 model: acceptance: CER regression ≤ +0.3 pts absolute and | |
| ≥ 2× throughput. | |
| - **Measured local guidance (2026-10-20, WSL CPU, threads=4, smoke crops; | |
| indicative, not a benchmark):** ORT *dynamic* (weights-only) quantization is | |
| a **trap** for this Conv+SE model — 0.69–0.27× fp32 throughput (ConvInteger + | |
| per-activation quant/dequant dominates) though greedy ids stay identical and | |
| max |Δconf| = 0.077. *Static* QDQ uint8 (per-tensor, MinMax, 64 calib crops) | |
| built 3.5× smaller (4.98 → 1.43 MB) and ran **~1.4× faster** than fp32 — on a | |
| stronger box the gap should widen; the ≥2× acceptance must be re-measured | |
| there. Tool: `tools/exp/e_int8_quantize.py` (static, calibration through the | |
| shared `crop_line`; `--check-smoke` reports id/conf drift; writes a JSON report). | |
| Use real line crops for calibration (the local test used noise → id drift 56 | |
| steps on the smoke page; real calibration should cut that). | |
| - Latency budget on our box is meaningless (WSL starved) — measure only on | |
| the T4: lines/s at long-side 1200/1600/2048 (det scales with page size). | |
| - Memory: with `enable_cpu_mem_arena=False` (already set in the package), | |
| line-batched rec is flat per page. | |
| --- | |
| ## §11 Data plan for fine-tuning (Experiment B) | |
| - Source: the same frozen train corpus that produced the deployed fine-tuned | |
| PP-OCR model (`~/ocr/ocr-deva-ft/data` — [B] inventory is on remote; | |
| `docs/DEVANAGARI_RESOURCES_INVENTORY.md` documents it). **No dataset is | |
| copied off the remote machine** — training artifacts/results files transfer, | |
| not eval sets. | |
| - Emphasis slices for Nepali: conjunct-heavy book/print pages, Devanagari | |
| digit blocks, ₹-annotated amounts, मध्य-मस useful — determined by the | |
| per-line CER profile of A-zero (error concentration, not vibes). | |
| - Same-crop exactness: train the augmented pages through the exact | |
| `crop_line`/`pad_batch` code (import from the cloned package), so zero-shot | |
| and fine-tuned are measured through one code path. | |
| - Leakage (§12): any *train* page rendered by the deployed pipeline must be | |
| excluded from ALL three frozen eval sets; confirm set boundaries by | |
| re-running `tools/eval_metrics.py`'s pairing logic [B]. | |
| ## §12 Leakage review [B] | |
| 1. **Set purity:** frozen eval sets must not overlap the rec fine-tune | |
| corpus. Verify by page-hash (sha256 of rendered page images), not | |
| filenames ([B] run on remote). | |
| 2. **Synth contamination:** if synthetic rendered pages were part of | |
| `ocr-deva-ft` training and the eval sets contain natural scans only, no | |
| leakage; confirm the eval-set provenance doc. Rendered-by-us | |
| (`make_synth_page.py`) pages are NEVER used for training or reporting. | |
| 3. **Adaptation leakage:** Experiment D swaps components inside a pipeline | |
| built on eval sets — acceptable for comparison because GT boxes come from | |
| the DETECTOR of the deployed pipeline on EVAL images only, and neither | |
| component is trained on them. | |
| 4. **Font leakage:** if fonts in the training corpus are also in the eval | |
| sets' print source (same family), CER estimates are optimistic for BOTH | |
| systems equally — acceptable for A/B ranking, not for absolute claims; | |
| state it in the report. | |
| ## §13 Licensing matrix [B for dataset rows] | |
| | Item | License | Fine-tune allowed? | Deploy allowed? | Notes | | |
| |---|---|---|---|---| | |
| | Indra weights v3.1.1 | AI PUBS Open RAIL-M (modified) | **YES, explicit** (§II ¶36) | YES (restrictions = Attachment A misuse list; internal OCR unaffected) | Derivative weights → same license + notice on redistribution | | |
| | `indra_ocr` code | Apache-2.0 | n/a | YES | | | |
| | PaddleOCR (baseline) | Apache-2.0 | YES (we already fine-tune it) | YES | current deployed | | |
| | PP-OCRv5 Mobile Devanagari weights | Apache-2.0 | YES | YES | deployed | | |
| | Nepali train corpus | per `docs/DATA_LICENSING.md` [B] | pending review | — | no removal of datasets from remote | | |
| | Frozen eval sets | per `docs/DATA_LICENSING.md` [B] | evaluation only | — | | | |
| | Noto Sans Devanagari (testrender only) | OFL 1.1 | YES (fonts embedding fine) | test asset only | `tools/indra_eval/testdata/` | | |
| ## §14 Deployment comparison | |
| | Axis | Indra v3.1.1 (rec-only swap / full) | PP-OCRv5 Mobile Devanagari deployed | | |
| |---|---|---| | |
| | Recognizer params | **1.236 M** (verified) | rec ~similar class (Mobile range; exact count from remote bench) | | |
| | Runtime | onnxruntime CPU (already installed here) | paddle (remote) | | |
| | Extra machinery needed | **none** — single ONNX + meta.json vocab JSON | paddle + FineTuned assets + converter glue | | |
| | Latency (lines/s, T4 CPU) | measure (A/D/E) | measure | | |
| | Deterministic CPU memory | flat batch (mask) | paddle per-page buffers | | |
| | Order/reading model | included (order.onnx; optional) | not used for plain OCR flow | | |
| | Honest staffing: fine-tune effort | reimplement trainer (~1–2 days) | already exists for Paddle | | |
| | Dependency risk | pinned HF revision + local files | paddle version drift | | |
| ## §17 Decision tree | |
| 1. **Run A (zero-shot rec-lines + full) on the frozen sets.** | |
| - If Indra rec CER < deployed rec CER **on 2 of 3 sets**: proceed to the | |
| hybrid experiment D (PP-det + Indra-rec); if D holds the win, plan a | |
| **rec-swap deployment trial** alongside the existing PP pipeline(1 week). | |
| - If Indra rec CER ≥ deployed rec CER: record the gap per slice. If the | |
| gap is train-material (conjuncts/₹/digits): run **B** (fine-tune rec | |
| from safetensors, ~10–20k steps); expected to close a | |
| distribution-gap-shaped loss; decide by ΔCER on the eval sets. | |
| 2. **Regardless of A's outcome,** the graceful fallback stands: bringing Indra | |
| recognizer into our PP-OCR det pipeline is a **drop-in component** (same | |
| crops→ink-map contract), so nothing about A/B/D commits us. | |
| 3. **Det/order fine-tuning (C) stays out of scope** until det-on-real-scan | |
| failure is demonstrated in A-full (`det recall < line recall of deployed`). | |
| 4. **Never** touch: datasets on the remote, the deployed pipeline's recipes, | |
| its eval scripts, or `~/.ssh`/`~/.bashrc`. All experiments run in a | |
| separate directory (`~/ocr/OCR_INDRA/`), copying only code/weights in. | |
| ## §18 Status summary | |
| - [V] runnable here end-to-end: smoke chain (synth page → rec-only → full → | |
| CER), all inputs/outputs/paths in Investigation §8. | |
| - [B] everything requiring `khalti-t4`: A.2 (frozen sets), B data, C skip | |
| decision input, licensing rows, E throughput on deployment hardware. | |
| - [V] fine-tune toolkit verified offline 2026-10-20 (`tools/exp/verify_bundle.sh`: | |
| export exactness pass; mechanics training + post-train gate pass; A/B | |
| orchestrations complete on a local stand-in set); int8 static/dynamic | |
| behaviour probed (dynamic = slower trap, static = ~1.4× faster, 3.5× smaller). | |
| - Open: WP det recall numbers, paddle latency comparison, int8 CER on real | |
| calibration + deployment-box throughput. | |