File size: 14,908 Bytes
f9bc0c1 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 | # Indra × Nepali — Experiment Plan (A–E)
Companion to `docs/INDRA_INVESTIGATION.md`. Everything runnable is designed
around `tools/indra_eval/eval_indra.py` (works on CPU; the remote T4 simply
makes batches fast). Remote-dependent sections are marked **[B]** and are
blocked until SSH access to `khalti-t4` (202.51.0.7) is restored; every
command to run there is written out so the set can be executed as-is.
Fairness principle: **recognizer comparisons use identical line crops**
(rec-lines mode); **pipeline comparisons use page-level text**. Both modes are
built and smoke-tested (`results/smoke/`: CER 3.49% rec-only / 2.62% full on a
synthetic page — indicational only).
---
## Experiment A — zero-shot Indra vs deployed PP-OCRv5 (do FIRST) [B with A.1 runnable]
**Goal:** close "does the ~1.24 M-param Indra recognizer beat the deployed
PP-OCRv5 Mobile Devanagari recognizer?" without training anything.
**Materials (all remote):** frozen eval sets `~/ocr/ocr-eval-20260924*/` +
`docs/DATA_LICENSING.md`, the deployed paddle pipeline
(`~/ocr/ocr-deva-ft`, `tools/deva_rec_torch`, `tools/eval_metrics.py`), the
CLONE of this repo + `models/indra-ocr-v3.1.1/` (copy `indra/` and `models/`
up: ~46 MB — no dataset leaves the machine).
**A.1 runnable from here already indep. of remote sets:**
- `make_synth_page.py` renders pages at **matching font metrics** for a sanity
envelope (no migration of datasets needed: it was built here, scored here).
**A.2 (remote) recognizer head-to-head on identical crops:**
1. Export the deployed pipeline's GT line boxes for an eval set once:
PP-OCRv5 det results on the set → boxes jsonl
(`{"file", lines: [{i, text: GT, box}]}`).
2. Run our adapter on the same crops:
```
PYTHONPATH=$HOME/ocr/OCR_INDRA/indra:$HOME/ocr/dep python3 \
tools/indra_eval/eval_indra.py predict <page.jpg> \
--boxes <gt_boxes.jsonl> --jsonl results/indra_rec/<set>.jsonl
```
and score with the identical-crop baseline (PP-OCRv5 rec on the same cropped
images saved as png), then `cer` both against GT.
3. Report: micro/macro CER + WER + per-line detail + conf histogram;(any
`MIN_CONF`-dropped lines are reported separately (they hit the full-pipeline
number but not rec-lines).
4. Also submit **full-pipeline** Indra (`predict` without `--boxes`) on the
same set for the det+rec+order end-to-end number.
**Pass/fail:** Indra rec-lines CER < deployed rec CER on ≥ 2 of 3 eval sets,
with per-set mean confidence reported. Also record det recall (lines found /
GT lines) for the full-pipeline run.
## Experiment B — fine-tune the recognizer (the supported path)
**Goal:** Adapt `rec.onnx`'s weights to Nepali-specific distribution
(conjunct-heavy printing, nukta, ₹, Devanagari digits in running text), retrain
with `model.safetensors` as init, export an ONNX with the *identical*
interface, drop-in replace.
**Architecture requirements to reproduce byte-compatible behavior (from
Investigation §1):**
- module: stem Conv(1→128, k32×4, s32×4) + BN + HardSwish → mask Mul →
10 × [dw 9-k1 g128 pad4 → expand 1×1 128→256 → HS → SE(masked-mean→256→64→256
sigmoid gate) → project 1×1 256→128 → +residual → mask Mul] → head 1×1
128→1678 → softmax → argmax.
- Padding mask `(arange(W/4) < lengths).float()` with `lengths =
ceil(widths/4)` (clip W/4); apply post-stem, at SE pools (mean over VALID
steps only), and after each block's residual add. **The mask is functional,
not cosmetic: it keeps SE statistics from being diluted by padding.**
- Parameter mapping table: `model.safetensors` keys `rec/<name>` —
`stem.weight` [128,1,32,4], `stem.bias` [128], `stem_bn.{weight,bias,
running_mean,running_var}` [128], `blocks.<i>.dw.weight` [128,1,9] /
`dw.bias` [128] (ONNX names `onnx::Conv_{369,370};372,373,…396,397`),
`blocks.<i>.expand.{weight,bias}` [256,128], `blocks.<i>.se.fc1/fc2`,
`blocks.<i>.project.{weight,bias}` [128,256], `head.weight` [1678,128,1],
`head.bias` [1678].
- Loss: CTC (log_softmax over [B,T,1678]; blank id 0; input_lengths = lengths,
target = `vocab.encode(canonical(text))` IDs list; NFC-normalize exactly as
`vocab.py:42`).
- Optimizer: AdamW, lr 3e-4 cosine, ~10–20k steps, batch 32 lines by
width-bucket, crop 32px — the pipeline above runs on 1× A100/40 GB in
minutes per epoch for our line counts ([B] exact data volume from
`~/ocr/ocr-deva-ft/data`).
**Unique risk (verified):** the trainer's crop augmentation recipe is closed;
starting too aggressively (`rotate`, `tone`) would shift the input
distribution away from `crop_line`'s exact ops. Mitigation: augmentations
limited to elastic/scale jitter of ±5% consistent with `PAD_Y/PAD_X` bend,
train/eval identical crop code (shared module imported from `rec_preproc.py`
— the DOC that matters).
**Export/eval:** torch → ONNX with same outputs (`ids,conf,lengths`) or a
logits-only export + a 15-line Python decode; either way validate against
`rec.onnx` outputs on 512 in-crops before scoring (**exactness test**:
max |Δconf| < 1e-3 on identical inputs).
**Verified offline 2026-10-20** (torch 2.14.1+cpu / onnxruntime 1.31.0,
`tools/exp/verify_bundle.sh`): `model.safetensors` → IndraRec (1,236,366
tensors) → ONNX export matches the shipped `rec.onnx` with **0 id mismatches,
0 lengths mismatches, max |Δconf| 7.1e-6** on the 1,590-step probe battery;
the post-train gate (torch .pt vs its own export) passes at the same tolerance;
`run_rec.py` reproduces the recorded smoke CER (3.49 %) bit-for-bit. The
exporter forces the legacy TorchScript path (`dynamo=False`) because torch ≥ 2.6
defaults to the torch.export exporter, which needs the `onnxscript` package.
**Acceptance:** fine-tuned rec-lines CER < zero-shot rec-lines CER on all
3 frozen sets by ≥ 2 points relative, on Devanagari digit-heavy and
conjunct-heavy slices specifically (those are where the smoke run showed
weakness: `सङ्केत→सड्लेत`, `₹→र`, `ऋतु→तऋहतु`).
## Experiment C — fine-tune the line detector / order model
**Verdict from Investigation §3: blocked by shipped-artifact reality** —
`det.onnx`/`order.onnx` come with NO state dict and NO recipe. Two paths:
1. **Torch re-implementation from scratch** using the target specs that ARE
in the repo (`det_post.encode_targets` → line core logit + top/bottom
distance regression; `order_post.label_order` → 10 block classes +
pairwise salience). Cost: reimplementing the backbone (which is unknown —
the ONNX graphs would need the same node-by-node reverse archaeology as
rec.onnx: feasible, we did it, but heavy) + a data pipeline + synth label
generator. **Not recommended as the first move** — zero-shot + rec-only
fine-tuning captures most value: the det model already detects 10/10
synthetic lines and machine-printed Devanagari well.
2. **Skip det/order entirely**; if detection quality on real scan material
turns out to be the bottleneck in A-full, locally replace with our known
corresponding PP-OCR det model and keep only the Indra rec — that is
Experiment D.
## Experiment D — hybrid: keep PP-OCR det, swap rec → Indra
**Goal:** combine a known-good Devanagari line finder (PP-OCRv5 det) with the
Indra recognizer under identical crops, and vice-versa. Outputs line boxes
jsonl exactly like A.2-step 1. This is a 1-day experiment **with zero training**
and makes the fair-comparison claim airtight: if `PP-det + Indra-rec` beats
`PP-det + PP-rec` on the same crops, the recognizer itself is the win; if not,
Indra's advantage (if any) lives elsewhere (orientation/order/layout).
## Experiment E — productionization / distill for WSL CPU box
**Goal:** a deployable artifact for the 8-core/3 GB (or T4) runtime:
- Indra package supports `rec.int8.onnx` automatically (`pipeline.py:195–200`)
— quantize the rec model once (no retraining), then re-run Experiment A with
the int8 model: acceptance: CER regression ≤ +0.3 pts absolute and
≥ 2× throughput.
- **Measured local guidance (2026-10-20, WSL CPU, threads=4, smoke crops;
indicative, not a benchmark):** ORT *dynamic* (weights-only) quantization is
a **trap** for this Conv+SE model — 0.69–0.27× fp32 throughput (ConvInteger +
per-activation quant/dequant dominates) though greedy ids stay identical and
max |Δconf| = 0.077. *Static* QDQ uint8 (per-tensor, MinMax, 64 calib crops)
built 3.5× smaller (4.98 → 1.43 MB) and ran **~1.4× faster** than fp32 — on a
stronger box the gap should widen; the ≥2× acceptance must be re-measured
there. Tool: `tools/exp/e_int8_quantize.py` (static, calibration through the
shared `crop_line`; `--check-smoke` reports id/conf drift; writes a JSON report).
Use real line crops for calibration (the local test used noise → id drift 56
steps on the smoke page; real calibration should cut that).
- Latency budget on our box is meaningless (WSL starved) — measure only on
the T4: lines/s at long-side 1200/1600/2048 (det scales with page size).
- Memory: with `enable_cpu_mem_arena=False` (already set in the package),
line-batched rec is flat per page.
---
## §11 Data plan for fine-tuning (Experiment B)
- Source: the same frozen train corpus that produced the deployed fine-tuned
PP-OCR model (`~/ocr/ocr-deva-ft/data` — [B] inventory is on remote;
`docs/DEVANAGARI_RESOURCES_INVENTORY.md` documents it). **No dataset is
copied off the remote machine** — training artifacts/results files transfer,
not eval sets.
- Emphasis slices for Nepali: conjunct-heavy book/print pages, Devanagari
digit blocks, ₹-annotated amounts, मध्य-मस useful — determined by the
per-line CER profile of A-zero (error concentration, not vibes).
- Same-crop exactness: train the augmented pages through the exact
`crop_line`/`pad_batch` code (import from the cloned package), so zero-shot
and fine-tuned are measured through one code path.
- Leakage (§12): any *train* page rendered by the deployed pipeline must be
excluded from ALL three frozen eval sets; confirm set boundaries by
re-running `tools/eval_metrics.py`'s pairing logic [B].
## §12 Leakage review [B]
1. **Set purity:** frozen eval sets must not overlap the rec fine-tune
corpus. Verify by page-hash (sha256 of rendered page images), not
filenames ([B] run on remote).
2. **Synth contamination:** if synthetic rendered pages were part of
`ocr-deva-ft` training and the eval sets contain natural scans only, no
leakage; confirm the eval-set provenance doc. Rendered-by-us
(`make_synth_page.py`) pages are NEVER used for training or reporting.
3. **Adaptation leakage:** Experiment D swaps components inside a pipeline
built on eval sets — acceptable for comparison because GT boxes come from
the DETECTOR of the deployed pipeline on EVAL images only, and neither
component is trained on them.
4. **Font leakage:** if fonts in the training corpus are also in the eval
sets' print source (same family), CER estimates are optimistic for BOTH
systems equally — acceptable for A/B ranking, not for absolute claims;
state it in the report.
## §13 Licensing matrix [B for dataset rows]
| Item | License | Fine-tune allowed? | Deploy allowed? | Notes |
|---|---|---|---|---|
| Indra weights v3.1.1 | AI PUBS Open RAIL-M (modified) | **YES, explicit** (§II ¶36) | YES (restrictions = Attachment A misuse list; internal OCR unaffected) | Derivative weights → same license + notice on redistribution |
| `indra_ocr` code | Apache-2.0 | n/a | YES | |
| PaddleOCR (baseline) | Apache-2.0 | YES (we already fine-tune it) | YES | current deployed |
| PP-OCRv5 Mobile Devanagari weights | Apache-2.0 | YES | YES | deployed |
| Nepali train corpus | per `docs/DATA_LICENSING.md` [B] | pending review | — | no removal of datasets from remote |
| Frozen eval sets | per `docs/DATA_LICENSING.md` [B] | evaluation only | — | |
| Noto Sans Devanagari (testrender only) | OFL 1.1 | YES (fonts embedding fine) | test asset only | `tools/indra_eval/testdata/` |
## §14 Deployment comparison
| Axis | Indra v3.1.1 (rec-only swap / full) | PP-OCRv5 Mobile Devanagari deployed |
|---|---|---|
| Recognizer params | **1.236 M** (verified) | rec ~similar class (Mobile range; exact count from remote bench) |
| Runtime | onnxruntime CPU (already installed here) | paddle (remote) |
| Extra machinery needed | **none** — single ONNX + meta.json vocab JSON | paddle + FineTuned assets + converter glue |
| Latency (lines/s, T4 CPU) | measure (A/D/E) | measure |
| Deterministic CPU memory | flat batch (mask) | paddle per-page buffers |
| Order/reading model | included (order.onnx; optional) | not used for plain OCR flow |
| Honest staffing: fine-tune effort | reimplement trainer (~1–2 days) | already exists for Paddle |
| Dependency risk | pinned HF revision + local files | paddle version drift |
## §17 Decision tree
1. **Run A (zero-shot rec-lines + full) on the frozen sets.**
- If Indra rec CER < deployed rec CER **on 2 of 3 sets**: proceed to the
hybrid experiment D (PP-det + Indra-rec); if D holds the win, plan a
**rec-swap deployment trial** alongside the existing PP pipeline(1 week).
- If Indra rec CER ≥ deployed rec CER: record the gap per slice. If the
gap is train-material (conjuncts/₹/digits): run **B** (fine-tune rec
from safetensors, ~10–20k steps); expected to close a
distribution-gap-shaped loss; decide by ΔCER on the eval sets.
2. **Regardless of A's outcome,** the graceful fallback stands: bringing Indra
recognizer into our PP-OCR det pipeline is a **drop-in component** (same
crops→ink-map contract), so nothing about A/B/D commits us.
3. **Det/order fine-tuning (C) stays out of scope** until det-on-real-scan
failure is demonstrated in A-full (`det recall < line recall of deployed`).
4. **Never** touch: datasets on the remote, the deployed pipeline's recipes,
its eval scripts, or `~/.ssh`/`~/.bashrc`. All experiments run in a
separate directory (`~/ocr/OCR_INDRA/`), copying only code/weights in.
## §18 Status summary
- [V] runnable here end-to-end: smoke chain (synth page → rec-only → full →
CER), all inputs/outputs/paths in Investigation §8.
- [B] everything requiring `khalti-t4`: A.2 (frozen sets), B data, C skip
decision input, licensing rows, E throughput on deployment hardware.
- [V] fine-tune toolkit verified offline 2026-10-20 (`tools/exp/verify_bundle.sh`:
export exactness pass; mechanics training + post-train gate pass; A/B
orchestrations complete on a local stand-in set); int8 static/dynamic
behaviour probed (dynamic = slower trap, static = ~1.4× faster, 3.5× smaller).
- Open: WP det recall numbers, paddle latency comparison, int8 CER on real
calibration + deployment-box throughput.
|