File size: 14,908 Bytes
f9bc0c1
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
# Indra × Nepali — Experiment Plan (A–E)

Companion to `docs/INDRA_INVESTIGATION.md`. Everything runnable is designed
around `tools/indra_eval/eval_indra.py` (works on CPU; the remote T4 simply
makes batches fast). Remote-dependent sections are marked **[B]** and are
blocked until SSH access to `khalti-t4` (202.51.0.7) is restored; every
command to run there is written out so the set can be executed as-is.

Fairness principle: **recognizer comparisons use identical line crops**
(rec-lines mode); **pipeline comparisons use page-level text**. Both modes are
built and smoke-tested (`results/smoke/`: CER 3.49% rec-only / 2.62% full on a
synthetic page — indicational only).

---

## Experiment A — zero-shot Indra vs deployed PP-OCRv5 (do FIRST) [B with A.1 runnable]

**Goal:** close "does the ~1.24 M-param Indra recognizer beat the deployed
PP-OCRv5 Mobile Devanagari recognizer?" without training anything.

**Materials (all remote):** frozen eval sets `~/ocr/ocr-eval-20260924*/` +
`docs/DATA_LICENSING.md`, the deployed paddle pipeline
(`~/ocr/ocr-deva-ft`, `tools/deva_rec_torch`, `tools/eval_metrics.py`), the
CLONE of this repo + `models/indra-ocr-v3.1.1/` (copy `indra/` and `models/`
up: ~46 MB — no dataset leaves the machine).

**A.1 runnable from here already indep. of remote sets:**
- `make_synth_page.py` renders pages at **matching font metrics** for a sanity
  envelope (no migration of datasets needed: it was built here, scored here).

**A.2 (remote) recognizer head-to-head on identical crops:**
1. Export the deployed pipeline's GT line boxes for an eval set once:
   PP-OCRv5 det results on the set → boxes jsonl
   (`{"file", lines: [{i, text: GT, box}]}`).
2. Run our adapter on the same crops:
   ```
   PYTHONPATH=$HOME/ocr/OCR_INDRA/indra:$HOME/ocr/dep python3 \
     tools/indra_eval/eval_indra.py predict <page.jpg> \
     --boxes <gt_boxes.jsonl> --jsonl results/indra_rec/<set>.jsonl
   ```
   and score with the identical-crop baseline (PP-OCRv5 rec on the same cropped
   images saved as png), then `cer` both against GT.
3. Report: micro/macro CER + WER + per-line detail + conf histogram;(any
   `MIN_CONF`-dropped lines are reported separately (they hit the full-pipeline
   number but not rec-lines).
4. Also submit **full-pipeline** Indra (`predict` without `--boxes`) on the
   same set for the det+rec+order end-to-end number.

**Pass/fail:** Indra rec-lines CER < deployed rec CER on ≥ 2 of 3 eval sets,
with per-set mean confidence reported. Also record det recall (lines found /
GT lines) for the full-pipeline run.

## Experiment B — fine-tune the recognizer (the supported path)

**Goal:** Adapt `rec.onnx`'s weights to Nepali-specific distribution
(conjunct-heavy printing, nukta, ₹, Devanagari digits in running text), retrain
with `model.safetensors` as init, export an ONNX with the *identical*
interface, drop-in replace.

**Architecture requirements to reproduce byte-compatible behavior (from
Investigation §1):**
- module: stem Conv(1→128, k32×4, s32×4) + BN + HardSwish → mask Mul →
  10 × [dw 9-k1 g128 pad4 → expand 1×1 128→256 → HS → SE(masked-mean→256→64→256
  sigmoid gate) → project 1×1 256→128 → +residual → mask Mul] → head 1×1
  128→1678 → softmax → argmax.
- Padding mask `(arange(W/4) < lengths).float()` with `lengths =
  ceil(widths/4)` (clip W/4); apply post-stem, at SE pools (mean over VALID
  steps only), and after each block's residual add. **The mask is functional,
  not cosmetic: it keeps SE statistics from being diluted by padding.**
- Parameter mapping table: `model.safetensors` keys `rec/<name>` —
  `stem.weight` [128,1,32,4], `stem.bias` [128], `stem_bn.{weight,bias,
  running_mean,running_var}` [128], `blocks.<i>.dw.weight` [128,1,9] /
  `dw.bias` [128] (ONNX names `onnx::Conv_{369,370};372,373,…396,397`),
  `blocks.<i>.expand.{weight,bias}` [256,128], `blocks.<i>.se.fc1/fc2`,
  `blocks.<i>.project.{weight,bias}` [128,256], `head.weight` [1678,128,1],
  `head.bias` [1678].
- Loss: CTC (log_softmax over [B,T,1678]; blank id 0; input_lengths = lengths,
  target = `vocab.encode(canonical(text))` IDs list; NFC-normalize exactly as
  `vocab.py:42`).
- Optimizer: AdamW, lr 3e-4 cosine, ~10–20k steps, batch 32 lines by
  width-bucket, crop 32px — the pipeline above runs on 1× A100/40 GB in
  minutes per epoch for our line counts ([B] exact data volume from
  `~/ocr/ocr-deva-ft/data`).

**Unique risk (verified):** the trainer's crop augmentation recipe is closed;
starting too aggressively (`rotate`, `tone`) would shift the input
distribution away from `crop_line`'s exact ops. Mitigation: augmentations
limited to elastic/scale jitter of ±5% consistent with `PAD_Y/PAD_X` bend,
train/eval identical crop code (shared module imported from `rec_preproc.py`
— the DOC that matters).

**Export/eval:** torch → ONNX with same outputs (`ids,conf,lengths`) or a
logits-only export + a 15-line Python decode; either way validate against
`rec.onnx` outputs on 512 in-crops before scoring (**exactness test**:
max |Δconf| < 1e-3 on identical inputs).

**Verified offline 2026-10-20** (torch 2.14.1+cpu / onnxruntime 1.31.0,
`tools/exp/verify_bundle.sh`): `model.safetensors` → IndraRec (1,236,366
tensors) → ONNX export matches the shipped `rec.onnx` with **0 id mismatches,
0 lengths mismatches, max |Δconf| 7.1e-6** on the 1,590-step probe battery;
the post-train gate (torch .pt vs its own export) passes at the same tolerance;
`run_rec.py` reproduces the recorded smoke CER (3.49 %) bit-for-bit. The
exporter forces the legacy TorchScript path (`dynamo=False`) because torch ≥ 2.6
defaults to the torch.export exporter, which needs the `onnxscript` package.

**Acceptance:** fine-tuned rec-lines CER < zero-shot rec-lines CER on all
3 frozen sets by ≥ 2 points relative, on Devanagari digit-heavy and
conjunct-heavy slices specifically (those are where the smoke run showed
weakness: `सङ्केत→सड्लेत`, `₹→र`, `ऋतु→तऋहतु`).

## Experiment C — fine-tune the line detector / order model

**Verdict from Investigation §3: blocked by shipped-artifact reality** —
`det.onnx`/`order.onnx` come with NO state dict and NO recipe. Two paths:
1. **Torch re-implementation from scratch** using the target specs that ARE
   in the repo (`det_post.encode_targets` → line core logit + top/bottom
   distance regression; `order_post.label_order` → 10 block classes +
   pairwise salience). Cost: reimplementing the backbone (which is unknown —
   the ONNX graphs would need the same node-by-node reverse archaeology as
   rec.onnx: feasible, we did it, but heavy) + a data pipeline + synth label
   generator. **Not recommended as the first move** — zero-shot + rec-only
   fine-tuning captures most value: the det model already detects 10/10
   synthetic lines and machine-printed Devanagari well.
2. **Skip det/order entirely**; if detection quality on real scan material
   turns out to be the bottleneck in A-full, locally replace with our known
   corresponding PP-OCR det model and keep only the Indra rec — that is
   Experiment D.

## Experiment D — hybrid: keep PP-OCR det, swap rec → Indra

**Goal:** combine a known-good Devanagari line finder (PP-OCRv5 det) with the
Indra recognizer under identical crops, and vice-versa. Outputs line boxes
jsonl exactly like A.2-step 1. This is a 1-day experiment **with zero training**
and makes the fair-comparison claim airtight: if `PP-det + Indra-rec` beats
`PP-det + PP-rec` on the same crops, the recognizer itself is the win; if not,
Indra's advantage (if any) lives elsewhere (orientation/order/layout).

## Experiment E — productionization / distill for WSL CPU box

**Goal:** a deployable artifact for the 8-core/3 GB (or T4) runtime:
- Indra package supports `rec.int8.onnx` automatically (`pipeline.py:195–200`)
  — quantize the rec model once (no retraining), then re-run Experiment A with
  the int8 model: acceptance: CER regression ≤ +0.3 pts absolute and
  ≥ 2× throughput.
- **Measured local guidance (2026-10-20, WSL CPU, threads=4, smoke crops;
  indicative, not a benchmark):** ORT *dynamic* (weights-only) quantization is
  a **trap** for this Conv+SE model — 0.69–0.27× fp32 throughput (ConvInteger +
  per-activation quant/dequant dominates) though greedy ids stay identical and
  max |Δconf| = 0.077.  *Static* QDQ uint8 (per-tensor, MinMax, 64 calib crops)
  built 3.5× smaller (4.98 → 1.43 MB) and ran **~1.4× faster** than fp32 — on a
  stronger box the gap should widen; the ≥2× acceptance must be re-measured
  there.  Tool: `tools/exp/e_int8_quantize.py` (static, calibration through the
  shared `crop_line`; `--check-smoke` reports id/conf drift; writes a JSON report).
  Use real line crops for calibration (the local test used noise → id drift 56
  steps on the smoke page; real calibration should cut that).
- Latency budget on our box is meaningless (WSL starved) — measure only on
  the T4: lines/s at long-side 1200/1600/2048 (det scales with page size).
- Memory: with `enable_cpu_mem_arena=False` (already set in the package),
  line-batched rec is flat per page.

---

## §11 Data plan for fine-tuning (Experiment B)

- Source: the same frozen train corpus that produced the deployed fine-tuned
  PP-OCR model (`~/ocr/ocr-deva-ft/data` — [B] inventory is on remote;
  `docs/DEVANAGARI_RESOURCES_INVENTORY.md` documents it). **No dataset is
  copied off the remote machine** — training artifacts/results files transfer,
  not eval sets.
- Emphasis slices for Nepali: conjunct-heavy book/print pages, Devanagari
  digit blocks, ₹-annotated amounts, मध्य-मस useful — determined by the
  per-line CER profile of A-zero (error concentration, not vibes).
- Same-crop exactness: train the augmented pages through the exact
  `crop_line`/`pad_batch` code (import from the cloned package), so zero-shot
  and fine-tuned are measured through one code path.
- Leakage (§12): any *train* page rendered by the deployed pipeline must be
  excluded from ALL three frozen eval sets; confirm set boundaries by
  re-running `tools/eval_metrics.py`'s pairing logic [B].

## §12 Leakage review [B]

1. **Set purity:** frozen eval sets must not overlap the rec fine-tune
   corpus. Verify by page-hash (sha256 of rendered page images), not
   filenames ([B] run on remote).
2. **Synth contamination:** if synthetic rendered pages were part of
   `ocr-deva-ft` training and the eval sets contain natural scans only, no
   leakage; confirm the eval-set provenance doc. Rendered-by-us
   (`make_synth_page.py`) pages are NEVER used for training or reporting.
3. **Adaptation leakage:** Experiment D swaps components inside a pipeline
   built on eval sets — acceptable for comparison because GT boxes come from
   the DETECTOR of the deployed pipeline on EVAL images only, and neither
   component is trained on them.
4. **Font leakage:** if fonts in the training corpus are also in the eval
   sets' print source (same family), CER estimates are optimistic for BOTH
   systems equally — acceptable for A/B ranking, not for absolute claims;
   state it in the report.

## §13 Licensing matrix [B for dataset rows]

| Item | License | Fine-tune allowed? | Deploy allowed? | Notes |
|---|---|---|---|---|
| Indra weights v3.1.1 | AI PUBS Open RAIL-M (modified) | **YES, explicit** (§II ¶36) | YES (restrictions = Attachment A misuse list; internal OCR unaffected) | Derivative weights → same license + notice on redistribution |
| `indra_ocr` code | Apache-2.0 | n/a | YES | |
| PaddleOCR (baseline) | Apache-2.0 | YES (we already fine-tune it) | YES | current deployed |
| PP-OCRv5 Mobile Devanagari weights | Apache-2.0 | YES | YES | deployed |
| Nepali train corpus | per `docs/DATA_LICENSING.md` [B] | pending review | — | no removal of datasets from remote |
| Frozen eval sets | per `docs/DATA_LICENSING.md` [B] | evaluation only | — | |
| Noto Sans Devanagari (testrender only) | OFL 1.1 | YES (fonts embedding fine) | test asset only | `tools/indra_eval/testdata/` |

## §14 Deployment comparison

| Axis | Indra v3.1.1 (rec-only swap / full) | PP-OCRv5 Mobile Devanagari deployed |
|---|---|---|
| Recognizer params | **1.236 M** (verified) | rec ~similar class (Mobile range; exact count from remote bench) |
| Runtime | onnxruntime CPU (already installed here) | paddle (remote) |
| Extra machinery needed | **none** — single ONNX + meta.json vocab JSON | paddle + FineTuned assets + converter glue |
| Latency (lines/s, T4 CPU) | measure (A/D/E) | measure |
| Deterministic CPU memory | flat batch (mask) | paddle per-page buffers |
| Order/reading model | included (order.onnx; optional) | not used for plain OCR flow |
| Honest staffing: fine-tune effort | reimplement trainer (~1–2 days) | already exists for Paddle |
| Dependency risk | pinned HF revision + local files | paddle version drift |

## §17 Decision tree

1. **Run A (zero-shot rec-lines + full) on the frozen sets.**
   - If Indra rec CER < deployed rec CER **on 2 of 3 sets**: proceed to the
     hybrid experiment D (PP-det + Indra-rec); if D holds the win, plan a
     **rec-swap deployment trial** alongside the existing PP pipeline(1 week).
   - If Indra rec CER ≥ deployed rec CER: record the gap per slice. If the
     gap is train-material (conjuncts/₹/digits): run **B** (fine-tune rec
     from safetensors, ~10–20k steps); expected to close a
     distribution-gap-shaped loss; decide by ΔCER on the eval sets.
2. **Regardless of A's outcome,** the graceful fallback stands: bringing Indra
   recognizer into our PP-OCR det pipeline is a **drop-in component** (same
   crops→ink-map contract), so nothing about A/B/D commits us.
3. **Det/order fine-tuning (C) stays out of scope** until det-on-real-scan
   failure is demonstrated in A-full (`det recall < line recall of deployed`).
4. **Never** touch: datasets on the remote, the deployed pipeline's recipes,
   its eval scripts, or `~/.ssh`/`~/.bashrc`. All experiments run in a
   separate directory (`~/ocr/OCR_INDRA/`), copying only code/weights in.

## §18 Status summary

- [V] runnable here end-to-end: smoke chain (synth page → rec-only → full →
  CER), all inputs/outputs/paths in Investigation §8.
- [B] everything requiring `khalti-t4`: A.2 (frozen sets), B data, C skip
  decision input, licensing rows, E throughput on deployment hardware.
- [V] fine-tune toolkit verified offline 2026-10-20 (`tools/exp/verify_bundle.sh`:
  export exactness pass; mechanics training + post-train gate pass; A/B
  orchestrations complete on a local stand-in set); int8 static/dynamic
  behaviour probed (dynamic = slower trap, static = ~1.4× faster, 3.5× smaller).
- Open: WP det recall numbers, paddle latency comparison, int8 CER on real
  calibration + deployment-box throughput.