ocr-projects-archive / OCR_INDRA /docs /INDRA_EXPERIMENTS.md
prosperious's picture
Add 1000 files (OCR_INDRA/.gitignore ...)
f9bc0c1 verified
|
Raw History Blame Contribute Delete
14.9 kB
# Indra × Nepali — Experiment Plan (A–E)
Companion to `docs/INDRA_INVESTIGATION.md`. Everything runnable is designed
around `tools/indra_eval/eval_indra.py` (works on CPU; the remote T4 simply
makes batches fast). Remote-dependent sections are marked **[B]** and are
blocked until SSH access to `khalti-t4` (202.51.0.7) is restored; every
command to run there is written out so the set can be executed as-is.
Fairness principle: **recognizer comparisons use identical line crops**
(rec-lines mode); **pipeline comparisons use page-level text**. Both modes are
built and smoke-tested (`results/smoke/`: CER 3.49% rec-only / 2.62% full on a
synthetic page — indicational only).
---
## Experiment A — zero-shot Indra vs deployed PP-OCRv5 (do FIRST) [B with A.1 runnable]
**Goal:** close "does the ~1.24 M-param Indra recognizer beat the deployed
PP-OCRv5 Mobile Devanagari recognizer?" without training anything.
**Materials (all remote):** frozen eval sets `~/ocr/ocr-eval-20260924*/` +
`docs/DATA_LICENSING.md`, the deployed paddle pipeline
(`~/ocr/ocr-deva-ft`, `tools/deva_rec_torch`, `tools/eval_metrics.py`), the
CLONE of this repo + `models/indra-ocr-v3.1.1/` (copy `indra/` and `models/`
up: ~46 MB — no dataset leaves the machine).
**A.1 runnable from here already indep. of remote sets:**
- `make_synth_page.py` renders pages at **matching font metrics** for a sanity
envelope (no migration of datasets needed: it was built here, scored here).
**A.2 (remote) recognizer head-to-head on identical crops:**
1. Export the deployed pipeline's GT line boxes for an eval set once:
PP-OCRv5 det results on the set → boxes jsonl
(`{"file", lines: [{i, text: GT, box}]}`).
2. Run our adapter on the same crops:
```
PYTHONPATH=$HOME/ocr/OCR_INDRA/indra:$HOME/ocr/dep python3 \
tools/indra_eval/eval_indra.py predict <page.jpg> \
--boxes <gt_boxes.jsonl> --jsonl results/indra_rec/<set>.jsonl
```
and score with the identical-crop baseline (PP-OCRv5 rec on the same cropped
images saved as png), then `cer` both against GT.
3. Report: micro/macro CER + WER + per-line detail + conf histogram;(any
`MIN_CONF`-dropped lines are reported separately (they hit the full-pipeline
number but not rec-lines).
4. Also submit **full-pipeline** Indra (`predict` without `--boxes`) on the
same set for the det+rec+order end-to-end number.
**Pass/fail:** Indra rec-lines CER < deployed rec CER on ≥ 2 of 3 eval sets,
with per-set mean confidence reported. Also record det recall (lines found /
GT lines) for the full-pipeline run.
## Experiment B — fine-tune the recognizer (the supported path)
**Goal:** Adapt `rec.onnx`'s weights to Nepali-specific distribution
(conjunct-heavy printing, nukta, ₹, Devanagari digits in running text), retrain
with `model.safetensors` as init, export an ONNX with the *identical*
interface, drop-in replace.
**Architecture requirements to reproduce byte-compatible behavior (from
Investigation §1):**
- module: stem Conv(1→128, k32×4, s32×4) + BN + HardSwish → mask Mul →
10 × [dw 9-k1 g128 pad4 → expand 1×1 128→256 → HS → SE(masked-mean→256→64→256
sigmoid gate) → project 1×1 256→128 → +residual → mask Mul] → head 1×1
128→1678 → softmax → argmax.
- Padding mask `(arange(W/4) < lengths).float()` with `lengths =
ceil(widths/4)` (clip W/4); apply post-stem, at SE pools (mean over VALID
steps only), and after each block's residual add. **The mask is functional,
not cosmetic: it keeps SE statistics from being diluted by padding.**
- Parameter mapping table: `model.safetensors` keys `rec/<name>` —
`stem.weight` [128,1,32,4], `stem.bias` [128], `stem_bn.{weight,bias,
running_mean,running_var}` [128], `blocks.<i>.dw.weight` [128,1,9] /
`dw.bias` [128] (ONNX names `onnx::Conv_{369,370};372,373,…396,397`),
`blocks.<i>.expand.{weight,bias}` [256,128], `blocks.<i>.se.fc1/fc2`,
`blocks.<i>.project.{weight,bias}` [128,256], `head.weight` [1678,128,1],
`head.bias` [1678].
- Loss: CTC (log_softmax over [B,T,1678]; blank id 0; input_lengths = lengths,
target = `vocab.encode(canonical(text))` IDs list; NFC-normalize exactly as
`vocab.py:42`).
- Optimizer: AdamW, lr 3e-4 cosine, ~10–20k steps, batch 32 lines by
width-bucket, crop 32px — the pipeline above runs on 1× A100/40 GB in
minutes per epoch for our line counts ([B] exact data volume from
`~/ocr/ocr-deva-ft/data`).
**Unique risk (verified):** the trainer's crop augmentation recipe is closed;
starting too aggressively (`rotate`, `tone`) would shift the input
distribution away from `crop_line`'s exact ops. Mitigation: augmentations
limited to elastic/scale jitter of ±5% consistent with `PAD_Y/PAD_X` bend,
train/eval identical crop code (shared module imported from `rec_preproc.py`
— the DOC that matters).
**Export/eval:** torch → ONNX with same outputs (`ids,conf,lengths`) or a
logits-only export + a 15-line Python decode; either way validate against
`rec.onnx` outputs on 512 in-crops before scoring (**exactness test**:
max |Δconf| < 1e-3 on identical inputs).
**Verified offline 2026-10-20** (torch 2.14.1+cpu / onnxruntime 1.31.0,
`tools/exp/verify_bundle.sh`): `model.safetensors` → IndraRec (1,236,366
tensors) → ONNX export matches the shipped `rec.onnx` with **0 id mismatches,
0 lengths mismatches, max |Δconf| 7.1e-6** on the 1,590-step probe battery;
the post-train gate (torch .pt vs its own export) passes at the same tolerance;
`run_rec.py` reproduces the recorded smoke CER (3.49 %) bit-for-bit. The
exporter forces the legacy TorchScript path (`dynamo=False`) because torch ≥ 2.6
defaults to the torch.export exporter, which needs the `onnxscript` package.
**Acceptance:** fine-tuned rec-lines CER < zero-shot rec-lines CER on all
3 frozen sets by ≥ 2 points relative, on Devanagari digit-heavy and
conjunct-heavy slices specifically (those are where the smoke run showed
weakness: `सङ्केत→सड्लेत`, `₹→र`, `ऋतु→तऋहतु`).
## Experiment C — fine-tune the line detector / order model
**Verdict from Investigation §3: blocked by shipped-artifact reality** —
`det.onnx`/`order.onnx` come with NO state dict and NO recipe. Two paths:
1. **Torch re-implementation from scratch** using the target specs that ARE
in the repo (`det_post.encode_targets` → line core logit + top/bottom
distance regression; `order_post.label_order` → 10 block classes +
pairwise salience). Cost: reimplementing the backbone (which is unknown —
the ONNX graphs would need the same node-by-node reverse archaeology as
rec.onnx: feasible, we did it, but heavy) + a data pipeline + synth label
generator. **Not recommended as the first move** — zero-shot + rec-only
fine-tuning captures most value: the det model already detects 10/10
synthetic lines and machine-printed Devanagari well.
2. **Skip det/order entirely**; if detection quality on real scan material
turns out to be the bottleneck in A-full, locally replace with our known
corresponding PP-OCR det model and keep only the Indra rec — that is
Experiment D.
## Experiment D — hybrid: keep PP-OCR det, swap rec → Indra
**Goal:** combine a known-good Devanagari line finder (PP-OCRv5 det) with the
Indra recognizer under identical crops, and vice-versa. Outputs line boxes
jsonl exactly like A.2-step 1. This is a 1-day experiment **with zero training**
and makes the fair-comparison claim airtight: if `PP-det + Indra-rec` beats
`PP-det + PP-rec` on the same crops, the recognizer itself is the win; if not,
Indra's advantage (if any) lives elsewhere (orientation/order/layout).
## Experiment E — productionization / distill for WSL CPU box
**Goal:** a deployable artifact for the 8-core/3 GB (or T4) runtime:
- Indra package supports `rec.int8.onnx` automatically (`pipeline.py:195–200`)
— quantize the rec model once (no retraining), then re-run Experiment A with
the int8 model: acceptance: CER regression ≤ +0.3 pts absolute and
≥ 2× throughput.
- **Measured local guidance (2026-10-20, WSL CPU, threads=4, smoke crops;
indicative, not a benchmark):** ORT *dynamic* (weights-only) quantization is
a **trap** for this Conv+SE model — 0.69–0.27× fp32 throughput (ConvInteger +
per-activation quant/dequant dominates) though greedy ids stay identical and
max |Δconf| = 0.077. *Static* QDQ uint8 (per-tensor, MinMax, 64 calib crops)
built 3.5× smaller (4.98 → 1.43 MB) and ran **~1.4× faster** than fp32 — on a
stronger box the gap should widen; the ≥2× acceptance must be re-measured
there. Tool: `tools/exp/e_int8_quantize.py` (static, calibration through the
shared `crop_line`; `--check-smoke` reports id/conf drift; writes a JSON report).
Use real line crops for calibration (the local test used noise → id drift 56
steps on the smoke page; real calibration should cut that).
- Latency budget on our box is meaningless (WSL starved) — measure only on
the T4: lines/s at long-side 1200/1600/2048 (det scales with page size).
- Memory: with `enable_cpu_mem_arena=False` (already set in the package),
line-batched rec is flat per page.
---
## §11 Data plan for fine-tuning (Experiment B)
- Source: the same frozen train corpus that produced the deployed fine-tuned
PP-OCR model (`~/ocr/ocr-deva-ft/data` — [B] inventory is on remote;
`docs/DEVANAGARI_RESOURCES_INVENTORY.md` documents it). **No dataset is
copied off the remote machine** — training artifacts/results files transfer,
not eval sets.
- Emphasis slices for Nepali: conjunct-heavy book/print pages, Devanagari
digit blocks, ₹-annotated amounts, मध्य-मस useful — determined by the
per-line CER profile of A-zero (error concentration, not vibes).
- Same-crop exactness: train the augmented pages through the exact
`crop_line`/`pad_batch` code (import from the cloned package), so zero-shot
and fine-tuned are measured through one code path.
- Leakage (§12): any *train* page rendered by the deployed pipeline must be
excluded from ALL three frozen eval sets; confirm set boundaries by
re-running `tools/eval_metrics.py`'s pairing logic [B].
## §12 Leakage review [B]
1. **Set purity:** frozen eval sets must not overlap the rec fine-tune
corpus. Verify by page-hash (sha256 of rendered page images), not
filenames ([B] run on remote).
2. **Synth contamination:** if synthetic rendered pages were part of
`ocr-deva-ft` training and the eval sets contain natural scans only, no
leakage; confirm the eval-set provenance doc. Rendered-by-us
(`make_synth_page.py`) pages are NEVER used for training or reporting.
3. **Adaptation leakage:** Experiment D swaps components inside a pipeline
built on eval sets — acceptable for comparison because GT boxes come from
the DETECTOR of the deployed pipeline on EVAL images only, and neither
component is trained on them.
4. **Font leakage:** if fonts in the training corpus are also in the eval
sets' print source (same family), CER estimates are optimistic for BOTH
systems equally — acceptable for A/B ranking, not for absolute claims;
state it in the report.
## §13 Licensing matrix [B for dataset rows]
| Item | License | Fine-tune allowed? | Deploy allowed? | Notes |
|---|---|---|---|---|
| Indra weights v3.1.1 | AI PUBS Open RAIL-M (modified) | **YES, explicit** (§II ¶36) | YES (restrictions = Attachment A misuse list; internal OCR unaffected) | Derivative weights → same license + notice on redistribution |
| `indra_ocr` code | Apache-2.0 | n/a | YES | |
| PaddleOCR (baseline) | Apache-2.0 | YES (we already fine-tune it) | YES | current deployed |
| PP-OCRv5 Mobile Devanagari weights | Apache-2.0 | YES | YES | deployed |
| Nepali train corpus | per `docs/DATA_LICENSING.md` [B] | pending review | — | no removal of datasets from remote |
| Frozen eval sets | per `docs/DATA_LICENSING.md` [B] | evaluation only | — | |
| Noto Sans Devanagari (testrender only) | OFL 1.1 | YES (fonts embedding fine) | test asset only | `tools/indra_eval/testdata/` |
## §14 Deployment comparison
| Axis | Indra v3.1.1 (rec-only swap / full) | PP-OCRv5 Mobile Devanagari deployed |
|---|---|---|
| Recognizer params | **1.236 M** (verified) | rec ~similar class (Mobile range; exact count from remote bench) |
| Runtime | onnxruntime CPU (already installed here) | paddle (remote) |
| Extra machinery needed | **none** — single ONNX + meta.json vocab JSON | paddle + FineTuned assets + converter glue |
| Latency (lines/s, T4 CPU) | measure (A/D/E) | measure |
| Deterministic CPU memory | flat batch (mask) | paddle per-page buffers |
| Order/reading model | included (order.onnx; optional) | not used for plain OCR flow |
| Honest staffing: fine-tune effort | reimplement trainer (~1–2 days) | already exists for Paddle |
| Dependency risk | pinned HF revision + local files | paddle version drift |
## §17 Decision tree
1. **Run A (zero-shot rec-lines + full) on the frozen sets.**
- If Indra rec CER < deployed rec CER **on 2 of 3 sets**: proceed to the
hybrid experiment D (PP-det + Indra-rec); if D holds the win, plan a
**rec-swap deployment trial** alongside the existing PP pipeline(1 week).
- If Indra rec CER ≥ deployed rec CER: record the gap per slice. If the
gap is train-material (conjuncts/₹/digits): run **B** (fine-tune rec
from safetensors, ~10–20k steps); expected to close a
distribution-gap-shaped loss; decide by ΔCER on the eval sets.
2. **Regardless of A's outcome,** the graceful fallback stands: bringing Indra
recognizer into our PP-OCR det pipeline is a **drop-in component** (same
crops→ink-map contract), so nothing about A/B/D commits us.
3. **Det/order fine-tuning (C) stays out of scope** until det-on-real-scan
failure is demonstrated in A-full (`det recall < line recall of deployed`).
4. **Never** touch: datasets on the remote, the deployed pipeline's recipes,
its eval scripts, or `~/.ssh`/`~/.bashrc`. All experiments run in a
separate directory (`~/ocr/OCR_INDRA/`), copying only code/weights in.
## §18 Status summary
- [V] runnable here end-to-end: smoke chain (synth page → rec-only → full →
CER), all inputs/outputs/paths in Investigation §8.
- [B] everything requiring `khalti-t4`: A.2 (frozen sets), B data, C skip
decision input, licensing rows, E throughput on deployment hardware.
- [V] fine-tune toolkit verified offline 2026-10-20 (`tools/exp/verify_bundle.sh`:
export exactness pass; mechanics training + post-train gate pass; A/B
orchestrations complete on a local stand-in set); int8 static/dynamic
behaviour probed (dynamic = slower trap, static = ~1.4× faster, 3.5× smaller).
- Open: WP det recall numbers, paddle latency comparison, int8 CER on real
calibration + deployment-box throughput.