File size: 17,615 Bytes
f9bc0c1 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 | # Indra OCR β Engineering Investigation (v3.1.1, package 0.3.2)
Investigator: OCR_INDRA workspace, 2026-10-17.
Method: every claim below is backed by (a) model artifacts inspected bit-for-bit
(`models/indra-ocr-v3.1.1/`, Hub revision `v3.1.1`) or (b) source at a cited
file:line in the cloned repo (`indra/`, commit `3d773ae`, tag v0.3.2).
Verification scripts live in `tools/indra_eval/` and are re-runnable.
Status legend: **[V]** verified with evidence here, **[B]** blocked (needs the
remote machine `khalti-t4`), **[X]** closed as refuted/inaccurate vs the
earlier report `INDRA_OCR_NEPALI_FEASIBILITY.md`.
---
## 0. Verdict
**The recognizer is trainable in principle and Nepali-ready in vocabulary, but
fine-tuning is a reimplementation project: the repo ships no training code and
no training weights for anything but the recognizer.**
- Zero-shot evaluation (Experiment A) is *ready today* β the adapter
(`tools/indra_eval/eval_indra.py`) runs Indra end-to-end on CPU and scores
CER; it needs only the frozen Nepali eval sets from the remote machine [B].
- The CTC recognizer (`rec.onnx`) covers 100% of Devanagari (all 128 assigned
codepoints U+0900βU+097F verified in the vocab) β no head surgery needed [V].
- In a same-crop head-to-head under our environment, the ~1.24 M-param
Indra recognizer scored **CER 3.49%** on a synthetic Nepali page vs the
deployed PP-OCRv5-Mobile Devanagari recognizer's published behavior on
similar synthetic material (to be re-measured on identical crops) [V smoke / B].
- Fine-tuning only the recognizer with `model.safetensors` as init is the
*only* supported adaptation path: the artifacts ship trainable weights for
`rec/` exclusively; det/order/orient/math exist only as ONNX graphs with no
accompanying state dict [V].
---
## 1. What exactly is the ~1M-param recognizer? [V]
**Answer: a MobileNetV3-style 1-D CNN trunk with padding-masked SE blocks and
a flat CTC head β 1,236,366 parameters exactly, no RNN, no attention.**
Evidence: `tools/indra_eval/inspect_rec_nodes.py` (201-node dump of
`rec.onnx`), `tools/indra_eval/inspect_rec_mask.py` (intermediate-tensor run),
`tools/indra_eval/inspect_models.py` (safetensors β ONNX initializer
cross-check: 108 tensors, 1,236,366 F32 params, byte-identical set).
### 1.1 Tensor flow (all verified from the graph)
```
input x [B,1,32,W] f32 ink map (0=paper, 1=ink; see Β§1.3)
input widths [B] int64 (true crop widths BEFORE padding)
0 Conv(k=32x4, stride=32x4, 1β128, pad 0) "collapse height, stride time 4"
2 Squeeze axis 2 β [B,128,T] with T = W/4
1 BN(eps 1e-5, stem-only BatchNormalization) β HardSwish
43 Mul Γ Cast_4 (padding mask, see Β§1.2) β trunk input
10 Γ { dw Conv [128,1,9] g=128 pad 4/4
expand Conv 1Γ1 128β256 β HardSwish
SE: Mul(mask) β ReduceSum(t) β/β ReduceSum(mask) = valid count
Div (mean over VALID timesteps, Clip min 1)
fc1 256β64 β Relu β fc2 64β256 β Sigmoid β gate Mul
project Conv 1Γ1 256β128
Add residual β Mul(mask) } β mask re-applied per block
188 Conv 1Γ1 128β1678 (head) β Transpose [B,T,1678]
191 ArgMax β ids [B,T]
190/193..200 softmax-max (logsumexp-free: ReduceMax, Exp, ReduceSum, Recip)
β conf [B,T]
27 Clip(lengths) β lengths [B]
```
Op census (from the node dump): 52 Conv, 11 HardSwish, 10 Relu, 10 Sigmoid
(SE gates), 32 Mul, 12 ReduceSum, 1 BatchNormalization, 1 ArgMax β plus
glue. The parameters: 10 identical residual blocks (dw 128Γ1Γ9 + bias,
expand 256Γ128, project 128Γ256, SE 64Γ256 + 256Γ64) β 118 k/block + stem
128Γ32Γ4 + BN 4Γ128 + head 1678Γ128 β 1,236,366 total [V].
### 1.2 The padding mask `Cast_4` β semantics now fully pinned [V, closes a [U]]
`Cast_4 = (arange(W/4) < lengths).float()` of shape `[B,1,W']`, built in-graph
from:
- nodes 3β27: `lengths = floor((widths+3)/4) = ceil(widths/4)`, clipped to
`W/4` (Clip max = `Shape(x)[2] / 4`... clipped to the feature length);
numeric table: `widths 1,2,3,4 β 1; 5 β 2; 9 β 3; 65 β 17; 129 β 32 (= W/4,
clipped); 200 β 32`.
- nodes 31β40: `Range(0, W/4) < lengths` (broadcast `[1,W'] < [B,1]`) β
`Cast` f32 twice (PyTorch export artifact of `unsqueeze().to(f32)`).
The mask is consumed **three ways** (this closes the earlier "[U] what does
Cast_4 do?"):
1. **post-stem zeroing** (node 43) and **per-block output re-zeroing**
(nodes 43, 61, 75, β¦, 187): the trunk never carries energy into padded
timesteps β proven numerically: with ink drawn across the whole 128-column
input but `widths=64`, final-block features at t β₯ 16 are *exactly 0.0*
(max |f| = 0.0) while the same pixels unmasked (widths=128) reach |f|=15.6;
2. **padding-aware SE pooling** (each block: `ReduceSum(HW Γ mask) /
max(ReduceSum(mask), 1)` β channel means over VALID timesteps only; the
mask sum is the divisor, so padded columns cannot dilute the squeeze stats);
3. the **head reads masked features** (node 187β188), so at padded timesteps
the graph emits whatever the head computes on a zero vector β measured:
argmax = 0 (blank) with conf β 0.78. **`ids/conf` at t β₯ lengths are NOT
special-cased; callers must truncate by `lengths`.** `pipeline.py:297`
does exactly this (`n = int(lengths[k])`).
### 1.3 Time stride is 4, not 8 [X β closes a wrong claim]
`rec_preproc.py:24` says `WIDTH_MULTIPLE = 8 # widths are padded to a
multiple of this (timestep stride)`. The graph contradicts the parenthetical:
the stem Conv strides are `[32,4]`, giving **one timestep per 4 input
columns** (`T = W/4`, `lengths = ceil(widths/4)`), and padding to a multiple
of 8 is about the W dimension of the input only (it makes every 2 timesteps
aligned, a vectorization nicety). Verified numerically: `T = 32` for
`W = 128`, and `lengths` equals `ceil(widths/4)` at every measured value.
The comment is wrong as written; the code is right.
### 1.4 Output contract [V β closes a [U]]
Graph outputs are exactly `ids [B,T] int64`, `conf [B,T] f32`,
`lengths [B] int64` (argmax + softmax max prob + valid count computed
in-graph). `pipeline.py:294` consumes them positionally as
`best, pmax, lengths`, then:
- truncates to `n = int(lengths[k])` (:297),
- CTC-greedy collapse: keep β blank(0) and skip repeats (:299),
- confidence = mean `pmax` over kept tokens (:315),
- **drops the whole line if conf < `MIN_CONF = 0.5`** (:60, :316β319).
Any retrained recognizer exported to ONNX must reproduce this contract
(argmax + softmax-max + lengths on the same output names/order), or
`pipeline.py:294` and the x-positions step (which maps CTC peak timesteps
back to page x via the crop scale, :295β306) must be updated.
### 1.5 Recognizer input preprocessing (the shard that fine-tuning must clone)
`rec_preproc.py` β `crop_line` (:27): expand box by `PAD_Y=0.2 Γ h` vertically
and `PAD_X=0.3 Γ h` horizontally (:22β23), restrict to page, resize so
height = `LINE_HEIGHT = 32` keeping aspect (`out_w = max(WIDTH_MULTIPLE,
min(max_width=4096, round(cw*32/ch)))`), producing an ink map where paper = 0.
`pad_batch` (:56) right-pads W to a multiple of 8 in the batch; `width_batches`
(:69) groups crops so W-dim batch padding is bounded. The TRAINER uses the
same crop per its docstring (:5β10 "expand β¦ resize β¦ ink map") β the docstring
states train/infer share this shard, which is why zero-shot works at all and
why fine-tuning must reimplement `crop_line` + `pad_batch` bit-exactly.
## 2. What handles Devanagari? [V]
**Only `rec.onnx` reads Devanagari β and its vocabulary already includes
every assigned Devanagari codepoint.**
- `vocab.py:170` `Vocab.multi()` β 1,677 tokens + blank class 0 (:3, :207),
head = 1,678 classes. The `meta.json` vocab string is exactly this vocab
(`tools/indra_eval/inspect_vocab.py`).
- Coverage: **128/128 assigned codepoints in U+0900βU+097F** β 80 Lo letters,
11 Mc vowel signs, 23 Mn marks incl. ΰ€Ό (U+093C nukta) and the conjunct
virama ΰ€ΰ₯, plus ΰ₯ (U+0950), ΰ€ ΰ₯€ (U+0964), ΰ₯₯ (U+0965), digits ΰ₯¦βΰ₯―
(U+0966β096F), βΉ (U+20B9). Bengali block also present (96 codepoints).
- The recognizer is routed by two *fixed* guards, not the network:
- `script.onnx` (hanzi binary classifier, sigmoid scalar; `pipeline.py:274β289`)
sends hanzi lines to `rec_cjk.onnx` β irrelevant for Nepali pages
(all-Devanagari pages never trigger it; the classifier is line-level and
trained on hanzi).
- `vocab.canonical` (:42) NFC-normalizes; `Vocab.encode` (:212) drops
out-of-vocab codepoints β fine-tuning data must be pre-normalized the
same way or the loss silently degrades (the trainer's data pipeline is
closed; safest is to clone `canonical`/`encode`).
- Confirmed counter-evidence collected in the smoke run: ΰ€Ό (nukta) is often
*dropped contextually* at 32 px height (ΰ€°ΰ€Ό and ΰ€° differ only by a 1β2 px
dot); βΉ was read as ΰ€°. Both are training-distribution issues, not capability
gaps β exactly the kind of thing fine-tuning would fix (Experiment B).
## 3. Component map: trainable / fixed-but-replaceable / fixed [V]
| Component | Where weights come from | Trainable via shipped artifacts? | Replaceability |
|---|---|---|---|
| `rec.onnx` recognizer | `model.safetensors` (108 tensors, all `rec/`) | **YES** β the only shipped state dict; init for a torch CTC retrain (Experiment B) | head/vocab identical β drop-in for +1,678 classes |
| `det.onnx` line detector | ONNX only | NO state dict shipped | could retrain in torch (targets spec exists: `det_post.encode_targets`, `det_post.py:48` β `core logit + top/bottom distances` per line, `CORE_FRAC=0.4`, `CORE_X_SHRINK=0.15`) β but training code is closed β high effort |
| `order.onnx` reading order/layout | ONNX only | NO | same: `order_post.label_order`/`decode_*` reveal supervision (10 block classes) but no weights |
| `orient.onnx` page rotation | ONNX only | NO | trivial to replace with a torch/tlclassifier or heuristic |
| `mask.onnx` CenterMask | NOT SHIPPED in v3.1.1 (`pipeline.py:201β205` optional) | β | absent here; `crop_lines(gray, boxes, 32, None)` runs w/o it (verified) |
| `script.onnx` + `rec_cjk.onnx` | ONNX only | NO | optional; would be excluded from a Devanagari build |
| `math*.onnx` (4 files) | NOT shipped unless `math=True` (`models.py:22β44`) | β | out of scope |
| geometry/deskew/layout postproc | pure Python (`pipeline.py:37β149`, `layout_post.py`) | fixed code, no training needed | replaceable with own code paths |
| text-layer (PDF) naive path | `textlayer.py` | fixed code | irrelevant to scans/Nepali notes |
**The decisive asymmetry:** Google ships a *recognizer-only* training starting
point. Fine-tuning anything else means either (a) training from scratch (det
targets spec'd, but no reference weights/recipe) or (b) leaving det/order
frozen ONNX and only exercising the rec head β that's Experiment B; both are
documented in `INDRA_EXPERIMENTS.md`.
## 4. Integration architecture (how a page becomes text) [V]
`pipeline.py:456β470` (`OrtOCR.__call__`):
1. `upright()`: optional `orient.onnx` turn above `ORIENT_MIN = 0.95` (:54) +
skew estimate/deskew up to 5Β° (:62β63).
2. `detect()`: `det.onnx` on a page resized long-side to detector input,
threshold β line boxes (uses `det_post.page_to_input` / `decode`;
`ADAPTIVE_MIN_H=9px` re-detect pass at a larger scale, :37β40).
3. `recognize()` (:264β320): CenterMask if present (not shipped); per-line
`crop_line` β `pad_batch` β rec session, CTC collapse & conf gating
(Β§1.2/Β§1.4).
4. line splitting at wide gaps (`SPLIT_GAP=3.0`, `CELL_GAP=1.8`,
`GUTTER_GAP=1.6`, :44β47) and gutter detection (:342).
5. `reading_order()` on `order.onnx` (`feats [1,N,16]`, outputs pairwise salience
`a`, block class `cls` [10 classes], same-block `same`, table t-heads `tab`)
then layout (`layout_post.py: page_blocks`, `page_markdown`).
MIN_CONF=0.5 means **a busy page with low-quality Devanagari can silently
emit empty lines** in Experiment A β compare-on-GT-boxes (rec-lines mode)
sidesteps this entirely, which is why Experiment A does both full-pipeline
and rec-lines scoring.
## 5. Licensing [V]
- **Model weights** (`models/indra-ocr-v3.1.1/LICENSE`): **AI PUBS OPEN RAIL-M
(MODIFIED), v0.1, 2023-03-02**. Section II ΒΆ36 states verbatim: *"Use may
include creating any content with, fine-tuning, updating, running, training,
evaluating and/or re-parametrizing the Model"* β **fine-tuning is explicitly
licensed**. ΒΆ5 attaches Attachment A use-based restrictions (standard
OpenRAIL misuse list β weapons/illegal/defamation; OCR and document
processing is untouched) and requires passing the same restrictions to
distributees; ΒΆ4 permits hosting/redistribution with notice. Derivative
weights (fine-tuned checkpoints, distillations) fall under the same license.
- **Code** (`indra/LICENSE`): **Apache-2.0** β reuse, modification,
redistribution of the inference package are standard.
- Practical consequence: our fine-tuned recognizer (Experiment B) is legally a
"Derivative of the Model" under OpenRAIL-M β internal deployment and
evaluation are unrestricted; redistribution requires keeping the license +
restriction notice. Full matrix incl. Nepali datasets: `Β§12` of
`INDRA_EXPERIMENTS.md` (dataset side **[B]** β needs `docs/DATA_LICENSING.md`
on the remote machine).
## 6. Closing the prior report's [U] items
| Prior unknown | Status | Evidence |
|---|---|---|
| exact recognizer arch (RNN? CNN?) | **[V] + [X]**: pure CNN+CTC, NO RNN/attention | Β§1.1, node dump |
| timestep stride | **[X]**: 4, not 8 | Β§1.3 (graph constants + inc.timings) |
| output contract (best/pmax docstrings vs solo head) | **[V]**: `ids`,`conf`,`lengths` names; truncation mandatory | Β§1.4 |
| safetensors β ONNX mapping (esp. depthwise `onnx::Conv_369β¦397`) | **[V]**: 108 tensors = 108 initializers, 1,236,366 params both ways; dw weights are `rec/onnx::Conv_369` [128,1,9] + `β¦370` [128] bias pairs | `inspect_models.py` |
| Devanagari coverage claim | **[V]**: 128/128 codepoints in vocab | `inspect_vocab.py` |
| What re-export must reproduce | **[V]**: mask + per-block re-zeroing + SE-valid-norm + decode tail; `lengths = ceil(widths/4)` clip W/4 | Β§1.2, `inspect_rec_mask.py` |
| Confidence semantics of padded steps | **[X]**: head-on-zeros β blank conf 0.78; caller truncates | Β§1.2(3) |
| vocab[token i+1] β blank 0 | **[V]** | `vocab.py:3,207,241β242` |
| rec input normalization | **[V]**: ink map only (0 paper, 1 ink) per `crop_line`, no mean/std | `rec_preproc.py:1β10` |
| enables fine-tune? | **[V] yes for rec only** (weights exist; no training code; CTC contract reconstructed) | Β§3, `Β§B` doc |
| full pipeline runs on Nepali test image | **[V] smoke**: synthetic shaped page, 10/10 lines, CER 2.6β3.5% | `results/smoke/` |
| what differs in real (non-synth) Nepali | **[B]** frozen remote sets + baseline | Experiment A |
| trained det/order weights for fine-tune | **[V] not shipped** (ONNX only) | Β§3 |
## 7. Open items (not closed by this investigation)
1. **[B]** Zero-shot numbers on the frozen Nepali eval sets (needs `khalti-t4`).
2. **[B]** Head-to-head vs PP-OCRv5 Mobile Devanagari *on the same crops* β the
remote machine holds the paddle recognizer + eval harness. The local
smoke CER of 2.6β3.5% is on a synthetic page with a 32 px font and is
*not* a benchmark.
3. **[B]** Licensing matrix final values need `docs/DATA_LICENSING.md` and
`docs/DEVANAGARI_RESOURCES_INVENTORY.md` (remote).
4. FX-detector quality on Devanagari (qubee endpoints, aggressively slanted
conjuncts) is unknowable without the det model being exercised on real
scans β that's part of Experiment A full-pipeline.
5. The *root* training repo (referenced by name in the closed code) is not
accessible; Archived recipe unknowns: batch sizes, LR schedule, data mix.
## 8. Reproduce everything
```bash
cd /mnt/c/Users/Dell/Desktop/OCR_INDRA # or wherever the repo is
PY=/tmp/opencode/indra-lib # pip --target deps
export PYTHONPATH="$PWD/indra:$PY"
python3 tools/indra_eval/inspect_vocab.py # vocab + Devanagari coverage
python3 tools/indra_eval/inspect_models.py # graph I/O + param count
python3 tools/indra_eval/inspect_rec_nodes.py # full node dump + smoke run
python3 tools/indra_eval/inspect_rec_mask.py # mask semantics + lengths table
python3 tools/indra_eval/make_synth_page.py results/smoke # synth page + GT
python3 tools/indra_eval/eval_indra.py predict results/smoke/synth_nepali_page.png \
--boxes results/smoke/synth_nepali_gt.jsonl --jsonl results/smoke/rec_only.jsonl
python3 tools/indra_eval/eval_indra.py predict results/smoke/synth_nepali_page.png \
--jsonl results/smoke/full_run.jsonl --print-text
python3 tools/indra_eval/eval_indra.py cer results/smoke/synth_nepali_record.jsonl \
results/smoke/rec_only.jsonl --detail results/smoke/cer_rec_only.json --rec-only
python3 tools/indra_eval/eval_indra.py cer results/smoke/synth_nepali_record.jsonl \
results/smoke/full_run.jsonl --detail results/smoke/cer_full.json
```
Observed (this machine, WSL, CPU, 8 cores, 3 GB RAM): rec-only CER 3.49% /
WER 15.79%; full-pipeline CER 2.62% / WER 13.16%; 10/10 lines found;
mean_conf 0.962β0.963. Timings on this box are meaningless (det β 18.6 min
β machine-starved; bench on the T4).
|