janakhpon commited on
Commit
bde40db
·
1 Parent(s): 56ef728

docs(card): add the broader held-out test and correct two errors

Browse files

- lead with CER 0.0095 on 1,997 synthetic Mon lines, with 0.0061 over all 3,002 lines
- state what the number does not cover: half the errors are pixel-identical spellings, small text, degraded renders
- note that Pyidaungsu-Numbers mislabels three of the old 150 lines (0.0100 becomes 0.0086)
- say yunghkio is not held out: its outlines match the training font TharLon

Files changed (1) hide show
  1. README.md +50 -9
README.md CHANGED
@@ -23,9 +23,11 @@ It runs in production on the web at [ocr.mondevhub.com](https://ocr.mondevhub.co
23
  The Android and iOS apps bundle the same artifacts and build from source; neither
24
  is in an app store yet.
25
 
26
- **Held-out CER 0.0100** on 150 unseen lines in a typeface the model never trained
27
- on. Read [what that number does not cover](#what-the-held-out-number-does-not-cover)
28
- before quoting it.
 
 
29
 
30
  ## Use
31
 
@@ -100,9 +102,45 @@ and nothing else.
100
 
101
  ## Evaluation
102
 
103
- ### Held-out test
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
104
 
105
  Measured 2026-08-16 on the test split, held out from training and scored once.
 
 
 
 
106
 
107
  | | CER | CI95 | sequence accuracy |
108
  |---|---:|---|---:|
@@ -133,10 +171,13 @@ batch size, so it is a mean, not a tail.
133
 
134
  ### What the held-out number does not cover
135
 
136
- 1. **n = 150.** The interval is [0.0056, 0.0147]; treat the width as real.
137
- 2. **One typeface.** All 150 lines are Pyidaungsu, held out from training. A
138
- second held-out design, `yunghkio`, is not represented: this split was
139
- generated before that design was set aside.
 
 
 
140
  3. **Unseen text, not an unseen renderer.** The same generator and augmentation
141
  pipeline produced training and test images, so a renderer defect is learned,
142
  validated and tested against identically. **This is the largest caveat on the
@@ -158,7 +199,7 @@ character it has no class for. 600 lines per arm, rendered 2026-08-15.
158
  |---|---:|---:|---:|---:|
159
  | the 32 trained designs | 600 | 0.1470 | **0.0396** | 73% |
160
  | `namkhon`, held out from v3.5 | 600 | 0.0521 | **0.0188** | 64% |
161
- | `pyidaungsu`, `yunghkio`, held out from v3.5 | 600 | 0.0342 | **0.0051** | 85% |
162
 
163
  v2 predates font-disjoint splits, so the bottom two rows are held out from v3.5
164
  only, which tilts the comparison against v3.5. **This is a preview, not an
 
23
  The Android and iOS apps bundle the same artifacts and build from source; neither
24
  is in an app store yet.
25
 
26
+ **Held-out CER 0.0095** on 1,997 synthetic Mon lines (0.0061 over all 3,002 Mon,
27
+ Burmese and English lines), in a typeface design the model never trained on. These
28
+ are synthetic lines, not real pages. Read
29
+ [what that number does not cover](#what-the-held-out-number-does-not-cover) before
30
+ quoting it.
31
 
32
  ## Use
33
 
 
102
 
103
  ## Evaluation
104
 
105
+ ### Broader held-out test (2026-11-11)
106
+
107
+ The same `monocr.onnx` (sha256 `b95de1ea…`), greedy decoding, the same grapheme CER
108
+ and bootstrap as below, on **3,002 synthetic lines** in one typeface design the model
109
+ did not train on (Pyidaungsu, Regular and Bold). Every line's text comes from a
110
+ hash-held-out partition and shares no 20-codepoint run and no 8-word run with any
111
+ training or validation line.
112
+
113
+ | slice | lines | CER | CI95 |
114
+ |---|---:|---:|---|
115
+ | **Mon** | 1,997 | **0.0095** | [0.0079, 0.0111] |
116
+ | All (Mon, Burmese, English) | 3,002 | 0.0061 | [0.0053, 0.0072] |
117
+ | Burmese | 450 | 0.0016 | [0.0008, 0.0025] |
118
+ | English | 451 | 0.0016 | [0.0010, 0.0023] |
119
+
120
+ Three things to read with it:
121
+
122
+ - **Half the errors cannot be read from the pixels.** `င်` and `ၚ်` render as identical
123
+ glyphs in the common Mon fonts, and the corpus spells the same syllable both ways.
124
+ They are 194 of the 391 errors. They are counted as errors here (the same definition
125
+ as the figure below); with the two treated as equal, Mon is 0.0043.
126
+ - **Small text is the weak spot.** Glyphs under 28 px score 0.0121; larger glyphs
127
+ score about 0.004.
128
+ - **Degraded renders are much worse.** With stacked scan-style degradation (harsher
129
+ than the training augmentation) CER is 0.32 (0.31 on Mon), almost all of it on small
130
+ text. Treat that as a known weakness, not as a robustness figure.
131
+
132
+ The PyTorch checkpoint and the ONNX graph gave the same string on all 6,004 images of
133
+ an earlier build of this set. The per-line results and scripts are kept in the
134
+ private training repository. **This is still synthetic data and one typeface design:
135
+ it is not accuracy on photographed or scanned pages.**
136
+
137
+ ### Held-out test, 2026-08-16 (superseded as the headline)
138
 
139
  Measured 2026-08-16 on the test split, held out from training and scored once.
140
+ **Correction, 2026-11-11:** `Pyidaungsu-Numbers.ttf` draws the ASCII digits 0-9 with
141
+ Myanmar-digit glyphs, so three of these 150 lines carry digit labels the image does
142
+ not show. With digits treated as equal the overall figure is 0.0086 [0.0049, 0.0128]
143
+ instead of 0.0100. The table below is left as measured.
144
 
145
  | | CER | CI95 | sequence accuracy |
146
  |---|---:|---|---:|
 
171
 
172
  ### What the held-out number does not cover
173
 
174
+ 1. **Sample size.** The 2026-08-16 set has n = 150 (interval [0.0056, 0.0147]);
175
+ the 2026-11-11 set has 3,002 lines. Treat the widths as real.
176
+ 2. **One typeface design.** Both sets use Pyidaungsu, which was held out from
177
+ training. **`yunghkio` is not a held-out design:** its Myanmar outlines are
178
+ identical to `TharLon`, a training font (63 of 63 probed glyphs), so earlier
179
+ statements here that treated it as held out were wrong. Results that list
180
+ `yunghkio` as held out below are contaminated by that and should not be quoted.
181
  3. **Unseen text, not an unseen renderer.** The same generator and augmentation
182
  pipeline produced training and test images, so a renderer defect is learned,
183
  validated and tested against identically. **This is the largest caveat on the
 
199
  |---|---:|---:|---:|---:|
200
  | the 32 trained designs | 600 | 0.1470 | **0.0396** | 73% |
201
  | `namkhon`, held out from v3.5 | 600 | 0.0521 | **0.0188** | 64% |
202
+ | `pyidaungsu`, `yunghkio`, listed as held out from v3.5 (`yunghkio` is not: see above) | 600 | 0.0342 | **0.0051** | 85% |
203
 
204
  v2 predates font-disjoint splits, so the bottom two rows are held out from v3.5
205
  only, which tilts the comparison against v3.5. **This is a preview, not an