docs(card): add the broader held-out test and correct two errors
Browse files- lead with CER 0.0095 on 1,997 synthetic Mon lines, with 0.0061 over all 3,002 lines
- state what the number does not cover: half the errors are pixel-identical spellings, small text, degraded renders
- note that Pyidaungsu-Numbers mislabels three of the old 150 lines (0.0100 becomes 0.0086)
- say yunghkio is not held out: its outlines match the training font TharLon
README.md
CHANGED
|
@@ -23,9 +23,11 @@ It runs in production on the web at [ocr.mondevhub.com](https://ocr.mondevhub.co
|
|
| 23 |
The Android and iOS apps bundle the same artifacts and build from source; neither
|
| 24 |
is in an app store yet.
|
| 25 |
|
| 26 |
-
**Held-out CER 0.
|
| 27 |
-
|
| 28 |
-
|
|
|
|
|
|
|
| 29 |
|
| 30 |
## Use
|
| 31 |
|
|
@@ -100,9 +102,45 @@ and nothing else.
|
|
| 100 |
|
| 101 |
## Evaluation
|
| 102 |
|
| 103 |
-
###
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 104 |
|
| 105 |
Measured 2026-08-16 on the test split, held out from training and scored once.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 106 |
|
| 107 |
| | CER | CI95 | sequence accuracy |
|
| 108 |
|---|---:|---|---:|
|
|
@@ -133,10 +171,13 @@ batch size, so it is a mean, not a tail.
|
|
| 133 |
|
| 134 |
### What the held-out number does not cover
|
| 135 |
|
| 136 |
-
1. **
|
| 137 |
-
|
| 138 |
-
|
| 139 |
-
|
|
|
|
|
|
|
|
|
|
| 140 |
3. **Unseen text, not an unseen renderer.** The same generator and augmentation
|
| 141 |
pipeline produced training and test images, so a renderer defect is learned,
|
| 142 |
validated and tested against identically. **This is the largest caveat on the
|
|
@@ -158,7 +199,7 @@ character it has no class for. 600 lines per arm, rendered 2026-08-15.
|
|
| 158 |
|---|---:|---:|---:|---:|
|
| 159 |
| the 32 trained designs | 600 | 0.1470 | **0.0396** | 73% |
|
| 160 |
| `namkhon`, held out from v3.5 | 600 | 0.0521 | **0.0188** | 64% |
|
| 161 |
-
| `pyidaungsu`, `yunghkio`, held out from v3.5 | 600 | 0.0342 | **0.0051** | 85% |
|
| 162 |
|
| 163 |
v2 predates font-disjoint splits, so the bottom two rows are held out from v3.5
|
| 164 |
only, which tilts the comparison against v3.5. **This is a preview, not an
|
|
|
|
| 23 |
The Android and iOS apps bundle the same artifacts and build from source; neither
|
| 24 |
is in an app store yet.
|
| 25 |
|
| 26 |
+
**Held-out CER 0.0095** on 1,997 synthetic Mon lines (0.0061 over all 3,002 Mon,
|
| 27 |
+
Burmese and English lines), in a typeface design the model never trained on. These
|
| 28 |
+
are synthetic lines, not real pages. Read
|
| 29 |
+
[what that number does not cover](#what-the-held-out-number-does-not-cover) before
|
| 30 |
+
quoting it.
|
| 31 |
|
| 32 |
## Use
|
| 33 |
|
|
|
|
| 102 |
|
| 103 |
## Evaluation
|
| 104 |
|
| 105 |
+
### Broader held-out test (2026-11-11)
|
| 106 |
+
|
| 107 |
+
The same `monocr.onnx` (sha256 `b95de1ea…`), greedy decoding, the same grapheme CER
|
| 108 |
+
and bootstrap as below, on **3,002 synthetic lines** in one typeface design the model
|
| 109 |
+
did not train on (Pyidaungsu, Regular and Bold). Every line's text comes from a
|
| 110 |
+
hash-held-out partition and shares no 20-codepoint run and no 8-word run with any
|
| 111 |
+
training or validation line.
|
| 112 |
+
|
| 113 |
+
| slice | lines | CER | CI95 |
|
| 114 |
+
|---|---:|---:|---|
|
| 115 |
+
| **Mon** | 1,997 | **0.0095** | [0.0079, 0.0111] |
|
| 116 |
+
| All (Mon, Burmese, English) | 3,002 | 0.0061 | [0.0053, 0.0072] |
|
| 117 |
+
| Burmese | 450 | 0.0016 | [0.0008, 0.0025] |
|
| 118 |
+
| English | 451 | 0.0016 | [0.0010, 0.0023] |
|
| 119 |
+
|
| 120 |
+
Three things to read with it:
|
| 121 |
+
|
| 122 |
+
- **Half the errors cannot be read from the pixels.** `င်` and `ၚ်` render as identical
|
| 123 |
+
glyphs in the common Mon fonts, and the corpus spells the same syllable both ways.
|
| 124 |
+
They are 194 of the 391 errors. They are counted as errors here (the same definition
|
| 125 |
+
as the figure below); with the two treated as equal, Mon is 0.0043.
|
| 126 |
+
- **Small text is the weak spot.** Glyphs under 28 px score 0.0121; larger glyphs
|
| 127 |
+
score about 0.004.
|
| 128 |
+
- **Degraded renders are much worse.** With stacked scan-style degradation (harsher
|
| 129 |
+
than the training augmentation) CER is 0.32 (0.31 on Mon), almost all of it on small
|
| 130 |
+
text. Treat that as a known weakness, not as a robustness figure.
|
| 131 |
+
|
| 132 |
+
The PyTorch checkpoint and the ONNX graph gave the same string on all 6,004 images of
|
| 133 |
+
an earlier build of this set. The per-line results and scripts are kept in the
|
| 134 |
+
private training repository. **This is still synthetic data and one typeface design:
|
| 135 |
+
it is not accuracy on photographed or scanned pages.**
|
| 136 |
+
|
| 137 |
+
### Held-out test, 2026-08-16 (superseded as the headline)
|
| 138 |
|
| 139 |
Measured 2026-08-16 on the test split, held out from training and scored once.
|
| 140 |
+
**Correction, 2026-11-11:** `Pyidaungsu-Numbers.ttf` draws the ASCII digits 0-9 with
|
| 141 |
+
Myanmar-digit glyphs, so three of these 150 lines carry digit labels the image does
|
| 142 |
+
not show. With digits treated as equal the overall figure is 0.0086 [0.0049, 0.0128]
|
| 143 |
+
instead of 0.0100. The table below is left as measured.
|
| 144 |
|
| 145 |
| | CER | CI95 | sequence accuracy |
|
| 146 |
|---|---:|---|---:|
|
|
|
|
| 171 |
|
| 172 |
### What the held-out number does not cover
|
| 173 |
|
| 174 |
+
1. **Sample size.** The 2026-08-16 set has n = 150 (interval [0.0056, 0.0147]);
|
| 175 |
+
the 2026-11-11 set has 3,002 lines. Treat the widths as real.
|
| 176 |
+
2. **One typeface design.** Both sets use Pyidaungsu, which was held out from
|
| 177 |
+
training. **`yunghkio` is not a held-out design:** its Myanmar outlines are
|
| 178 |
+
identical to `TharLon`, a training font (63 of 63 probed glyphs), so earlier
|
| 179 |
+
statements here that treated it as held out were wrong. Results that list
|
| 180 |
+
`yunghkio` as held out below are contaminated by that and should not be quoted.
|
| 181 |
3. **Unseen text, not an unseen renderer.** The same generator and augmentation
|
| 182 |
pipeline produced training and test images, so a renderer defect is learned,
|
| 183 |
validated and tested against identically. **This is the largest caveat on the
|
|
|
|
| 199 |
|---|---:|---:|---:|---:|
|
| 200 |
| the 32 trained designs | 600 | 0.1470 | **0.0396** | 73% |
|
| 201 |
| `namkhon`, held out from v3.5 | 600 | 0.0521 | **0.0188** | 64% |
|
| 202 |
+
| `pyidaungsu`, `yunghkio`, listed as held out from v3.5 (`yunghkio` is not: see above) | 600 | 0.0342 | **0.0051** | 85% |
|
| 203 |
|
| 204 |
v2 predates font-disjoint splits, so the bottom two rows are held out from v3.5
|
| 205 |
only, which tilts the comparison against v3.5. **This is a preview, not an
|