File size: 6,935 Bytes
f9bc0c1 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 | # Sikkim Herald Sunuwar legacy converter status
Status: **converter mapping complete for the proven `koits` / `kirat1`
layout; the selected 2024-04-12 and 2025-01-26 editions pass exact font proof.**
The earlier conclusion that no faithful converter existed is obsolete. The
project derived the legacy-byte map from exact embedded-font evidence and
resolved the final previously uncertain byte, `|`, as U+11BC5 SUNUWAR LETTER
UTTHI. The correction is implemented in `src/ocrtech/sunuwar_legacy.py` and is
packaged by commit `68528a5`. Commit `6a89292` adds the two extraction-level
corrections established by the 2024/2025 audit: empty backslash normalizes to
space, and the one U+201D value whose printed glyph is a colon normalizes to
ASCII colon.
This does not automatically admit every PDF whose resource happens to be named
`F2`, `koits`, or `kirat1`. PDF font names are hints, not identities. Every new
edition must bind the converter route to its full PDF hash and an embedded font
program or glyph-outline/cmap fingerprint before any converted text can be used
as a label.
## Proven mapping
The Mukhia editions investigated here store visible Sunuwar as a legacy Latin
byte stream. The `koits` and `kirat1` font programs were shown to use the same
byte-to-glyph layout. The map contains:
- 10 digits, `0` through `9`, mapped in order to U+11BF0..U+11BF9;
- all observed Sunuwar letter bytes, including the eight second-pass letters
`v q x r u g h j`;
- `| → U+11BC5` UTTHI;
- empty `\\ → U+0020` normalization;
- the hash- and bbox-bound 2025 extraction correction `U+201D → U+003A`;
- an explicit literal-punctuation set; and
- fail-closed reporting for every other byte.
`SUNUWAR_LETTERS_UNCERTAIN` remains as an empty compatibility constant.
`apply_uncertain=True` is therefore a compatibility no-op. The strict API
rejects any unknown byte, including unknown ASCII, instead of allowing a
valid-looking mixed-script string to pass.
## Final UTTHI evidence
The legacy `|` glyph is the flowing open-2 Sikkim form of UTTHI. The evidence
record in the released converter repository reports:
- 2,748 occurrences across eight PDFs;
- a 600-dpi corpus crop compared against separately labelled Sikkim UTTHI and
SHYELE forms;
- normalized largest-component IoU 0.7395 against Sikkim UTTHI and 0.3681
against Sikkim SHYELE; and
- the independently mapped corpus form `t|v|`, which reads `m-u-r-u` under
`t → MA`, `| → UTTHI`, and `v → REU`.
Primary references:
- Richard Ishida's reviewed [Sunuwar orthography notes](https://r12a.github.io/scripts/sunu/suz.html#writingstyles), including labelled Sikkim forms.
- Unicode proposal [L2/21-157R](https://www.unicode.org/L2/L2021/21157r-sunuwar.pdf), which identifies UTTHI `/u/`.
- Unicode font comparison [L2/24-022](https://www.unicode.org/L2/L2024/24022-sunuwar-font-comp.pdf), which documents regional glyph differences.
- The released [`nepal-ttf2utf` evidence record](https://github.com/Ampixa/nepal-ttf2utf/blob/main/docs/EVIDENCE.md) and converter implementation.
The local derivation artifacts remain under
`outputs/sunuwar-map-derivation/`. They include the embedded-font comparisons,
printed-crop checks, and round-trip panels used before the final regional-form
adjudication.
## Selected-edition attestation (2026-07-15 r11b)
The full-hash 2024-04-12 and 2025-01-26 Mukhia editions are 289 days apart.
Their embedded `kirat1` programs have SHA-256
`a56a09338742c2563bcc355b3c701cd96093d813c3fa05582e131331e0783f42`
and `45af39726717d8b6cd211da6a1966e706d85dd60408d4ef3c36e0dad2db3c22d`.
Every observed mapped script code matches the program extracted from the
released converter's cited `sunuwar_01348c63.pdf` derivation source by exact
decomposed outline commands and advance width: 38/38 per edition, 76 summed,
with no mismatch. The reference PDF/font hashes are
`b3c001ff428de3b4e539c88d6d93566ae969a46b9ba947fe680fc1b68eb10456`
and `484550543232f587fab3a5b3e900c922845c5ed44ef7bdd493cace4a5b6bce78`.
The semantic map provenance is pinned to MIT-licensed `nepal-ttf2utf` 0.2.0
commit `a158a8cbdbe230df6849afd9c9736bf31cb9c24b`.
Both editions reconcile at 100% aggregate converter coverage. This attests
font-layout compatibility only. It stores no converted strings and supplies no
native transcription, line label, training row, evaluation row, or CER.
## Required per-edition gate
Before applying this converter to a newly acquired PDF, record all of the
following:
1. Canonical source URL, retrieval time, complete PDF SHA-256, byte count, page
count, and printed issue identity.
2. Embedded font stream SHA-256 and the PDF object/xref that supplies every
target-bearing span.
3. A normalized code/CID-to-outline-command, advance-width, and cmap/ToUnicode
fingerprint. Generic names such as `F2` are insufficient.
4. Exact compatibility of every observed source code with the proven layout.
A partial visual similarity or block-valid output is not sufficient.
5. Raw and shadow-deduplicated source-unit counts reconciled as mapped script,
verified literal, verified blank/whitespace, or unresolved.
6. An aggregate coverage report that stores counts and hashes only. Do not
persist converted strings, crops, labels, or evaluation rows at this stage.
If any target-bearing font object fails the layout proof, classify it as
`unsupported_layout`. Do not call that a converter failure and do not try a
different converter merely because the output looks plausible.
## Unicode validity boundary
The local Python runtime uses Unicode 15.1 and treats the Unicode 16 Sunuwar
characters as unassigned. Consequently, `unicodedata.name`, broad block
membership, NFC, and successful rendering are not acceptance evidence by
themselves. A coverage audit must constrain output to the exact pinned map
target set, reject replacement/private-use/control/residual-Latin values, and
require nonzero mapped Sunuwar content.
These checks establish encoding compatibility, not semantic transcription.
Native pixel/label review remains mandatory before any row can enter a frozen
train or evaluation set.
## Current source status
The bounded 2026-07-15 State Portal catalog contains 11 Mukhia/Sunuwar URLs.
The selected 2024-04-12 and 2025-01-26 editions now have full hashes, printed
identity records, all-page reviewer-1 provenance decisions, and embedded-font
layout proof. They remain unadmitted because independent provenance reviewer 2,
adjudication, and native-script pixel/label review do not exist. They contribute
zero train rows, zero evaluation rows, and no claim evidence.
The old raw-span extractor and corpus builders still contain historical prose
and generic font-name routing. They must not be used for admission until they
are wired to the fingerprint gate and the completed converter. Their old
`RAW_ONLY_NO_FAITHFUL_CONVERTER` wording describes an earlier investigation,
not the present mapping status.
|