The note quoted a rounded zero for byte fallback and claimed the README
matched model_card.json field for field. The README now reports counts
where the card reports rates, so the note says that instead of asserting
an identity that no longer holds.
docs: split the paired dashes in the corpus-licence sentence
- Mon's byte fallback is 20 tokens in 486,631, which rounds to a
two-decimal zero and reads as "never"; Burmese's zero is a real zero.
A percentage renders the two identically.
- Counts recomputed over the whole val split against the shipped
tokenizer.json, using scripts/train_tokenizer.py's own split and
fallback-id set. They reproduce the rounded rates in model_card.json
exactly: 20/486,631, 0/24,942, 96/453,350, 116/59,440.
fix: stop declaring MIT over an artifact this project cannot license
The front matter said `license: mit` and there was no LICENSE file. The
vocabulary was fitted to a corpus this project neither collected nor
owns, so tokenizer.json is a derived work and its terms follow that
corpus — largest source CC BY-SA Mon Wikipedia, second largest a news
archive with no established licence.
MIT still covers the card and the configs. Nothing here grants rights
over tokenizer.json, because none were obtained.
fix: restate the algorithm comparison from one run
1.524 chars/token against 4.507, both at 48,000 pieces over the 29,600-line
Mon validation split. The previous pair crossed two runs on two corpora.
tokenizer.json is untouched and still hashes to 34d18153.
- 392 of 397 distinct characters over all 29,600 Mon validation lines
- record lines_measured, so the sampling is auditable
The card said 100% with sampling labelled "whole split, no cap", but the
figure came from the first 5,000 lines. tokenizer.json is unchanged and
still hashes to 34d181532eee7e6754bfbad693753ee4660bdda00eb36a2ccc7cd2f372c71282,
byte-identical to the packaged copy; only the card was re-measured.
- remove AUDIT-2026-08-08.md; it is an internal document with severities
and ratings, and this repo is a published artifact surface
- move it to mon_tokenizer/docs/AUDIT-2026-08-08-hf-artifact.md, beside the
package audit it belongs with
- ignore AUDIT-*, FINDINGS-*, LESSONS*, RETROSPECTIVE* so the next one
cannot land here by habit
feat: migrate the artifact from SentencePiece to tokenizers JSON
- replace tokenizer.model and mon_tokenizer.vocab with tokenizer.json,
which carries the normalizer inside the artifact
- add model_card.json so the Hub copy and the PyPI package assert the
same machine-readable facts
- artifact is byte identical to the copy in mon-tokenizer 1.0.0
(sha256 34d18153...), which is what the package preflight checks
- 4.6 MB is under the Hub's 10 MB threshold, and HF's own default
.gitattributes template does not track *.json, so the current patterns
are the convention rather than a gap
- LFS would add a failure mode for plain clones without removing one,
since from_pretrained resolves either way
- keep the original text under a details block; a retracted finding is
more useful than one that quietly disappears
- add the per-repo findings from the cross-repo audit of the Mon
toolchain, with severity, location, quoted evidence and confidence
- every finding was reproduced against the source before being written
down; unreproducible ones were dropped rather than softened
- include the ratings, the roadmap, and what was deliberately not
recommended
feat: update tokenizer artifacts with 41.4M character corpus
