Publish NTokenizer v0.3-polished (model + model card + manifest fingerprints)
Browse files- .gitattributes +1 -0
- NOTICE +38 -0
- README.md +100 -0
- SHA256SUMS +7 -0
- ntokenizer-0.3-256k.ntok +3 -0
- special_tokens_map.json +6 -0
- tokenizer.json +0 -0
- tokenizer_config.json +8 -0
- training_manifest.json +0 -0
.gitattributes
CHANGED
|
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
+
ntokenizer-0.3-256k.ntok filter=lfs diff=lfs merge=lfs -text
|
NOTICE
ADDED
|
@@ -0,0 +1,38 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# NOTICE — Neuratal NTokenizer v0.3-polished model
|
| 2 |
+
|
| 3 |
+
This NOTICE covers the attribution and ShareAlike terms attached to the model
|
| 4 |
+
artifact `ntokenizer-0.3-256k.ntok` and to the `tokenizer.json` export included in
|
| 5 |
+
this repository.
|
| 6 |
+
|
| 7 |
+
1. The model is a machine-learned derivative of a 1.96 GB training corpus dominated
|
| 8 |
+
by CC BY-SA/GFDL text from Wikimedia projects (simplewiki, trwiki, hiwiki,
|
| 9 |
+
eswikisource, arwikisource, enwikibooks, jawikisource, kowikisource, zhwiktionary,
|
| 10 |
+
jawiktionary, kowiktionary, trwiktionary and others).
|
| 11 |
+
|
| 12 |
+
2. It is therefore distributed under **CC BY-SA 4.0**, not Apache-2.0.
|
| 13 |
+
ShareAlike applies to adaptations of the model.
|
| 14 |
+
|
| 15 |
+
3. Attribution pattern: `"<project> contributors, <license>, <URL>"`. Example:
|
| 16 |
+
"Simple English Wikipedia contributors, CC BY-SA 4.0, https://simple.wikipedia.org".
|
| 17 |
+
The complete per-source table (dumps, licenses, URLs, SHA-256) is in
|
| 18 |
+
`data/v0.2/LICENSES.md` of the source repository
|
| 19 |
+
(<https://github.com/Neuratal/neuratal-tokenizer>) and summarized in this repo's
|
| 20 |
+
`training_manifest.json` (fingerprints only; no corpus text included).
|
| 21 |
+
|
| 22 |
+
4. CC BY sources (arXiv per-article CC BY 4.0 — 187 papers; enwikinews CC BY
|
| 23 |
+
2.5/4.0; Tatoeba CC BY 2.0 FR) require attribution, no ShareAlike.
|
| 24 |
+
|
| 25 |
+
5. Public-domain sources (Project Gutenberg) and permissively licensed code
|
| 26 |
+
(CPython PSF 2.0, TypeScript Apache-2.0, Go BSD-3-Clause, Rust MIT OR Apache-2.0)
|
| 27 |
+
impose only notice retention, satisfied by this NOTICE.
|
| 28 |
+
|
| 29 |
+
6. No training corpus is distributed. No benchmark file is distributed. The
|
| 30 |
+
benchmark used in evaluation is a development/validation set.
|
| 31 |
+
|
| 32 |
+
7. Founder and creator: Rüzgar Erdem. Neuratal is an independent AI research and
|
| 33 |
+
software project; it is not claimed to be an incorporated company. The model
|
| 34 |
+
is an independent implementation; no third-party tokenizer vocabulary, merge
|
| 35 |
+
table, or implementation was used.
|
| 36 |
+
|
| 37 |
+
8. If you cannot comply with CC BY-SA 4.0, do not redistribute the model.
|
| 38 |
+
The NTokenizer source code remains MIT OR Apache-2.0.
|
README.md
ADDED
|
@@ -0,0 +1,100 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
language:
|
| 3 |
+
- multilingual
|
| 4 |
+
license: cc-by-sa-4.0
|
| 5 |
+
tags:
|
| 6 |
+
- tokenizer
|
| 7 |
+
- bpe
|
| 8 |
+
- byte-level-bpe
|
| 9 |
+
- multilingual
|
| 10 |
+
- nlp
|
| 11 |
+
- rust
|
| 12 |
+
- text-tokenization
|
| 13 |
+
- neuratal
|
| 14 |
+
---
|
| 15 |
+
|
| 16 |
+
# Neuratal NTokenizer v0.3-polished
|
| 17 |
+
|
| 18 |
+
**NTokenizer** is an independently developed, multilingual **byte-level BPE tokenizer** written from scratch in Rust by **Neuratal** (founder and creator: **Rüzgar Erdem**). NTokenizer is a **tokenizer, not a language model**.
|
| 19 |
+
|
| 20 |
+
- Repository (source, Rust + Python bindings): <https://github.com/Neuratal/neuratal-tokenizer>
|
| 21 |
+
- Release: https://github.com/Neuratal/neuratal-tokenizer/releases/tag/v0.3.0
|
| 22 |
+
- Organization site: none published yet — unverified look-alike domains should not be trusted.
|
| 23 |
+
|
| 24 |
+
## Model details
|
| 25 |
+
|
| 26 |
+
- **Name:** Neuratal NTokenizer v0.3-polished
|
| 27 |
+
- **Organization:** Neuratal (independent AI research and software project; not claimed to be an incorporated company)
|
| 28 |
+
- **Creator:** Rüzgar Erdem
|
| 29 |
+
- **Architecture:** byte-level Byte Pair Encoding (BPE), single deterministic merge table
|
| 30 |
+
- **Vocabulary:** 262,144 (256 base byte tokens + 261,881 learned merges + 7 special tokens: `<|bos|>`, `<|eos|>`, `<|pad|>`, `<|unk|>`, `<|system|>`, `<|user|>`, `<|assistant|>`)
|
| 31 |
+
- **Native format:** `.ntok` v3 (NTOK magic + CRC32 + payload)
|
| 32 |
+
- **Implementation language:** Rust (reference encoder/decoder); Python bindings via ctypes
|
| 33 |
+
- **Intended use:** text tokenization for language-model training and inference pipelines; research tooling
|
| 34 |
+
- **Out of scope:** serving as a language model, or as a normalization/rewriter of Unicode text
|
| 35 |
+
|
| 36 |
+
## Description
|
| 37 |
+
|
| 38 |
+
NTokenizer converts text into token IDs (`encode`) and reconstructs text/raw bytes from IDs (`decode`). The byte-level design guarantees that **any** byte sequence — including invalid UTF-8 — is recoverable exactly: `decode_bytes(encode_bytes(x)) == x` always holds.
|
| 39 |
+
|
| 40 |
+
Highlights of this release (v0.3-polished, 2026-11-15):
|
| 41 |
+
|
| 42 |
+
- A Unicode data-loss bug fixed: combining marks at the start of a string were previously dropped by the production pre-tokenizer family (MathSymbolAware) and by the leading-whitespace pre-tokenizer. They now attach correctly and round-trip byte-exactly. Covered by a regression suite (30 adversarial samples × 12 pre-tokenizer modes; ~40 samples × 2 production modes; 136/136 Rust tests pass).
|
| 43 |
+
- Encoding throughput measured ~1.6–1.7× the pre-polish implementation on the same machine (interleaved A/B; 8.9–9.0 → 14.5–15.9 MB/s single-thread; 44.6–50.6 → 92.3–96.6 MB/s at 8 threads; peak encode RSS 415 → 360 MB). These are project test-machine measurements, not universal guarantees.
|
| 44 |
+
- Normalization is **off by default** (`Normalization::None`). NFC/NFD/NFKC/NFKD are explicit opt-ins and may alter bytes by design.
|
| 45 |
+
|
| 46 |
+
## Training and data
|
| 47 |
+
|
| 48 |
+
- Corpus: **1.96 GB, 10,555 files**, assembled exclusively from the v0.2 leakage-audited clean tree (`data/v0.2/clean/train`). No benchmark file, byte or near-duplicate was included (leakage audit documented in the repository reports).
|
| 49 |
+
- Composition (bytes): Arabic 15.0%, Turkish 14.6%, Hindi 11.7%, English 11.2%, STEM 9.6%, Korean 9.5%, Spanish 7.5%, Programming 7.2%, Japanese 6.9%, Chinese 6.6%, Other 0.2%.
|
| 50 |
+
- Sources: Wikimedia projects (the largest share, mostly CC BY-SA/GFDL), arXiv per-article **CC BY 4.0** (187 papers only; the rest were quarantined in v0.2), Project Gutenberg (public domain), enwikinews (CC BY, not ShareAlike), Tatoeba (CC BY 2.0 FR), permissively licensed code (CPython PSF 2.0, TypeScript Apache-2.0, Go BSD-3, Rust MIT/Apache-2.0).
|
| 51 |
+
- Recipe: `--pretokenizer math --min-freq 2 --min-word-freq 2 --seed 42`, 256K vocabulary.
|
| 52 |
+
- `training_manifest.json` in this repo fingerprints every training file (path, bytes, sha256, source, license). No training corpora are distributed.
|
| 53 |
+
|
| 54 |
+
## Evaluation
|
| 55 |
+
|
| 56 |
+
Development benchmark (544 files, 207,712,944 bytes; **development/validation set — not a pristine blind test set**):
|
| 57 |
+
|
| 58 |
+
| Metric | v0.3-polished |
|
| 59 |
+
|---|---:|
|
| 60 |
+
| Overall byte-weighted B/T | **5.5542** |
|
| 61 |
+
| Equal-category B/T | **5.1056** |
|
| 62 |
+
| Byte-exact decode round-trip | 544/544 files |
|
| 63 |
+
| Used vocabulary (dev benchmark) | 245,334 / 262,144 (93.59%) |
|
| 64 |
+
|
| 65 |
+
Historical v0.3 record: 37,397,108 tokens / 5.5543 / 5.1057. The polished-tree figure is 37,397,319 / 5.5542; the +211-token difference (+0.00056%, 24 files) predates the polishing cycle and is disclosed in the repository.
|
| 66 |
+
|
| 67 |
+
Reference points (different tokenizers, same benchmark, same byte counts): o200k_base 4.0751 B/T; NTokenizer v0.2 4.9290 B/T.
|
| 68 |
+
|
| 69 |
+
**B/T measures compression efficiency, not language-model quality.** No downstream 50M-language-model evaluation exists yet; do not read this as proof of downstream superiority. Decode throughput ~160–278 MB/s; encode throughput as above (test machine; interleaved A/B).
|
| 70 |
+
|
| 71 |
+
Also disclosed:
|
| 72 |
+
- Chinese 3.54 B/T and Japanese 5.01 B/T remain imperfect; CJK-chunking candidates were rejected (Japanese regression).
|
| 73 |
+
- Programming 4.10 B/T, English 4.46 B/T.
|
| 74 |
+
- A 2026-11-16 research cycle (v0.4-v2, 64K pilots) found a corpus-expansion arm with +0.55% overall B/T but guardrail regressions; **nothing from that cycle was promoted** and no v0.4-v2 model exists.
|
| 75 |
+
- The 6.00 B/T aspiration was not reached; this release is the polished v0.3 model.
|
| 76 |
+
|
| 77 |
+
## Compatibility
|
| 78 |
+
|
| 79 |
+
- Native `.ntok` (this repository): fully validated (`ntokenizer validate`), byte-exact both directions, special tokens isolated.
|
| 80 |
+
- `tokenizer.json` (Hugging Face export, included here as a convenience): loads via `tokenizers` 0.23; byte-exact encode→decode in both directions; **token-ID parity is only 3/16** (HF ByteLevel pre-tokenization cannot express MathSymbolAware). Decode-exact ≠ ID-parity. The native format is authoritative. Do not claim full HF parity.
|
| 81 |
+
|
| 82 |
+
## Limitations
|
| 83 |
+
|
| 84 |
+
- The development benchmark is not a blind test; no untouched test set exists in this environment.
|
| 85 |
+
- Other-languages bucket (0.16 MB) lags o200k_base.
|
| 86 |
+
- ASAN-instrumented fuzzing is blocked by a pre-existing linker issue (not claimed). Load/round-trip fuzzing ran ~1.0M combined cases with zero crashes.
|
| 87 |
+
- No fine-grained safety, bias, or fairness evaluation is provided.
|
| 88 |
+
- Do not redistribute this model under Apache-2.0 without respecting ShareAlike.
|
| 89 |
+
|
| 90 |
+
## License and attribution
|
| 91 |
+
|
| 92 |
+
- **Source code**: MIT OR Apache-2.0.
|
| 93 |
+
- **This model artifact and the HF export**: **CC BY-SA 4.0**. The model is a machine-learned derivative of mostly CC BY-SA/GFDL text (Wikimedia) and CC BY material (arXiv 187 papers; enwikinews; Tatoeba). ShareAlike applies to adaptations. Attribution follows the pattern `"<project> contributors, <license>, <URL>"` (e.g. "Simple English Wikipedia contributors, CC BY-SA 4.0, https://simple.wikipedia.org"). Full attribution set: see `NOTICE` in the GitHub repository.
|
| 94 |
+
- Nothing here licenses the training corpora themselves; they are not distributed.
|
| 95 |
+
|
| 96 |
+
## Links
|
| 97 |
+
|
| 98 |
+
- Source: <https://github.com/Neuratal/neuratal-tokenizer>
|
| 99 |
+
- Release: <https://github.com/Neuratal/neuratal-tokenizer/releases/tag/v0.3.0>
|
| 100 |
+
- Citation: see `CITATION.cff` in the repository (author Rüzgar Erdem, Neuratal, 2026)
|
SHA256SUMS
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
1f0d4ad7c7fb609001b2af4d9741ec5b8a5d7686fa3b92eb8eefd057fea936d9 ntokenizer-0.3-256k.ntok
|
| 2 |
+
a4719b2737820deb38990be82f5d0f54f7ae0eb35d746e73fe1abeecb9fc937b tokenizer.json
|
| 3 |
+
d9f480332ef6d10e614965071f4f7003757af4d7072971d8632ef8a4fd6984bb tokenizer_config.json
|
| 4 |
+
eeace98df5c6a98b278aa50da80c2fd8d63957fce0d31dfa87e7dba90f545625 special_tokens_map.json
|
| 5 |
+
2b5a5c523d28b76aa11c371c312814be400cce82d95a0ec37c989b0a7a2cc186 training_manifest.json
|
| 6 |
+
6bc8c77fefb4e7dccda661f4ca331f21d6abed2f595c4f27703d80732de8dff9 NOTICE
|
| 7 |
+
13fbb49d075bc91d582bff7329350b6ccc3475bcb17f3800beef05760e6410db README.md
|
ntokenizer-0.3-256k.ntok
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:1f0d4ad7c7fb609001b2af4d9741ec5b8a5d7686fa3b92eb8eefd057fea936d9
|
| 3 |
+
size 23403000
|
special_tokens_map.json
ADDED
|
@@ -0,0 +1,6 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"bos_token": "<|bos|>",
|
| 3 |
+
"eos_token": "<|eos|>",
|
| 4 |
+
"pad_token": "<|pad|>",
|
| 5 |
+
"unk_token": "<|unk|>"
|
| 6 |
+
}
|
tokenizer.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
tokenizer_config.json
ADDED
|
@@ -0,0 +1,8 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"bos_token": "<|bos|>",
|
| 3 |
+
"eos_token": "<|eos|>",
|
| 4 |
+
"name_or_path": "ntokenizer-hf-export",
|
| 5 |
+
"pad_token": "<|pad|>",
|
| 6 |
+
"tokenizer_class": "PreTrainedTokenizerFast",
|
| 7 |
+
"unk_token": "<|unk|>"
|
| 8 |
+
}
|
training_manifest.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|