Neuratal commited on
Commit
4b750ba
·
verified ·
1 Parent(s): 1dd6096

Publish NTokenizer v0.3-polished (model + model card + manifest fingerprints)

Browse files
.gitattributes CHANGED
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ ntokenizer-0.3-256k.ntok filter=lfs diff=lfs merge=lfs -text
NOTICE ADDED
@@ -0,0 +1,38 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # NOTICE — Neuratal NTokenizer v0.3-polished model
2
+
3
+ This NOTICE covers the attribution and ShareAlike terms attached to the model
4
+ artifact `ntokenizer-0.3-256k.ntok` and to the `tokenizer.json` export included in
5
+ this repository.
6
+
7
+ 1. The model is a machine-learned derivative of a 1.96 GB training corpus dominated
8
+ by CC BY-SA/GFDL text from Wikimedia projects (simplewiki, trwiki, hiwiki,
9
+ eswikisource, arwikisource, enwikibooks, jawikisource, kowikisource, zhwiktionary,
10
+ jawiktionary, kowiktionary, trwiktionary and others).
11
+
12
+ 2. It is therefore distributed under **CC BY-SA 4.0**, not Apache-2.0.
13
+ ShareAlike applies to adaptations of the model.
14
+
15
+ 3. Attribution pattern: `"<project> contributors, <license>, <URL>"`. Example:
16
+ "Simple English Wikipedia contributors, CC BY-SA 4.0, https://simple.wikipedia.org".
17
+ The complete per-source table (dumps, licenses, URLs, SHA-256) is in
18
+ `data/v0.2/LICENSES.md` of the source repository
19
+ (<https://github.com/Neuratal/neuratal-tokenizer>) and summarized in this repo's
20
+ `training_manifest.json` (fingerprints only; no corpus text included).
21
+
22
+ 4. CC BY sources (arXiv per-article CC BY 4.0 — 187 papers; enwikinews CC BY
23
+ 2.5/4.0; Tatoeba CC BY 2.0 FR) require attribution, no ShareAlike.
24
+
25
+ 5. Public-domain sources (Project Gutenberg) and permissively licensed code
26
+ (CPython PSF 2.0, TypeScript Apache-2.0, Go BSD-3-Clause, Rust MIT OR Apache-2.0)
27
+ impose only notice retention, satisfied by this NOTICE.
28
+
29
+ 6. No training corpus is distributed. No benchmark file is distributed. The
30
+ benchmark used in evaluation is a development/validation set.
31
+
32
+ 7. Founder and creator: Rüzgar Erdem. Neuratal is an independent AI research and
33
+ software project; it is not claimed to be an incorporated company. The model
34
+ is an independent implementation; no third-party tokenizer vocabulary, merge
35
+ table, or implementation was used.
36
+
37
+ 8. If you cannot comply with CC BY-SA 4.0, do not redistribute the model.
38
+ The NTokenizer source code remains MIT OR Apache-2.0.
README.md ADDED
@@ -0,0 +1,100 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - multilingual
4
+ license: cc-by-sa-4.0
5
+ tags:
6
+ - tokenizer
7
+ - bpe
8
+ - byte-level-bpe
9
+ - multilingual
10
+ - nlp
11
+ - rust
12
+ - text-tokenization
13
+ - neuratal
14
+ ---
15
+
16
+ # Neuratal NTokenizer v0.3-polished
17
+
18
+ **NTokenizer** is an independently developed, multilingual **byte-level BPE tokenizer** written from scratch in Rust by **Neuratal** (founder and creator: **Rüzgar Erdem**). NTokenizer is a **tokenizer, not a language model**.
19
+
20
+ - Repository (source, Rust + Python bindings): <https://github.com/Neuratal/neuratal-tokenizer>
21
+ - Release: https://github.com/Neuratal/neuratal-tokenizer/releases/tag/v0.3.0
22
+ - Organization site: none published yet — unverified look-alike domains should not be trusted.
23
+
24
+ ## Model details
25
+
26
+ - **Name:** Neuratal NTokenizer v0.3-polished
27
+ - **Organization:** Neuratal (independent AI research and software project; not claimed to be an incorporated company)
28
+ - **Creator:** Rüzgar Erdem
29
+ - **Architecture:** byte-level Byte Pair Encoding (BPE), single deterministic merge table
30
+ - **Vocabulary:** 262,144 (256 base byte tokens + 261,881 learned merges + 7 special tokens: `<|bos|>`, `<|eos|>`, `<|pad|>`, `<|unk|>`, `<|system|>`, `<|user|>`, `<|assistant|>`)
31
+ - **Native format:** `.ntok` v3 (NTOK magic + CRC32 + payload)
32
+ - **Implementation language:** Rust (reference encoder/decoder); Python bindings via ctypes
33
+ - **Intended use:** text tokenization for language-model training and inference pipelines; research tooling
34
+ - **Out of scope:** serving as a language model, or as a normalization/rewriter of Unicode text
35
+
36
+ ## Description
37
+
38
+ NTokenizer converts text into token IDs (`encode`) and reconstructs text/raw bytes from IDs (`decode`). The byte-level design guarantees that **any** byte sequence — including invalid UTF-8 — is recoverable exactly: `decode_bytes(encode_bytes(x)) == x` always holds.
39
+
40
+ Highlights of this release (v0.3-polished, 2026-11-15):
41
+
42
+ - A Unicode data-loss bug fixed: combining marks at the start of a string were previously dropped by the production pre-tokenizer family (MathSymbolAware) and by the leading-whitespace pre-tokenizer. They now attach correctly and round-trip byte-exactly. Covered by a regression suite (30 adversarial samples × 12 pre-tokenizer modes; ~40 samples × 2 production modes; 136/136 Rust tests pass).
43
+ - Encoding throughput measured ~1.6–1.7× the pre-polish implementation on the same machine (interleaved A/B; 8.9–9.0 → 14.5–15.9 MB/s single-thread; 44.6–50.6 → 92.3–96.6 MB/s at 8 threads; peak encode RSS 415 → 360 MB). These are project test-machine measurements, not universal guarantees.
44
+ - Normalization is **off by default** (`Normalization::None`). NFC/NFD/NFKC/NFKD are explicit opt-ins and may alter bytes by design.
45
+
46
+ ## Training and data
47
+
48
+ - Corpus: **1.96 GB, 10,555 files**, assembled exclusively from the v0.2 leakage-audited clean tree (`data/v0.2/clean/train`). No benchmark file, byte or near-duplicate was included (leakage audit documented in the repository reports).
49
+ - Composition (bytes): Arabic 15.0%, Turkish 14.6%, Hindi 11.7%, English 11.2%, STEM 9.6%, Korean 9.5%, Spanish 7.5%, Programming 7.2%, Japanese 6.9%, Chinese 6.6%, Other 0.2%.
50
+ - Sources: Wikimedia projects (the largest share, mostly CC BY-SA/GFDL), arXiv per-article **CC BY 4.0** (187 papers only; the rest were quarantined in v0.2), Project Gutenberg (public domain), enwikinews (CC BY, not ShareAlike), Tatoeba (CC BY 2.0 FR), permissively licensed code (CPython PSF 2.0, TypeScript Apache-2.0, Go BSD-3, Rust MIT/Apache-2.0).
51
+ - Recipe: `--pretokenizer math --min-freq 2 --min-word-freq 2 --seed 42`, 256K vocabulary.
52
+ - `training_manifest.json` in this repo fingerprints every training file (path, bytes, sha256, source, license). No training corpora are distributed.
53
+
54
+ ## Evaluation
55
+
56
+ Development benchmark (544 files, 207,712,944 bytes; **development/validation set — not a pristine blind test set**):
57
+
58
+ | Metric | v0.3-polished |
59
+ |---|---:|
60
+ | Overall byte-weighted B/T | **5.5542** |
61
+ | Equal-category B/T | **5.1056** |
62
+ | Byte-exact decode round-trip | 544/544 files |
63
+ | Used vocabulary (dev benchmark) | 245,334 / 262,144 (93.59%) |
64
+
65
+ Historical v0.3 record: 37,397,108 tokens / 5.5543 / 5.1057. The polished-tree figure is 37,397,319 / 5.5542; the +211-token difference (+0.00056%, 24 files) predates the polishing cycle and is disclosed in the repository.
66
+
67
+ Reference points (different tokenizers, same benchmark, same byte counts): o200k_base 4.0751 B/T; NTokenizer v0.2 4.9290 B/T.
68
+
69
+ **B/T measures compression efficiency, not language-model quality.** No downstream 50M-language-model evaluation exists yet; do not read this as proof of downstream superiority. Decode throughput ~160–278 MB/s; encode throughput as above (test machine; interleaved A/B).
70
+
71
+ Also disclosed:
72
+ - Chinese 3.54 B/T and Japanese 5.01 B/T remain imperfect; CJK-chunking candidates were rejected (Japanese regression).
73
+ - Programming 4.10 B/T, English 4.46 B/T.
74
+ - A 2026-11-16 research cycle (v0.4-v2, 64K pilots) found a corpus-expansion arm with +0.55% overall B/T but guardrail regressions; **nothing from that cycle was promoted** and no v0.4-v2 model exists.
75
+ - The 6.00 B/T aspiration was not reached; this release is the polished v0.3 model.
76
+
77
+ ## Compatibility
78
+
79
+ - Native `.ntok` (this repository): fully validated (`ntokenizer validate`), byte-exact both directions, special tokens isolated.
80
+ - `tokenizer.json` (Hugging Face export, included here as a convenience): loads via `tokenizers` 0.23; byte-exact encode→decode in both directions; **token-ID parity is only 3/16** (HF ByteLevel pre-tokenization cannot express MathSymbolAware). Decode-exact ≠ ID-parity. The native format is authoritative. Do not claim full HF parity.
81
+
82
+ ## Limitations
83
+
84
+ - The development benchmark is not a blind test; no untouched test set exists in this environment.
85
+ - Other-languages bucket (0.16 MB) lags o200k_base.
86
+ - ASAN-instrumented fuzzing is blocked by a pre-existing linker issue (not claimed). Load/round-trip fuzzing ran ~1.0M combined cases with zero crashes.
87
+ - No fine-grained safety, bias, or fairness evaluation is provided.
88
+ - Do not redistribute this model under Apache-2.0 without respecting ShareAlike.
89
+
90
+ ## License and attribution
91
+
92
+ - **Source code**: MIT OR Apache-2.0.
93
+ - **This model artifact and the HF export**: **CC BY-SA 4.0**. The model is a machine-learned derivative of mostly CC BY-SA/GFDL text (Wikimedia) and CC BY material (arXiv 187 papers; enwikinews; Tatoeba). ShareAlike applies to adaptations. Attribution follows the pattern `"<project> contributors, <license>, <URL>"` (e.g. "Simple English Wikipedia contributors, CC BY-SA 4.0, https://simple.wikipedia.org"). Full attribution set: see `NOTICE` in the GitHub repository.
94
+ - Nothing here licenses the training corpora themselves; they are not distributed.
95
+
96
+ ## Links
97
+
98
+ - Source: <https://github.com/Neuratal/neuratal-tokenizer>
99
+ - Release: <https://github.com/Neuratal/neuratal-tokenizer/releases/tag/v0.3.0>
100
+ - Citation: see `CITATION.cff` in the repository (author Rüzgar Erdem, Neuratal, 2026)
SHA256SUMS ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ 1f0d4ad7c7fb609001b2af4d9741ec5b8a5d7686fa3b92eb8eefd057fea936d9 ntokenizer-0.3-256k.ntok
2
+ a4719b2737820deb38990be82f5d0f54f7ae0eb35d746e73fe1abeecb9fc937b tokenizer.json
3
+ d9f480332ef6d10e614965071f4f7003757af4d7072971d8632ef8a4fd6984bb tokenizer_config.json
4
+ eeace98df5c6a98b278aa50da80c2fd8d63957fce0d31dfa87e7dba90f545625 special_tokens_map.json
5
+ 2b5a5c523d28b76aa11c371c312814be400cce82d95a0ec37c989b0a7a2cc186 training_manifest.json
6
+ 6bc8c77fefb4e7dccda661f4ca331f21d6abed2f595c4f27703d80732de8dff9 NOTICE
7
+ 13fbb49d075bc91d582bff7329350b6ccc3475bcb17f3800beef05760e6410db README.md
ntokenizer-0.3-256k.ntok ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1f0d4ad7c7fb609001b2af4d9741ec5b8a5d7686fa3b92eb8eefd057fea936d9
3
+ size 23403000
special_tokens_map.json ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": "<|bos|>",
3
+ "eos_token": "<|eos|>",
4
+ "pad_token": "<|pad|>",
5
+ "unk_token": "<|unk|>"
6
+ }
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json ADDED
@@ -0,0 +1,8 @@
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": "<|bos|>",
3
+ "eos_token": "<|eos|>",
4
+ "name_or_path": "ntokenizer-hf-export",
5
+ "pad_token": "<|pad|>",
6
+ "tokenizer_class": "PreTrainedTokenizerFast",
7
+ "unk_token": "<|unk|>"
8
+ }
training_manifest.json ADDED
The diff for this file is too large to render. See raw diff