File size: 17,893 Bytes
4b750ba
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
c728925
 
 
 
 
 
 
 
 
 
 
 
a1199ca
 
c728925
a1199ca
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
c728925
 
 
 
 
 
 
4b750ba
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
c728925
4b750ba
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
e1c336c
 
 
 
 
 
 
 
 
 
 
6bd2a64
e1c336c
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
4b750ba
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
---
language:
- multilingual
license: cc-by-sa-4.0
tags:
- tokenizer
- bpe
- byte-level-bpe
- multilingual
- nlp
- rust
- text-tokenization
- neuratal
---

# Neuratal NTokenizer v0.3-polished

**NTokenizer** is an independently developed, multilingual **byte-level BPE tokenizer** written from scratch in Rust by **Neuratal** (founder and creator: **Rüzgar Erdem**). NTokenizer is a **tokenizer, not a language model**.

- Repository (source, Rust + Python bindings): <https://github.com/Neuratal/neuratal-tokenizer>
- Release: https://github.com/Neuratal/neuratal-tokenizer/releases/tag/v0.3.0
- Organization site: none published yet — unverified look-alike domains should not be trusted.

## Model details

- **Name:** Neuratal NTokenizer v0.3-polished
- **Organization:** Neuratal (independent AI research and software project; not claimed to be an incorporated company)
- **Creator:** Rüzgar Erdem
- **Architecture:** byte-level Byte Pair Encoding (BPE), single deterministic merge table
- **Vocabulary:** 262,144 (256 base byte tokens + 261,881 learned merges + 7 special tokens: `<|bos|>`, `<|eos|>`, `<|pad|>`, `<|unk|>`, `<|system|>`, `<|user|>`, `<|assistant|>`)
- **Native format:** `.ntok` v3 (NTOK magic + CRC32 + payload)
- **Implementation language:** Rust (reference encoder/decoder); Python bindings via ctypes
- **Intended use:** text tokenization for language-model training and inference pipelines; research tooling
- **Out of scope:** serving as a language model, or as a normalization/rewriter of Unicode text

## Tokenizer Benchmark — NTokenizer vs Leading Tokenizers

**Development/validation benchmark — not a pristine blind evaluation set.** 544 files, 207,712,944 bytes, identical raw bytes, no normalization, no chat templates for any candidate. `**Bold**` = higher B/T (fewer tokens per input byte) than the comparator. Equal-category B/T = arithmetic mean of the 11 bucket-level B/Ts.

| Tokenizer (reference) | Vocabulary | Tokens | Overall B/T | Equal-category B/T | Token reduction vs o200k |
|---|---:|---:|---:|---:|---:|
| NTokenizer v0.1 | 262,144 | 65,573,261 | 3.1676 | 3.1203 | −28.65% |
| NTokenizer v0.2 | 262,144 | 42,140,674 | 4.9290 | 4.6855 | +17.32% |
| NTokenizer v0.3-original (historical) | 262,144 | 37,397,108 | 5.5543 | 5.1057 | +26.63% |
| **NTokenizer v0.3-polished (official)** | 262,144 | **37,397,319** | **5.5542** | **5.1056** | **+26.63%** |
| NTokenizer v0.4-fail (CJK8, archived) | 262,144 | 37,418,652 | 5.5511 | 5.1031 | +26.59% |
| NTokenizer v0.4-fail (Han4, archived) | 262,144 | 37,406,269 | 5.5529 | 5.1081 | +26.61% |
| o200k_base (tiktoken) | 200,019 | 50,971,306 | 4.0751 | 4.0899 | 0.00% |
| Bloom-7b1 | 250,680 | 54,773,073 | 3.7922 | 4.1584 | +7.4% |
| Qwen2.5-7B-Instruct tokenizer | 151,665 | 62,266,982 | 3.3358 | 3.3236 | −22.16% |
| cl100k_base (tiktoken) | 100,256 | 75,061,973 | 2.7672 | 2.8566 | −47.2% |
| Mistral-7B-Instruct-v0.2 | 32,000 | 87,276,011 | 2.3800 | 2.4943 | −71.3% |
| StableLM-base-alpha-3b | 50,277 | 83,207,383 | 2.4963 | 2.6430 | −63.2% |
| Llama-2-7b (community mirror) | 32,000 | 91,853,794 | 2.2613 | 2.3985 | −80.2% |
| CodeLlama-7b | 32,016 | 91,853,791 | 2.2613 | 2.3985 | −80.2% |
| DeepSeek-LLM-7b-base | 100,015 | 93,334,847 | 2.2255 | 2.4470 | −83.1% |
| Falcon-7b | 65,024 | 103,640,810 | 2.0042 | 2.3126 | −103.3% |
| GPT-2 (r50k_base) | 50,256 | 105,229,579 | 1.9739 | 2.1406 | −106.4% |
| DeepSeek-Coder-6.7b-base | 32,022 | 116,855,033 | 1.7775 | 2.1074 | −129.3% |
| MosaicML MPT | n/a | NOT MEASURED (401/404 on the hub) | — | — | — |
| Gemini 3.8 Flash | n/a | NOT MEASURED (no API credential) | — | — | — |

**Competitor landscape** (properties from public model cards; only the B/T column measured here):

| Tokenizer | Algorithm | Multilingual & code | License | Vocab | Measured B/T |
|---|---|---|---|---|---:|
| o200k_base | byte-level BPE (regex) | Multilingual; code/math emphasis | MIT (tiktoken) | 200,019 | 4.0751 |
| cl100k_base | byte-level BPE (regex) | Multilingual (narrower) | MIT (tiktoken) | 100,256 | 2.7672 |
| GPT-2 | byte-level BPE | largely English | MIT (code) | 50,256 | 1.9739 |
| Llama-2 | SentencePiece BPE, byte fallback | Multilingual (Europe-strong) | Llama 2 Community License | 32,000 | 2.2613 |
| CodeLlama-7b | byte-level BPE | code (Llama 2 family) | Llama 2 Community License | 32,016 | 2.2613 |
| Qwen2.5-7B-Instruct | byte-level BPE | Multilingual (29+ languages), code | Apache-2.0 | 151,665 | 3.3358 |
| Mistral-7B-Instruct-v0.2 | byte-level BPE, byte fallback | Multilingual; code | Apache-2.0 | 32,000 | 2.3800 |
| Bloom-7b1 | byte-level BPE (BBPE) | ~46 languages | BigScience Open RAIL-M-1.0 | 250,680 | 3.7922 |
| StableLM-base-alpha-3b | byte-level BPE (GPT-NeoX family) | English + code | CC BY-SA 4.0 | 50,277 | 2.4963 |
| DeepSeek-LLM-7b-base | byte-level BPE (custom) | English/Chinese focus, code | other (see card) | 100,015 | 2.2255 |
| DeepSeek-Coder-6.7b-base | byte-level BPE (custom) | code-focused | other (see card) | 32,022 | 1.7775 |
| Falcon-7b | byte-level BPE | English + European, code | Apache-2.0 | 65,024 | 2.0042 |
| **NTokenizer v0.3-polished** | byte-level BPE + MathSymbolAware | Multilingual by design (incl. code + CJK) | source MIT/Apache-2.0; model CC BY-SA 4.0 | 262,144 | **5.5542** |

**NTokenizer v0.3-polished wins 10 of 11 categories against o200k_base and against Qwen2.5-7B-Instruct** (both: loses only the tiny 0.16 MB `other_languages` bucket). Turkish +35.0%, Arabic +33.8%, Hindi +27.1%, Spanish +12.4%, STEM +17.6%, Programming +10.2%, English +6.9% fewer tokens than o200k_base. Full per-bucket table and methodology: [COMPARISON_REPORT](https://github.com/Neuratal/neuratal-tokenizer/blob/master/reports/public-benchmark/COMPARISON_REPORT.md).

> **B/T measures compression efficiency, not language-model quality.** No downstream 50M-language-model evaluation exists yet. Do not read this as proof of downstream superiority.

Gemini 3.8 Flash is **not measured**: no Gemini API credential exists in this environment, and no estimate was substituted. See the linked report for Qwen/o200k pinning, exact file hashes, and limitations.

## Description

NTokenizer converts text into token IDs (`encode`) and reconstructs text/raw bytes from IDs (`decode`). The byte-level design guarantees that **any** byte sequence — including invalid UTF-8 — is recoverable exactly: `decode_bytes(encode_bytes(x)) == x` always holds.

Highlights of this release (v0.3-polished, 2026-11-15):

- A Unicode data-loss bug fixed: combining marks at the start of a string were previously dropped by the production pre-tokenizer family (MathSymbolAware) and by the leading-whitespace pre-tokenizer. They now attach correctly and round-trip byte-exactly. Covered by a regression suite (30 adversarial samples × 12 pre-tokenizer modes; ~40 samples × 2 production modes; 136/136 Rust tests pass).
- Encoding throughput measured ~1.6–1.7× the pre-polish implementation on the same machine (interleaved A/B; 8.9–9.0 → 14.5–15.9 MB/s single-thread; 44.6–50.6 → 92.3–96.6 MB/s at 8 threads; peak encode RSS 415 → 360 MB). These are project test-machine measurements, not universal guarantees.
- Normalization is **off by default** (`Normalization::None`). NFC/NFD/NFKC/NFKD are explicit opt-ins and may alter bytes by design.

## Training and data

- Corpus: **1.96 GB, 10,555 files**, assembled exclusively from the v0.2 leakage-audited clean tree (`data/v0.2/clean/train`). No benchmark file, byte or near-duplicate was included (leakage audit documented in the repository reports).
- Composition (bytes): Arabic 15.0%, Turkish 14.6%, Hindi 11.7%, English 11.2%, STEM 9.6%, Korean 9.5%, Spanish 7.5%, Programming 7.2%, Japanese 6.9%, Chinese 6.6%, Other 0.2%.
- Sources: Wikimedia projects (the largest share, mostly CC BY-SA/GFDL), arXiv per-article **CC BY 4.0** (187 papers only; the rest were quarantined in v0.2), Project Gutenberg (public domain), enwikinews (CC BY, not ShareAlike), Tatoeba (CC BY 2.0 FR), permissively licensed code (CPython PSF 2.0, TypeScript Apache-2.0, Go BSD-3, Rust MIT/Apache-2.0).
- Recipe: `--pretokenizer math --min-freq 2 --min-word-freq 2 --seed 42`, 256K vocabulary.
- `training_manifest.json` in this repo fingerprints every training file (path, bytes, sha256, source, license). No training corpora are distributed.

## Evaluation

Development benchmark (544 files, 207,712,944 bytes; **development/validation set — not a pristine blind test set**):

| Metric | v0.3-polished |
|---|---:|
| Overall byte-weighted B/T | **5.5542** |
| Equal-category B/T | **5.1056** |
| Byte-exact decode round-trip | 544/544 files |
| Used vocabulary (dev benchmark) | 245,334 / 262,144 (93.59%) |

Historical v0.3 record: 37,397,108 tokens / 5.5543 / 5.1057. The polished-tree figure is 37,397,319 / 5.5542; the +211-token difference (+0.00056%, 24 files) predates the polishing cycle and is disclosed in the repository.

Reference points (identical files, identical bytes): o200k_base **4.0751** B/T (50,971,306 tokens, reproduced); Qwen2.5-7B-Instruct tokenizer **3.3358** B/T (62,266,982 tokens, pinned revision `a09a35458c702b33eeacc393d103063234e8bc28`, tokenizer.json SHA-256 `c0382117…` — byte-identical to the historical Qwen2.5 measurement file); NTokenizer v0.2 4.9290 B/T. Category-level and per-bucket deltas in the comparison section above; full detail in [COMPARISON_REPORT](https://github.com/Neuratal/neuratal-tokenizer/blob/master/reports/public-benchmark/COMPARISON_REPORT.md).

**B/T measures compression efficiency, not language-model quality.** No downstream 50M-language-model evaluation exists yet; do not read this as proof of downstream superiority. Decode throughput ~160–278 MB/s; encode throughput as above (test machine; interleaved A/B).

Also disclosed:
- Chinese 3.54 B/T and Japanese 5.01 B/T remain imperfect; CJK-chunking candidates were rejected (Japanese regression).
- Programming 4.10 B/T, English 4.46 B/T.
- A 2026-11-16 research cycle (v0.4-v2, 64K pilots) found a corpus-expansion arm with +0.55% overall B/T but guardrail regressions; **nothing from that cycle was promoted** and no v0.4-v2 model exists.
- The 6.00 B/T aspiration was not reached; this release is the polished v0.3 model.

## Compatibility

- Native `.ntok` (this repository): fully validated (`ntokenizer validate`), byte-exact both directions, special tokens isolated.
- `tokenizer.json` (Hugging Face export, included here as a convenience): loads via `tokenizers` 0.23; byte-exact encode→decode in both directions; **token-ID parity is only 3/16** (HF ByteLevel pre-tokenization cannot express MathSymbolAware). Decode-exact ≠ ID-parity. The native format is authoritative. Do not claim full HF parity.

## Limitations

- The development benchmark is not a blind test; no untouched test set exists in this environment.
- Other-languages bucket (0.16 MB) lags o200k_base.
- ASAN-instrumented fuzzing is blocked by a pre-existing linker issue (not claimed). Load/round-trip fuzzing ran ~1.0M combined cases with zero crashes.
- No fine-grained safety, bias, or fairness evaluation is provided.
- Do not redistribute this model under Apache-2.0 without respecting ShareAlike.

## License and attribution

- **Source code**: MIT OR Apache-2.0.
- **This model artifact and the HF export**: **CC BY-SA 4.0**. The model is a machine-learned derivative of mostly CC BY-SA/GFDL text (Wikimedia) and CC BY material (arXiv 187 papers; enwikinews; Tatoeba). ShareAlike applies to adaptations. Attribution follows the pattern `"<project> contributors, <license>, <URL>"` (e.g. "Simple English Wikipedia contributors, CC BY-SA 4.0, https://simple.wikipedia.org"). Full attribution set: see `NOTICE` in the GitHub repository.
- Nothing here licenses the training corpora themselves; they are not distributed.

## FAQ — honest answers to reasonable skepticism

NTokenizer is new and independent. We know why you'd hesitate; here are the questions we'd ask in your place, answered with evidence you can check yourself.

### "Why should I trust an unknown tokenizer over OpenAI's, Meta's or Google's?"
You shouldn't — *trust nothing, verify.* Every NTokenizer artifact is SHA-256-pinned; every benchmark number in the docs is reproducible by running one command (`./target/release/examples/benchmark_ntok --model output/v0.3/ntokenizer-0.3-256k.ntok --manifest <manifest>`); the model round-trips arbitrary bytes byte-exactly (544/544 on the development benchmark); 136/136 tests pass; ~1.0M fuzz cases ran without a crash. Nothing about this project asks you to believe a brand. If verification takes 15 minutes, do it — if it doesn't check out, don't use it.

### "Is the benchmark legit, or was it tuned to make NTokenizer win?"
It's the same 544 files, 207,712,944 bytes, used unchanged for every candidate, external tokenizers included; files and manifest are untouched (SHA-pinned). It is a **development/validation set used throughout development** — not a blind test — and we say so everywhere. Different slices of it will weight differently: English (NTokenizer beats o200k by +6.9%), programming (+10.2%), STEM (+17.6%), Turkish (+35.0%), Arabic (+33.8%), Hindi (+27.1%), Korean (+34.7%), Chinese (+22.3%), Japanese (+31.7%) fewer tokens than o200k_base. If you need numbers for *your* domain, run the harness on your own files — it takes minutes.

### "Does higher B/T mean the language model will be better?"
**No — and we say this in every public page.** B/T is compression density. It tells you how many tokens you pay for the same bytes. It does not measure next-token loss, task accuracy, fairness or safety. No 50M language model exists yet, and nothing in this repository should be read as a claim that NTokenizer produces better LLMs. What it does demonstrate, carefully scoped, is that NTokenizer compresses multilingual and technical text more efficiently than every public competitor we could measure — e.g. **26.6% fewer tokens than o200k_base, 39.9% fewer than Qwen2.5-7B-Instruct, 50.2% fewer than cl100k_base, 57.2% fewer than Mistral-7B, 59.3% fewer than Llama-2-class, 63.9% fewer than Falcon-7b and 64.5% fewer than GPT-2** (per-file evidence in [`reports/public-benchmark/STANDARDIZED_RESULTS.csv`](https://github.com/Neuratal/neuratal-tokenizer/blob/master/reports/public-benchmark/STANDARDIZED_RESULTS.csv)).

### "What are you hiding? What are the known weaknesses?"
Nothing by design; the weaknesses are documented rather than discovered late: `other_languages` (0.16 MB) loses to o200k_base (3.32 vs 5.19); Chinese and Japanese are improved over v0.2 but Han-chunking candidates were rejected (Japanese regression) and pure Han efficiency is still imperfect; Gemini 3.8 Flash and MosaicML MPT are marked NOT MEASURED (no credential / unavailable models); the 6.00 B/T target was not achieved (the v0.4-v2 cycle is archived, not promoted); the HF export has 3/16 token-ID parity (documented); ASAN-instrumented fuzzing remains blocked (disclosed, not claimed as done).

### "Why is English's gain smaller than the other languages?"
English is where the vendor tokenizers concentrate their training; o200k is genuinely strong there, and +6.9% is what an independent multilingual corpus buys. The NTokenizer edge is elsewhere: STEM (+17.6%), code (+10.2%), and the long tail of multilingual technical text — Turkish (+35.0%), Arabic (+33.8%), Hindi (+27.1%), Korean (+34.7%). If your corpus is pure English news, you will not see a dramatic difference; if it is mixed-language or code-heavy, you will see a large one.

### "Is it safe / stable on weird inputs?"
Byte-level by construction: *any* byte sequence — invalid UTF-8 included — is recoverable exactly (`decode_bytes(encode_bytes(x)) == x`), decoding never fabricates bytes, and the training-time guarantee (136 tests, ~1M fuzz cases, zero crashes) is enforced by CI-style local verification. Combining marks, ZWJ sequences, variation selectors, Arabic harakat, Devanagari matras, planar CJK, unassigned scalars — all round-trip byte-exactly; the regression suite covers 30 adversarial samples × 12 pre-tokenizer modes.

### "Why is it not famous?"
It is brand-new, independent and distributed under a model license (CC BY-SA 4.0) that doesn't allow advertising without care. Fame requires time and backing this project doesn't have — that is an honest constraint, not a hidden disqualifier. Fame has never been evidence of quality; in this space, it is usually marketing backstopped by a large corpus. Here, the correct evaluation is two commands and 15 minutes of your time.

### "Can I use it commercially?"
Source code: MIT OR Apache-2.0 — free for any use, including commercial, with no copyleft. The trained model artifact: CC BY-SA 4.0 — you may use it commercially; adaptations (e.g., redistributed fine-tuned variants of the tokenizer) must be distributed under CC BY-SA 4.0 with attribution. Training corpora are not redistributed. This is a summary, not legal advice; consult counsel for your case.

### "Should I switch from o200k_base today?"
You should try NTokenizer on your own data first — that is the only honest answer. If your workload is multilingual, STEM, code or Turkish/Arabic/Hindi/Korean-heavy, expect meaningful token-count reduction and thus cost/throughput gains (independent of any downstream LM claim). If it is monolingual English, o200k is a solid, battle-tested choice and the improvement will be modest. Nothing about either choice is irreversible — the harness prints the numbers for your files in one command.

## Links

- Source: <https://github.com/Neuratal/neuratal-tokenizer>
- Release: <https://github.com/Neuratal/neuratal-tokenizer/releases/tag/v0.3.0>
- Citation: see `CITATION.cff` in the repository (author Rüzgar Erdem, Neuratal, 2026)