--- language: - multilingual license: cc-by-sa-4.0 tags: - tokenizer - bpe - byte-level-bpe - multilingual - nlp - rust - text-tokenization - neuratal --- # Neuratal NTokenizer v0.3-polished **NTokenizer** is an independently developed, multilingual **byte-level BPE tokenizer** written from scratch in Rust by **Neuratal** (founder and creator: **Rüzgar Erdem**). NTokenizer is a **tokenizer, not a language model**. - Repository (source, Rust + Python bindings): - Release: https://github.com/Neuratal/neuratal-tokenizer/releases/tag/v0.3.0 - Organization site: none published yet — unverified look-alike domains should not be trusted. ## Model details - **Name:** Neuratal NTokenizer v0.3-polished - **Organization:** Neuratal (independent AI research and software project; not claimed to be an incorporated company) - **Creator:** Rüzgar Erdem - **Architecture:** byte-level Byte Pair Encoding (BPE), single deterministic merge table - **Vocabulary:** 262,144 (256 base byte tokens + 261,881 learned merges + 7 special tokens: `<|bos|>`, `<|eos|>`, `<|pad|>`, `<|unk|>`, `<|system|>`, `<|user|>`, `<|assistant|>`) - **Native format:** `.ntok` v3 (NTOK magic + CRC32 + payload) - **Implementation language:** Rust (reference encoder/decoder); Python bindings via ctypes - **Intended use:** text tokenization for language-model training and inference pipelines; research tooling - **Out of scope:** serving as a language model, or as a normalization/rewriter of Unicode text ## Tokenizer Benchmark — NTokenizer vs Leading Tokenizers **Development/validation benchmark — not a pristine blind evaluation set.** 544 files, 207,712,944 bytes, identical raw bytes, no normalization, no chat templates for any candidate. `**Bold**` = higher B/T (fewer tokens per input byte) than the comparator. Equal-category B/T = arithmetic mean of the 11 bucket-level B/Ts. | Tokenizer (reference) | Vocabulary | Tokens | Overall B/T | Equal-category B/T | Token reduction vs o200k | |---|---:|---:|---:|---:|---:| | NTokenizer v0.1 | 262,144 | 65,573,261 | 3.1676 | 3.1203 | −28.65% | | NTokenizer v0.2 | 262,144 | 42,140,674 | 4.9290 | 4.6855 | +17.32% | | NTokenizer v0.3-original (historical) | 262,144 | 37,397,108 | 5.5543 | 5.1057 | +26.63% | | **NTokenizer v0.3-polished (official)** | 262,144 | **37,397,319** | **5.5542** | **5.1056** | **+26.63%** | | NTokenizer v0.4-fail (CJK8, archived) | 262,144 | 37,418,652 | 5.5511 | 5.1031 | +26.59% | | NTokenizer v0.4-fail (Han4, archived) | 262,144 | 37,406,269 | 5.5529 | 5.1081 | +26.61% | | o200k_base (tiktoken) | 200,019 | 50,971,306 | 4.0751 | 4.0899 | 0.00% | | Bloom-7b1 | 250,680 | 54,773,073 | 3.7922 | 4.1584 | +7.4% | | Qwen2.5-7B-Instruct tokenizer | 151,665 | 62,266,982 | 3.3358 | 3.3236 | −22.16% | | cl100k_base (tiktoken) | 100,256 | 75,061,973 | 2.7672 | 2.8566 | −47.2% | | Mistral-7B-Instruct-v0.2 | 32,000 | 87,276,011 | 2.3800 | 2.4943 | −71.3% | | StableLM-base-alpha-3b | 50,277 | 83,207,383 | 2.4963 | 2.6430 | −63.2% | | Llama-2-7b (community mirror) | 32,000 | 91,853,794 | 2.2613 | 2.3985 | −80.2% | | CodeLlama-7b | 32,016 | 91,853,791 | 2.2613 | 2.3985 | −80.2% | | DeepSeek-LLM-7b-base | 100,015 | 93,334,847 | 2.2255 | 2.4470 | −83.1% | | Falcon-7b | 65,024 | 103,640,810 | 2.0042 | 2.3126 | −103.3% | | GPT-2 (r50k_base) | 50,256 | 105,229,579 | 1.9739 | 2.1406 | −106.4% | | DeepSeek-Coder-6.7b-base | 32,022 | 116,855,033 | 1.7775 | 2.1074 | −129.3% | | MosaicML MPT | n/a | NOT MEASURED (401/404 on the hub) | — | — | — | | Gemini 3.8 Flash | n/a | NOT MEASURED (no API credential) | — | — | — | **Competitor landscape** (properties from public model cards; only the B/T column measured here): | Tokenizer | Algorithm | Multilingual & code | License | Vocab | Measured B/T | |---|---|---|---|---|---:| | o200k_base | byte-level BPE (regex) | Multilingual; code/math emphasis | MIT (tiktoken) | 200,019 | 4.0751 | | cl100k_base | byte-level BPE (regex) | Multilingual (narrower) | MIT (tiktoken) | 100,256 | 2.7672 | | GPT-2 | byte-level BPE | largely English | MIT (code) | 50,256 | 1.9739 | | Llama-2 | SentencePiece BPE, byte fallback | Multilingual (Europe-strong) | Llama 2 Community License | 32,000 | 2.2613 | | CodeLlama-7b | byte-level BPE | code (Llama 2 family) | Llama 2 Community License | 32,016 | 2.2613 | | Qwen2.5-7B-Instruct | byte-level BPE | Multilingual (29+ languages), code | Apache-2.0 | 151,665 | 3.3358 | | Mistral-7B-Instruct-v0.2 | byte-level BPE, byte fallback | Multilingual; code | Apache-2.0 | 32,000 | 2.3800 | | Bloom-7b1 | byte-level BPE (BBPE) | ~46 languages | BigScience Open RAIL-M-1.0 | 250,680 | 3.7922 | | StableLM-base-alpha-3b | byte-level BPE (GPT-NeoX family) | English + code | CC BY-SA 4.0 | 50,277 | 2.4963 | | DeepSeek-LLM-7b-base | byte-level BPE (custom) | English/Chinese focus, code | other (see card) | 100,015 | 2.2255 | | DeepSeek-Coder-6.7b-base | byte-level BPE (custom) | code-focused | other (see card) | 32,022 | 1.7775 | | Falcon-7b | byte-level BPE | English + European, code | Apache-2.0 | 65,024 | 2.0042 | | **NTokenizer v0.3-polished** | byte-level BPE + MathSymbolAware | Multilingual by design (incl. code + CJK) | source MIT/Apache-2.0; model CC BY-SA 4.0 | 262,144 | **5.5542** | **NTokenizer v0.3-polished wins 10 of 11 categories against o200k_base and against Qwen2.5-7B-Instruct** (both: loses only the tiny 0.16 MB `other_languages` bucket). Turkish +35.0%, Arabic +33.8%, Hindi +27.1%, Spanish +12.4%, STEM +17.6%, Programming +10.2%, English +6.9% fewer tokens than o200k_base. Full per-bucket table and methodology: [COMPARISON_REPORT](https://github.com/Neuratal/neuratal-tokenizer/blob/master/reports/public-benchmark/COMPARISON_REPORT.md). > **B/T measures compression efficiency, not language-model quality.** No downstream 50M-language-model evaluation exists yet. Do not read this as proof of downstream superiority. Gemini 3.8 Flash is **not measured**: no Gemini API credential exists in this environment, and no estimate was substituted. See the linked report for Qwen/o200k pinning, exact file hashes, and limitations. ## Description NTokenizer converts text into token IDs (`encode`) and reconstructs text/raw bytes from IDs (`decode`). The byte-level design guarantees that **any** byte sequence — including invalid UTF-8 — is recoverable exactly: `decode_bytes(encode_bytes(x)) == x` always holds. Highlights of this release (v0.3-polished, 2026-11-15): - A Unicode data-loss bug fixed: combining marks at the start of a string were previously dropped by the production pre-tokenizer family (MathSymbolAware) and by the leading-whitespace pre-tokenizer. They now attach correctly and round-trip byte-exactly. Covered by a regression suite (30 adversarial samples × 12 pre-tokenizer modes; ~40 samples × 2 production modes; 136/136 Rust tests pass). - Encoding throughput measured ~1.6–1.7× the pre-polish implementation on the same machine (interleaved A/B; 8.9–9.0 → 14.5–15.9 MB/s single-thread; 44.6–50.6 → 92.3–96.6 MB/s at 8 threads; peak encode RSS 415 → 360 MB). These are project test-machine measurements, not universal guarantees. - Normalization is **off by default** (`Normalization::None`). NFC/NFD/NFKC/NFKD are explicit opt-ins and may alter bytes by design. ## Training and data - Corpus: **1.96 GB, 10,555 files**, assembled exclusively from the v0.2 leakage-audited clean tree (`data/v0.2/clean/train`). No benchmark file, byte or near-duplicate was included (leakage audit documented in the repository reports). - Composition (bytes): Arabic 15.0%, Turkish 14.6%, Hindi 11.7%, English 11.2%, STEM 9.6%, Korean 9.5%, Spanish 7.5%, Programming 7.2%, Japanese 6.9%, Chinese 6.6%, Other 0.2%. - Sources: Wikimedia projects (the largest share, mostly CC BY-SA/GFDL), arXiv per-article **CC BY 4.0** (187 papers only; the rest were quarantined in v0.2), Project Gutenberg (public domain), enwikinews (CC BY, not ShareAlike), Tatoeba (CC BY 2.0 FR), permissively licensed code (CPython PSF 2.0, TypeScript Apache-2.0, Go BSD-3, Rust MIT/Apache-2.0). - Recipe: `--pretokenizer math --min-freq 2 --min-word-freq 2 --seed 42`, 256K vocabulary. - `training_manifest.json` in this repo fingerprints every training file (path, bytes, sha256, source, license). No training corpora are distributed. ## Evaluation Development benchmark (544 files, 207,712,944 bytes; **development/validation set — not a pristine blind test set**): | Metric | v0.3-polished | |---|---:| | Overall byte-weighted B/T | **5.5542** | | Equal-category B/T | **5.1056** | | Byte-exact decode round-trip | 544/544 files | | Used vocabulary (dev benchmark) | 245,334 / 262,144 (93.59%) | Historical v0.3 record: 37,397,108 tokens / 5.5543 / 5.1057. The polished-tree figure is 37,397,319 / 5.5542; the +211-token difference (+0.00056%, 24 files) predates the polishing cycle and is disclosed in the repository. Reference points (identical files, identical bytes): o200k_base **4.0751** B/T (50,971,306 tokens, reproduced); Qwen2.5-7B-Instruct tokenizer **3.3358** B/T (62,266,982 tokens, pinned revision `a09a35458c702b33eeacc393d103063234e8bc28`, tokenizer.json SHA-256 `c0382117…` — byte-identical to the historical Qwen2.5 measurement file); NTokenizer v0.2 4.9290 B/T. Category-level and per-bucket deltas in the comparison section above; full detail in [COMPARISON_REPORT](https://github.com/Neuratal/neuratal-tokenizer/blob/master/reports/public-benchmark/COMPARISON_REPORT.md). **B/T measures compression efficiency, not language-model quality.** No downstream 50M-language-model evaluation exists yet; do not read this as proof of downstream superiority. Decode throughput ~160–278 MB/s; encode throughput as above (test machine; interleaved A/B). Also disclosed: - Chinese 3.54 B/T and Japanese 5.01 B/T remain imperfect; CJK-chunking candidates were rejected (Japanese regression). - Programming 4.10 B/T, English 4.46 B/T. - A 2026-11-16 research cycle (v0.4-v2, 64K pilots) found a corpus-expansion arm with +0.55% overall B/T but guardrail regressions; **nothing from that cycle was promoted** and no v0.4-v2 model exists. - The 6.00 B/T aspiration was not reached; this release is the polished v0.3 model. ## Compatibility - Native `.ntok` (this repository): fully validated (`ntokenizer validate`), byte-exact both directions, special tokens isolated. - `tokenizer.json` (Hugging Face export, included here as a convenience): loads via `tokenizers` 0.23; byte-exact encode→decode in both directions; **token-ID parity is only 3/16** (HF ByteLevel pre-tokenization cannot express MathSymbolAware). Decode-exact ≠ ID-parity. The native format is authoritative. Do not claim full HF parity. ## Limitations - The development benchmark is not a blind test; no untouched test set exists in this environment. - Other-languages bucket (0.16 MB) lags o200k_base. - ASAN-instrumented fuzzing is blocked by a pre-existing linker issue (not claimed). Load/round-trip fuzzing ran ~1.0M combined cases with zero crashes. - No fine-grained safety, bias, or fairness evaluation is provided. - Do not redistribute this model under Apache-2.0 without respecting ShareAlike. ## License and attribution - **Source code**: MIT OR Apache-2.0. - **This model artifact and the HF export**: **CC BY-SA 4.0**. The model is a machine-learned derivative of mostly CC BY-SA/GFDL text (Wikimedia) and CC BY material (arXiv 187 papers; enwikinews; Tatoeba). ShareAlike applies to adaptations. Attribution follows the pattern `" contributors, , "` (e.g. "Simple English Wikipedia contributors, CC BY-SA 4.0, https://simple.wikipedia.org"). Full attribution set: see `NOTICE` in the GitHub repository. - Nothing here licenses the training corpora themselves; they are not distributed. ## FAQ — honest answers to reasonable skepticism NTokenizer is new and independent. We know why you'd hesitate; here are the questions we'd ask in your place, answered with evidence you can check yourself. ### "Why should I trust an unknown tokenizer over OpenAI's, Meta's or Google's?" You shouldn't — *trust nothing, verify.* Every NTokenizer artifact is SHA-256-pinned; every benchmark number in the docs is reproducible by running one command (`./target/release/examples/benchmark_ntok --model output/v0.3/ntokenizer-0.3-256k.ntok --manifest `); the model round-trips arbitrary bytes byte-exactly (544/544 on the development benchmark); 136/136 tests pass; ~1.0M fuzz cases ran without a crash. Nothing about this project asks you to believe a brand. If verification takes 15 minutes, do it — if it doesn't check out, don't use it. ### "Is the benchmark legit, or was it tuned to make NTokenizer win?" It's the same 544 files, 207,712,944 bytes, used unchanged for every candidate, external tokenizers included; files and manifest are untouched (SHA-pinned). It is a **development/validation set used throughout development** — not a blind test — and we say so everywhere. Different slices of it will weight differently: English (NTokenizer beats o200k by +6.9%), programming (+10.2%), STEM (+17.6%), Turkish (+35.0%), Arabic (+33.8%), Hindi (+27.1%), Korean (+34.7%), Chinese (+22.3%), Japanese (+31.7%) fewer tokens than o200k_base. If you need numbers for *your* domain, run the harness on your own files — it takes minutes. ### "Does higher B/T mean the language model will be better?" **No — and we say this in every public page.** B/T is compression density. It tells you how many tokens you pay for the same bytes. It does not measure next-token loss, task accuracy, fairness or safety. No 50M language model exists yet, and nothing in this repository should be read as a claim that NTokenizer produces better LLMs. What it does demonstrate, carefully scoped, is that NTokenizer compresses multilingual and technical text more efficiently than every public competitor we could measure — e.g. **26.6% fewer tokens than o200k_base, 39.9% fewer than Qwen2.5-7B-Instruct, 50.2% fewer than cl100k_base, 57.2% fewer than Mistral-7B, 59.3% fewer than Llama-2-class, 63.9% fewer than Falcon-7b and 64.5% fewer than GPT-2** (per-file evidence in [`reports/public-benchmark/STANDARDIZED_RESULTS.csv`](https://github.com/Neuratal/neuratal-tokenizer/blob/master/reports/public-benchmark/STANDARDIZED_RESULTS.csv)). ### "What are you hiding? What are the known weaknesses?" Nothing by design; the weaknesses are documented rather than discovered late: `other_languages` (0.16 MB) loses to o200k_base (3.32 vs 5.19); Chinese and Japanese are improved over v0.2 but Han-chunking candidates were rejected (Japanese regression) and pure Han efficiency is still imperfect; Gemini 3.8 Flash and MosaicML MPT are marked NOT MEASURED (no credential / unavailable models); the 6.00 B/T target was not achieved (the v0.4-v2 cycle is archived, not promoted); the HF export has 3/16 token-ID parity (documented); ASAN-instrumented fuzzing remains blocked (disclosed, not claimed as done). ### "Why is English's gain smaller than the other languages?" English is where the vendor tokenizers concentrate their training; o200k is genuinely strong there, and +6.9% is what an independent multilingual corpus buys. The NTokenizer edge is elsewhere: STEM (+17.6%), code (+10.2%), and the long tail of multilingual technical text — Turkish (+35.0%), Arabic (+33.8%), Hindi (+27.1%), Korean (+34.7%). If your corpus is pure English news, you will not see a dramatic difference; if it is mixed-language or code-heavy, you will see a large one. ### "Is it safe / stable on weird inputs?" Byte-level by construction: *any* byte sequence — invalid UTF-8 included — is recoverable exactly (`decode_bytes(encode_bytes(x)) == x`), decoding never fabricates bytes, and the training-time guarantee (136 tests, ~1M fuzz cases, zero crashes) is enforced by CI-style local verification. Combining marks, ZWJ sequences, variation selectors, Arabic harakat, Devanagari matras, planar CJK, unassigned scalars — all round-trip byte-exactly; the regression suite covers 30 adversarial samples × 12 pre-tokenizer modes. ### "Why is it not famous?" It is brand-new, independent and distributed under a model license (CC BY-SA 4.0) that doesn't allow advertising without care. Fame requires time and backing this project doesn't have — that is an honest constraint, not a hidden disqualifier. Fame has never been evidence of quality; in this space, it is usually marketing backstopped by a large corpus. Here, the correct evaluation is two commands and 15 minutes of your time. ### "Can I use it commercially?" Source code: MIT OR Apache-2.0 — free for any use, including commercial, with no copyleft. The trained model artifact: CC BY-SA 4.0 — you may use it commercially; adaptations (e.g., redistributed fine-tuned variants of the tokenizer) must be distributed under CC BY-SA 4.0 with attribution. Training corpora are not redistributed. This is a summary, not legal advice; consult counsel for your case. ### "Should I switch from o200k_base today?" You should try NTokenizer on your own data first — that is the only honest answer. If your workload is multilingual, STEM, code or Turkish/Arabic/Hindi/Korean-heavy, expect meaningful token-count reduction and thus cost/throughput gains (independent of any downstream LM claim). If it is monolingual English, o200k is a solid, battle-tested choice and the improvement will be modest. Nothing about either choice is irreversible — the harness prints the numbers for your files in one command. ## Links - Source: - Release: - Citation: see `CITATION.cff` in the repository (author Rüzgar Erdem, Neuratal, 2026)