ntokenizer / README.md
Neuratal's picture
Model card: link CSV audit path explicitly
6bd2a64 verified
|
Raw History Blame Contribute Delete
17.9 kB
---
language:
- multilingual
license: cc-by-sa-4.0
tags:
- tokenizer
- bpe
- byte-level-bpe
- multilingual
- nlp
- rust
- text-tokenization
- neuratal
---
# Neuratal NTokenizer v0.3-polished
**NTokenizer** is an independently developed, multilingual **byte-level BPE tokenizer** written from scratch in Rust by **Neuratal** (founder and creator: **Rüzgar Erdem**). NTokenizer is a **tokenizer, not a language model**.
- Repository (source, Rust + Python bindings): <https://github.com/Neuratal/neuratal-tokenizer>
- Release: https://github.com/Neuratal/neuratal-tokenizer/releases/tag/v0.3.0
- Organization site: none published yet — unverified look-alike domains should not be trusted.
## Model details
- **Name:** Neuratal NTokenizer v0.3-polished
- **Organization:** Neuratal (independent AI research and software project; not claimed to be an incorporated company)
- **Creator:** Rüzgar Erdem
- **Architecture:** byte-level Byte Pair Encoding (BPE), single deterministic merge table
- **Vocabulary:** 262,144 (256 base byte tokens + 261,881 learned merges + 7 special tokens: `<|bos|>`, `<|eos|>`, `<|pad|>`, `<|unk|>`, `<|system|>`, `<|user|>`, `<|assistant|>`)
- **Native format:** `.ntok` v3 (NTOK magic + CRC32 + payload)
- **Implementation language:** Rust (reference encoder/decoder); Python bindings via ctypes
- **Intended use:** text tokenization for language-model training and inference pipelines; research tooling
- **Out of scope:** serving as a language model, or as a normalization/rewriter of Unicode text
## Tokenizer Benchmark — NTokenizer vs Leading Tokenizers
**Development/validation benchmark — not a pristine blind evaluation set.** 544 files, 207,712,944 bytes, identical raw bytes, no normalization, no chat templates for any candidate. `**Bold**` = higher B/T (fewer tokens per input byte) than the comparator. Equal-category B/T = arithmetic mean of the 11 bucket-level B/Ts.
| Tokenizer (reference) | Vocabulary | Tokens | Overall B/T | Equal-category B/T | Token reduction vs o200k |
|---|---:|---:|---:|---:|---:|
| NTokenizer v0.1 | 262,144 | 65,573,261 | 3.1676 | 3.1203 | −28.65% |
| NTokenizer v0.2 | 262,144 | 42,140,674 | 4.9290 | 4.6855 | +17.32% |
| NTokenizer v0.3-original (historical) | 262,144 | 37,397,108 | 5.5543 | 5.1057 | +26.63% |
| **NTokenizer v0.3-polished (official)** | 262,144 | **37,397,319** | **5.5542** | **5.1056** | **+26.63%** |
| NTokenizer v0.4-fail (CJK8, archived) | 262,144 | 37,418,652 | 5.5511 | 5.1031 | +26.59% |
| NTokenizer v0.4-fail (Han4, archived) | 262,144 | 37,406,269 | 5.5529 | 5.1081 | +26.61% |
| o200k_base (tiktoken) | 200,019 | 50,971,306 | 4.0751 | 4.0899 | 0.00% |
| Bloom-7b1 | 250,680 | 54,773,073 | 3.7922 | 4.1584 | +7.4% |
| Qwen2.5-7B-Instruct tokenizer | 151,665 | 62,266,982 | 3.3358 | 3.3236 | −22.16% |
| cl100k_base (tiktoken) | 100,256 | 75,061,973 | 2.7672 | 2.8566 | −47.2% |
| Mistral-7B-Instruct-v0.2 | 32,000 | 87,276,011 | 2.3800 | 2.4943 | −71.3% |
| StableLM-base-alpha-3b | 50,277 | 83,207,383 | 2.4963 | 2.6430 | −63.2% |
| Llama-2-7b (community mirror) | 32,000 | 91,853,794 | 2.2613 | 2.3985 | −80.2% |
| CodeLlama-7b | 32,016 | 91,853,791 | 2.2613 | 2.3985 | −80.2% |
| DeepSeek-LLM-7b-base | 100,015 | 93,334,847 | 2.2255 | 2.4470 | −83.1% |
| Falcon-7b | 65,024 | 103,640,810 | 2.0042 | 2.3126 | −103.3% |
| GPT-2 (r50k_base) | 50,256 | 105,229,579 | 1.9739 | 2.1406 | −106.4% |
| DeepSeek-Coder-6.7b-base | 32,022 | 116,855,033 | 1.7775 | 2.1074 | −129.3% |
| MosaicML MPT | n/a | NOT MEASURED (401/404 on the hub) | — | — | — |
| Gemini 3.8 Flash | n/a | NOT MEASURED (no API credential) | — | — | — |
**Competitor landscape** (properties from public model cards; only the B/T column measured here):
| Tokenizer | Algorithm | Multilingual & code | License | Vocab | Measured B/T |
|---|---|---|---|---|---:|
| o200k_base | byte-level BPE (regex) | Multilingual; code/math emphasis | MIT (tiktoken) | 200,019 | 4.0751 |
| cl100k_base | byte-level BPE (regex) | Multilingual (narrower) | MIT (tiktoken) | 100,256 | 2.7672 |
| GPT-2 | byte-level BPE | largely English | MIT (code) | 50,256 | 1.9739 |
| Llama-2 | SentencePiece BPE, byte fallback | Multilingual (Europe-strong) | Llama 2 Community License | 32,000 | 2.2613 |
| CodeLlama-7b | byte-level BPE | code (Llama 2 family) | Llama 2 Community License | 32,016 | 2.2613 |
| Qwen2.5-7B-Instruct | byte-level BPE | Multilingual (29+ languages), code | Apache-2.0 | 151,665 | 3.3358 |
| Mistral-7B-Instruct-v0.2 | byte-level BPE, byte fallback | Multilingual; code | Apache-2.0 | 32,000 | 2.3800 |
| Bloom-7b1 | byte-level BPE (BBPE) | ~46 languages | BigScience Open RAIL-M-1.0 | 250,680 | 3.7922 |
| StableLM-base-alpha-3b | byte-level BPE (GPT-NeoX family) | English + code | CC BY-SA 4.0 | 50,277 | 2.4963 |
| DeepSeek-LLM-7b-base | byte-level BPE (custom) | English/Chinese focus, code | other (see card) | 100,015 | 2.2255 |
| DeepSeek-Coder-6.7b-base | byte-level BPE (custom) | code-focused | other (see card) | 32,022 | 1.7775 |
| Falcon-7b | byte-level BPE | English + European, code | Apache-2.0 | 65,024 | 2.0042 |
| **NTokenizer v0.3-polished** | byte-level BPE + MathSymbolAware | Multilingual by design (incl. code + CJK) | source MIT/Apache-2.0; model CC BY-SA 4.0 | 262,144 | **5.5542** |
**NTokenizer v0.3-polished wins 10 of 11 categories against o200k_base and against Qwen2.5-7B-Instruct** (both: loses only the tiny 0.16 MB `other_languages` bucket). Turkish +35.0%, Arabic +33.8%, Hindi +27.1%, Spanish +12.4%, STEM +17.6%, Programming +10.2%, English +6.9% fewer tokens than o200k_base. Full per-bucket table and methodology: [COMPARISON_REPORT](https://github.com/Neuratal/neuratal-tokenizer/blob/master/reports/public-benchmark/COMPARISON_REPORT.md).
> **B/T measures compression efficiency, not language-model quality.** No downstream 50M-language-model evaluation exists yet. Do not read this as proof of downstream superiority.
Gemini 3.8 Flash is **not measured**: no Gemini API credential exists in this environment, and no estimate was substituted. See the linked report for Qwen/o200k pinning, exact file hashes, and limitations.
## Description
NTokenizer converts text into token IDs (`encode`) and reconstructs text/raw bytes from IDs (`decode`). The byte-level design guarantees that **any** byte sequence — including invalid UTF-8 — is recoverable exactly: `decode_bytes(encode_bytes(x)) == x` always holds.
Highlights of this release (v0.3-polished, 2026-11-15):
- A Unicode data-loss bug fixed: combining marks at the start of a string were previously dropped by the production pre-tokenizer family (MathSymbolAware) and by the leading-whitespace pre-tokenizer. They now attach correctly and round-trip byte-exactly. Covered by a regression suite (30 adversarial samples × 12 pre-tokenizer modes; ~40 samples × 2 production modes; 136/136 Rust tests pass).
- Encoding throughput measured ~1.6–1.7× the pre-polish implementation on the same machine (interleaved A/B; 8.9–9.0 → 14.5–15.9 MB/s single-thread; 44.6–50.6 → 92.3–96.6 MB/s at 8 threads; peak encode RSS 415 → 360 MB). These are project test-machine measurements, not universal guarantees.
- Normalization is **off by default** (`Normalization::None`). NFC/NFD/NFKC/NFKD are explicit opt-ins and may alter bytes by design.
## Training and data
- Corpus: **1.96 GB, 10,555 files**, assembled exclusively from the v0.2 leakage-audited clean tree (`data/v0.2/clean/train`). No benchmark file, byte or near-duplicate was included (leakage audit documented in the repository reports).
- Composition (bytes): Arabic 15.0%, Turkish 14.6%, Hindi 11.7%, English 11.2%, STEM 9.6%, Korean 9.5%, Spanish 7.5%, Programming 7.2%, Japanese 6.9%, Chinese 6.6%, Other 0.2%.
- Sources: Wikimedia projects (the largest share, mostly CC BY-SA/GFDL), arXiv per-article **CC BY 4.0** (187 papers only; the rest were quarantined in v0.2), Project Gutenberg (public domain), enwikinews (CC BY, not ShareAlike), Tatoeba (CC BY 2.0 FR), permissively licensed code (CPython PSF 2.0, TypeScript Apache-2.0, Go BSD-3, Rust MIT/Apache-2.0).
- Recipe: `--pretokenizer math --min-freq 2 --min-word-freq 2 --seed 42`, 256K vocabulary.
- `training_manifest.json` in this repo fingerprints every training file (path, bytes, sha256, source, license). No training corpora are distributed.
## Evaluation
Development benchmark (544 files, 207,712,944 bytes; **development/validation set — not a pristine blind test set**):
| Metric | v0.3-polished |
|---|---:|
| Overall byte-weighted B/T | **5.5542** |
| Equal-category B/T | **5.1056** |
| Byte-exact decode round-trip | 544/544 files |
| Used vocabulary (dev benchmark) | 245,334 / 262,144 (93.59%) |
Historical v0.3 record: 37,397,108 tokens / 5.5543 / 5.1057. The polished-tree figure is 37,397,319 / 5.5542; the +211-token difference (+0.00056%, 24 files) predates the polishing cycle and is disclosed in the repository.
Reference points (identical files, identical bytes): o200k_base **4.0751** B/T (50,971,306 tokens, reproduced); Qwen2.5-7B-Instruct tokenizer **3.3358** B/T (62,266,982 tokens, pinned revision `a09a35458c702b33eeacc393d103063234e8bc28`, tokenizer.json SHA-256 `c0382117…` — byte-identical to the historical Qwen2.5 measurement file); NTokenizer v0.2 4.9290 B/T. Category-level and per-bucket deltas in the comparison section above; full detail in [COMPARISON_REPORT](https://github.com/Neuratal/neuratal-tokenizer/blob/master/reports/public-benchmark/COMPARISON_REPORT.md).
**B/T measures compression efficiency, not language-model quality.** No downstream 50M-language-model evaluation exists yet; do not read this as proof of downstream superiority. Decode throughput ~160–278 MB/s; encode throughput as above (test machine; interleaved A/B).
Also disclosed:
- Chinese 3.54 B/T and Japanese 5.01 B/T remain imperfect; CJK-chunking candidates were rejected (Japanese regression).
- Programming 4.10 B/T, English 4.46 B/T.
- A 2026-11-16 research cycle (v0.4-v2, 64K pilots) found a corpus-expansion arm with +0.55% overall B/T but guardrail regressions; **nothing from that cycle was promoted** and no v0.4-v2 model exists.
- The 6.00 B/T aspiration was not reached; this release is the polished v0.3 model.
## Compatibility
- Native `.ntok` (this repository): fully validated (`ntokenizer validate`), byte-exact both directions, special tokens isolated.
- `tokenizer.json` (Hugging Face export, included here as a convenience): loads via `tokenizers` 0.23; byte-exact encode→decode in both directions; **token-ID parity is only 3/16** (HF ByteLevel pre-tokenization cannot express MathSymbolAware). Decode-exact ≠ ID-parity. The native format is authoritative. Do not claim full HF parity.
## Limitations
- The development benchmark is not a blind test; no untouched test set exists in this environment.
- Other-languages bucket (0.16 MB) lags o200k_base.
- ASAN-instrumented fuzzing is blocked by a pre-existing linker issue (not claimed). Load/round-trip fuzzing ran ~1.0M combined cases with zero crashes.
- No fine-grained safety, bias, or fairness evaluation is provided.
- Do not redistribute this model under Apache-2.0 without respecting ShareAlike.
## License and attribution
- **Source code**: MIT OR Apache-2.0.
- **This model artifact and the HF export**: **CC BY-SA 4.0**. The model is a machine-learned derivative of mostly CC BY-SA/GFDL text (Wikimedia) and CC BY material (arXiv 187 papers; enwikinews; Tatoeba). ShareAlike applies to adaptations. Attribution follows the pattern `"<project> contributors, <license>, <URL>"` (e.g. "Simple English Wikipedia contributors, CC BY-SA 4.0, https://simple.wikipedia.org"). Full attribution set: see `NOTICE` in the GitHub repository.
- Nothing here licenses the training corpora themselves; they are not distributed.
## FAQ — honest answers to reasonable skepticism
NTokenizer is new and independent. We know why you'd hesitate; here are the questions we'd ask in your place, answered with evidence you can check yourself.
### "Why should I trust an unknown tokenizer over OpenAI's, Meta's or Google's?"
You shouldn't — *trust nothing, verify.* Every NTokenizer artifact is SHA-256-pinned; every benchmark number in the docs is reproducible by running one command (`./target/release/examples/benchmark_ntok --model output/v0.3/ntokenizer-0.3-256k.ntok --manifest <manifest>`); the model round-trips arbitrary bytes byte-exactly (544/544 on the development benchmark); 136/136 tests pass; ~1.0M fuzz cases ran without a crash. Nothing about this project asks you to believe a brand. If verification takes 15 minutes, do it — if it doesn't check out, don't use it.
### "Is the benchmark legit, or was it tuned to make NTokenizer win?"
It's the same 544 files, 207,712,944 bytes, used unchanged for every candidate, external tokenizers included; files and manifest are untouched (SHA-pinned). It is a **development/validation set used throughout development** — not a blind test — and we say so everywhere. Different slices of it will weight differently: English (NTokenizer beats o200k by +6.9%), programming (+10.2%), STEM (+17.6%), Turkish (+35.0%), Arabic (+33.8%), Hindi (+27.1%), Korean (+34.7%), Chinese (+22.3%), Japanese (+31.7%) fewer tokens than o200k_base. If you need numbers for *your* domain, run the harness on your own files — it takes minutes.
### "Does higher B/T mean the language model will be better?"
**No — and we say this in every public page.** B/T is compression density. It tells you how many tokens you pay for the same bytes. It does not measure next-token loss, task accuracy, fairness or safety. No 50M language model exists yet, and nothing in this repository should be read as a claim that NTokenizer produces better LLMs. What it does demonstrate, carefully scoped, is that NTokenizer compresses multilingual and technical text more efficiently than every public competitor we could measure — e.g. **26.6% fewer tokens than o200k_base, 39.9% fewer than Qwen2.5-7B-Instruct, 50.2% fewer than cl100k_base, 57.2% fewer than Mistral-7B, 59.3% fewer than Llama-2-class, 63.9% fewer than Falcon-7b and 64.5% fewer than GPT-2** (per-file evidence in [`reports/public-benchmark/STANDARDIZED_RESULTS.csv`](https://github.com/Neuratal/neuratal-tokenizer/blob/master/reports/public-benchmark/STANDARDIZED_RESULTS.csv)).
### "What are you hiding? What are the known weaknesses?"
Nothing by design; the weaknesses are documented rather than discovered late: `other_languages` (0.16 MB) loses to o200k_base (3.32 vs 5.19); Chinese and Japanese are improved over v0.2 but Han-chunking candidates were rejected (Japanese regression) and pure Han efficiency is still imperfect; Gemini 3.8 Flash and MosaicML MPT are marked NOT MEASURED (no credential / unavailable models); the 6.00 B/T target was not achieved (the v0.4-v2 cycle is archived, not promoted); the HF export has 3/16 token-ID parity (documented); ASAN-instrumented fuzzing remains blocked (disclosed, not claimed as done).
### "Why is English's gain smaller than the other languages?"
English is where the vendor tokenizers concentrate their training; o200k is genuinely strong there, and +6.9% is what an independent multilingual corpus buys. The NTokenizer edge is elsewhere: STEM (+17.6%), code (+10.2%), and the long tail of multilingual technical text — Turkish (+35.0%), Arabic (+33.8%), Hindi (+27.1%), Korean (+34.7%). If your corpus is pure English news, you will not see a dramatic difference; if it is mixed-language or code-heavy, you will see a large one.
### "Is it safe / stable on weird inputs?"
Byte-level by construction: *any* byte sequence — invalid UTF-8 included — is recoverable exactly (`decode_bytes(encode_bytes(x)) == x`), decoding never fabricates bytes, and the training-time guarantee (136 tests, ~1M fuzz cases, zero crashes) is enforced by CI-style local verification. Combining marks, ZWJ sequences, variation selectors, Arabic harakat, Devanagari matras, planar CJK, unassigned scalars — all round-trip byte-exactly; the regression suite covers 30 adversarial samples × 12 pre-tokenizer modes.
### "Why is it not famous?"
It is brand-new, independent and distributed under a model license (CC BY-SA 4.0) that doesn't allow advertising without care. Fame requires time and backing this project doesn't have — that is an honest constraint, not a hidden disqualifier. Fame has never been evidence of quality; in this space, it is usually marketing backstopped by a large corpus. Here, the correct evaluation is two commands and 15 minutes of your time.
### "Can I use it commercially?"
Source code: MIT OR Apache-2.0 — free for any use, including commercial, with no copyleft. The trained model artifact: CC BY-SA 4.0 — you may use it commercially; adaptations (e.g., redistributed fine-tuned variants of the tokenizer) must be distributed under CC BY-SA 4.0 with attribution. Training corpora are not redistributed. This is a summary, not legal advice; consult counsel for your case.
### "Should I switch from o200k_base today?"
You should try NTokenizer on your own data first — that is the only honest answer. If your workload is multilingual, STEM, code or Turkish/Arabic/Hindi/Korean-heavy, expect meaningful token-count reduction and thus cost/throughput gains (independent of any downstream LM claim). If it is monolingual English, o200k is a solid, battle-tested choice and the improvement will be modest. Nothing about either choice is irreversible — the harness prints the numbers for your files in one command.
## Links
- Source: <https://github.com/Neuratal/neuratal-tokenizer>
- Release: <https://github.com/Neuratal/neuratal-tokenizer/releases/tag/v0.3.0>
- Citation: see `CITATION.cff` in the repository (author Rüzgar Erdem, Neuratal, 2026)