Update README.md

README.md

@@ -1,9 +1,253 @@
 ---
 license: apache-2.0
-language:
-- en
 tags:
-- BloatTokenizer
-- Tokenizer
-- 32K_Vocabulary
----
+  - tokenizer
+  - tokenization
+  - code
+  - mathematics
+  - nlp
+  - llm
+  - cpp
+  - language-modeling
+language:
+  - en
+---
+
+# BloatTokenizer32K
+
+**Compact vocabulary. Efficient tokenization. Lossless decoding.**
+
+BloatTokenizer32K is an open-source text tokenizer featuring a **32,000-entry vocabulary**, a native C++ encoding engine, and Python bindings.
+
+Designed for language-model training and inference, it supports English, programming languages, mathematics, and conversational text.
+
+On our mixed-text development benchmark, BloatTokenizer32K achieves compression comparable to GPT-4's `cl100k_base` and GPT-4o's `o200k_base`, despite using substantially smaller vocabulary capacity.
+
+**[GitHub Repository](https://github.com/Dev-qvinfor/BloatTokenizer32K)**
+
+## Key features
+
+- **32,000 vocabulary entries**, including 16 reserved control tokens.
+- **21.96% fewer tokens than GPT-2** on the evaluated mixed corpus.
+- **6.48% fewer code tokens than `cl100k_base`** on the code subset.
+- **0.26% fewer tokens than `cl100k_base`** overall.
+- **17.20 MB/s** four-core warm-cache batch encoding.
+- **17.10 MB/s** four-core fresh-instance batch encoding.
+- **3,412/3,412 successful lossless round trips.**
+- Native C++ engine with configurable parallel batch encoding.
+
+## Benchmarks
+
+### Token compression
+
+![BloatTokenizer32K Compression Benchmark](images/One.png)
+
+
+### Token compression results
+
+Lower token counts are better.
+
+| Tokenizer | Vocabulary | Output tokens |
+|---|---:|---:|
+| **BloatTokenizer32K** | **32,000** | **997,416** |
+| GPT-4o (`o200k_base`) | 200,000 | 998,383 |
+| GPT-4 (`cl100k_base`) | 100,261 | 1,000,000 |
+| Qwen2.5-Coder | 151,665 | 1,018,726 |
+| Mistral NeMo | 131,072 | 1,052,365 |
+| Phi-2 | 50,295 | 1,159,396 |
+| GPT-2 | 50,257 | 1,278,138 |
+
+### Performance by domain
+
+| Domain | BloatTokenizer32K | GPT-4 tokenizer |
+|---|---:|---:|
+| English | 522,543 | 500,000 |
+| Mathematics | 100,805 | 100,000 |
+| Code | **374,068** | 400,000 |
+| **Total** | **997,416** | **1,000,000** |
+
+The tokenizer's strongest measured compression advantage is on code. English and mathematics require slightly more tokens than `cl100k_base` on this evaluation.
+
+
+### Encoding performance
+
+![BloatTokenizer32K Speed Benchmark](images/Two.png)
+
+### Encoding speed
+
+
+Measured in input MB/s, using matched CPU budgets.
+
+| Tokenizer | 1-core warm | 4-core warm | 4-core fresh |
+|---|---:|---:|---:|
+| **BloatTokenizer32K** | **8.53** | **17.20** | **17.10** |
+| `cl100k_base` | 10.28 | 12.37 | 13.15 |
+| `o200k_base` | 13.83 | 20.91 | 19.37 |
+| Qwen2.5-Coder | 2.09 | 4.03 | 4.19 |
+| GPT-2 | 9.77 | 8.00 | 13.57 |
+
+In this configuration, BloatTokenizer32K achieved approximately **39% higher warm-cache four-core batch throughput** than `cl100k_base`, and approximately **30% higher fresh-instance batch throughput**.
+
+Single-core encoding remains slower than both OpenAI tokenizer implementations.
+
+## Evaluation methodology
+
+The benchmark used identical inputs and matched CPU budgets across 13 tokenizers.
+
+| Property | Configuration |
+|---|---|
+| Excerpts | 3,412 |
+| Input size | 4,427,375 UTF-8 bytes |
+| Reference tokens | 1,000,000 (`cl100k_base`) |
+| Corpus distribution | 50% English, 10% math, 40% code |
+| Trials | 5 per configuration |
+| Serial CPU budget | 1 logical CPU |
+| Batch CPU budget | 4 logical CPUs |
+| Decode failures (BloatTokenizer32K) | 0 |
+
+All measurements report median throughput. Fresh-instance measurements exclude tokenizer construction time; warm-cache measurements follow an untimed corpus pass.
+
+**Limitations:** The evaluation corpus was excluded from tokenizer training but was previously inspected during development. Results are specific to this corpus and hardware configuration. Multilingual performance beyond the evaluated distribution has not been established, and baseline training-data overlap is unknown.
+
+## Getting started
+
+The tokenizer artifacts are available in the `artifacts/` directory of this repository.
+
+The native encoding engine and Python interface are maintained on GitHub.
+
+### Requirements
+
+- Python 3.12 or 3.14 for prebuilt Windows extensions
+- Windows x64 for bundled binaries
+- C++17 and `pybind11` for building from source on other platforms
+
+### Download
+
+Install the Hugging Face Hub client:
+
+```bash
+pip install -U huggingface_hub
+```
+
+Download the tokenizer artifacts:
+
+```python
+from huggingface_hub import snapshot_download
+
+path = snapshot_download(
+    repo_id="Qvinfor/BloatTokenizer32K",
+    repo_type="model",
+)
+
+print(path)
+```
+
+Replace `YOUR_HF_USERNAME` with the Hugging Face repository owner.
+
+### Python usage
+
+First obtain the Python/C++ engine from the [GitHub repository](https://github.com/Dev-qvinfor/BloatTokenizer32K) and follow its installation instructions.
+
+Then load the downloaded Hugging Face artifacts:
+
+```python
+from pathlib import Path
+from huggingface_hub import snapshot_download
+from curriculum_tokenizer import CurriculumTokenizer
+
+repo_path = snapshot_download(
+    repo_id="YOUR_HF_USERNAME/BloatTokenizer32K"
+)
+
+tokenizer = CurriculumTokenizer.from_pretrained(
+    str(Path(repo_path) / "artifacts")
+)
+
+text = "Fast, lossless text encoding."
+
+ids = tokenizer.encode(
+    text,
+    add_special_tokens=False
+)
+
+decoded = tokenizer.decode(
+    ids,
+    skip_special_tokens=False
+)
+
+assert decoded == text
+
+print("Token IDs:", ids)
+print("Decoded:", decoded)
+```
+
+By default, `encode()` adds `<|endoftext|>` and `decode()` skips control tokens. Use the explicit options above when a round-trip should reproduce the input.
+
+### Batch tokenization
+
+```python
+texts = [
+    "Hello, world!",
+    "def add(a, b): return a + b",
+    "2 + 2 = 4"
+]
+
+batch = tokenizer.encode_batch(
+    texts,
+    add_special_tokens=False,
+    num_threads=4
+)
+
+print(batch["input_ids"])
+```
+
+Batch encoding preserves input order.
+
+**Compatibility note:** The published artifacts use the BloatTokenizer runtime. Direct loading through Hugging Face `AutoTokenizer` is not currently documented or guaranteed.
+
+## Special tokens
+
+The vocabulary includes 16 reserved control tokens:
+
+| Token | ID |
+|---|---:|
+| `<|endoftext|>` | 256 |
+| `<|pad|>` | 257 |
+| `<|system|>` | 258 |
+| `<|user|>` | 259 |
+| `<|assistant|>` | 260 |
+| `<|end_turn|>` | 261 |
+| `<|tool_call|>` | 262 |
+| `<|end_tool_call|>` | 263 |
+| `<|tool_result|>` | 264 |
+| `<|end_tool_result|>` | 265 |
+| `<|final|>` | 266 |
+| `<|end_final|>` | 267 |
+| `<|think|>` | 268 |
+| `<|end_think|>` | 269 |
+| `<|verify|>` | 270 |
+| `<|end_verify|>` | 271 |
+
+Applications determine how these control markers are used in prompts and structured conversations.
+
+## Model compatibility
+
+BloatTokenizer32K is intended for models trained using its vocabulary and token-ID mapping.
+
+It is not a drop-in replacement for unrelated pretrained models. Changing tokenizers requires appropriate model adaptation.
+
+## License
+
+**Apache License 2.0**
+
+The publicly released tokenizer artifacts and runtime are open-source.
+
+The proprietary vocabulary-training algorithm and its training implementation are not included.
+
+## Source code and feedback
+
+The full C++ runtime, Python bindings, usage examples, and technical documentation are available on GitHub.
+
+**[BloatTokenizer32K — GitHub](https://github.com/Dev-qvinfor/BloatTokenizer32K)**
+
+Bug reports, independent benchmarks, performance comparisons, and community contributions are welcome.