| --- |
| language: |
| - en |
| - hi |
| - ta |
| - te |
| - es |
| - fr |
| - de |
| - zh |
| license: mit |
| tags: |
| - tokenizer |
| - bpe |
| - byte-level |
| - code |
| - stem |
| - neuromorphic |
| - synaptic-edge |
| - pytorch |
| - triton |
| pipeline_tag: text-generation |
| --- |
| |
| # Synaptic Edge 65,536 ByteLevel BPE Tokenizer |
|
|
| The official **65,536-vocabulary ByteLevel BPE Tokenizer** custom-engineered for the **Synaptic Edge 150M & 1B Neuromorphic Foundation Models**. |
|
|
| Trained on over **1.6+ Billion tokens** of source code, AI system kernels, formal mathematics, and multilingual texts. |
|
|
| --- |
|
|
| ## Key Highlights |
|
|
| - **Exact $2^{16}$ Alignment**: Vocabulary size is strictly **65,536**, ensuring exact power-of-2 alignment with GPU warp registers and Triton megakernel tensor cores. |
| - **Zero OOV / Zero `<unk>`**: ByteLevel fallback guarantees that every byte of UTF-8 text can be processed losslessly without out-of-vocabulary errors. |
| - **Code & Kernel Optimized (~70% Mixture)**: |
| - Multi-space indentation tokens (` `, ` `, ` `, `\t`) eliminate token explosion in nested code blocks. |
| - Pre-seeded with Andrej Karpathy's `nanoGPT`/`llm.c`, OpenAI Triton megakernels, BitNet 1.58b STE, and Lean 4 formal proofs. |
| - Delivers a **~2.2× compression ratio improvement** over standard NLP tokenizers on Python, C++, Rust, and CUDA. |
| - **Structural Repo-Level Scoping**: Built-in first-class scoping tags (`<repo>`, `</repo>`, `<file>`, `</file>`, `<think>`, `</think>`) for continuous multi-file repository pretraining. |
| - **1,000,000-Token Native Context**: Calibrated for ultra-long context streams paired with **3D Bit-RoPE** and **Continual Dynamic Synaptic Plasticity**. |
|
|
| --- |
|
|
| ## Domain Mixture Breakdown |
|
|
| | Domain | Ingested Samples / Files | Key Sources | |
| | :--- | :--- | :--- | |
| | **Code & AI Systems (45%)** | 455,000+ files | `codeparrot-clean`, `Magicoder-OSS-75K`, `CodeFeedback`, `Evol-Instruct-Code` | |
| | **PCMB & Formal Math (25%)** | 300,000+ docs | `open-web-math`, LaTeX equations, Lean 4 theorems, SMILES molecular formulas | |
| | **Natural Language & Multi (30%)** | 330,000+ articles | `fineweb-edu`, Aya Multilingual (65 languages), Wikipedia (7 global languages) | |
|
|
| --- |
|
|
| ## Quickstart |
|
|
| ### 1. Using Hugging Face `transformers` |
| ```python |
| from transformers import AutoTokenizer |
| |
| tokenizer = AutoTokenizer.from_pretrained('SurendraVB/Synaptic-Edge-Tokenizer-65K') |
| |
| code_snippet = '''@triton.jit |
| def fused_kernel(a_ptr, b_ptr, BLOCK_M: tl.constexpr): |
| pass''' |
| |
| tokens = tokenizer(code_snippet) |
| print('Token IDs:', tokens.input_ids) |
| print('Decoded:', tokenizer.decode(tokens.input_ids)) |
| ``` |
|
|
| ### 2. Using Fast Rust `tokenizers` |
| ```python |
| from tokenizers import Tokenizer |
| |
| tokenizer = Tokenizer.from_pretrained('SurendraVB/Synaptic-Edge-Tokenizer-65K') |
| output = tokenizer.encode('def forward(self, x: torch.Tensor):\n return x * 2') |
| print('Subwords:', output.tokens) |
| ``` |
|
|
| --- |
|
|
| ## Special Scoping & Indentation Tokens |
|
|
| | Token | Purpose | Impact | |
| | :--- | :--- | :--- | |
| | ` ` (4 spaces) | Standard indentation | Encodes a 4-space tab in a single token instead of 4 separate tokens | |
| | ` ` (8 spaces) | Kernel / Deep indentation | Compresses 8 spaces into a single atomic ID | |
| | `<repo>`, `</repo>` | Repository scoping | Signals repository boundaries in streaming pretraining | |
| | `<file>`, `</file>` | File boundaries | Delimits multi-file continuous context streams | |
| | `<think>`, `</think>` | Cognitive attractor reasoning | Wraps cognitive latent attractor settling traces | |
|
|
| --- |
|
|
| ## Citation & Architecture |
|
|
| ```bibtex |
| @misc{synaptic_edge_2026, |
| author = {Surendra V B}, |
| title = {Synaptic Edge: 1-Bit Ternary Continual Learning Neuromorphic Foundation Architecture}, |
| year = {2026}, |
| publisher = {Hugging Face}, |
| howpublished = {\url{https://huggingface.co/SurendraVB/Synaptic-Edge-Tokenizer-65K}} |
| } |
| ``` |
|
|
