| language: | |
| - en | |
| - multilingual | |
| - code | |
| license: apache-2.0 | |
| tags: | |
| - tokenizer | |
| - bpe | |
| - omni | |
| - binary-analysis | |
| # Omni-Tokenizer (Experimental 539k Fusion) | |
| This is a highly experimental, massive **539,306-token Byte-Level BPE tokenizer** designed for extreme-scale "Omni" models. It is built to seamlessly process natural languages, highly dense code, and raw binary/machine executables natively. | |
| ## Composition | |
| This tokenizer was created by extracting and perfectly fusing the vocabularies of several state-of-the-art tokenizers into a single LLaMA-3 base: | |
| 1. `meta-llama/Meta-Llama-3-8B` (Base Byte-Level Foundation) | |
| 2. `google/gemma-7b` | |
| 3. `Qwen/Qwen1.5-7B` | |
| 4. `CohereForAI/c4ai-command-r-v01` | |
| 5. `microsoft/Phi-3-mini-4k-instruct` | |
| 6. `mjbommar/binary-tokenizer-001-64k` (Binary Analysis / Malware) | |
| 7. `mjbommar/binary-tokenizer-001-32k` | |
| ## Technical Details | |
| - **Vocabulary Size:** 539,306 | |
| - **Base Architecture:** Byte-Level BPE (No Unknown Tokens) | |
| - **Use Cases:** Multilingual NLP, Code Generation, Binary/Malware Analysis, Reverse Engineering. | |
| **Note on Usage:** Due to the massive 540k vocabulary size, this tokenizer will create an embedding matrix of roughly ~2.2 Billion parameters (at 4096 dimensions). It is intended for large-scale experimental models where extreme compression and cross-domain tokenization is required. | |
| ## How to use | |
| ```python | |
| from transformers import AutoTokenizer | |
| tokenizer = AutoTokenizer.from_pretrained("SurendraVB/omni-tokenizer") | |
| ``` | |