| library_name: tokenizers | |
| tags: | |
| - tokenizer | |
| - multilingual | |
| - byte-level-bpe | |
| - indigenous-languages | |
| - tachiwin | |
| # Tachiwin multilingual tokenizer | |
| A multilingual ByteLevel-BPE tokenizer trained with Hugging Face Tokenizers. | |
| ## Corpus weighting | |
| | Component | Target | | |
| |---|---:| | |
| | Modern + old exotic-language data | 70% | | |
| | English | 10% | | |
| | Spanish | 10% | | |
| | Code | 10% | | |
| The complete available exotic-language corpus is used as the 70% anchor. | |
| Its existing modern/old composition is preserved. | |
| ## Tokenizer | |
| - Model: BPE | |
| - Vocabulary target: 256,000 | |
| - Initial alphabet: complete ByteLevel alphabet | |
| - ByteLevel GPT-2 regex: disabled | |
| - Unicode normalizer: none | |
| - Special tokens: 126 | |
| - Human-language tags: 92 | |
| ## Corpus size | |
| Total materialized corpus: | |
| 113,921,401 bytes | |
| (0.106 GiB) | |
| ## Important training note | |
| The Hugging Face BPE trainer does not expose an internal resumable merge-state | |
| checkpoint. The recipe therefore treats the completed `tokenizer.json` as the | |
| training checkpoint: | |
| - corpus preparation is resumable; | |
| - recipe/statistics/checksums are stored in `recipe/`; | |
| - if `tokenizer.json` already exists, subsequent runs skip BPE training; | |
| - an interrupted BPE computation itself must be restarted. | |
| This avoids changing the training procedure merely to obtain artificial | |
| checkpointing. | |