sabiyarn-3B / tokenizer.json

Commit History

3 days ago

  1. SabiYarn 3B tokenizer: 131,072-token byte-level BPE with NFC normalizer, special ids 0-179 unchanged, tokenizer comparison across 73 languages

    Verified signature
    e7c19f5