sabiyarn-3B / tokenizer.json
BeardedMonster's picture
SabiYarn 3B tokenizer: 131,072-token byte-level BPE with NFC normalizer, special ids 0-179 unchanged, tokenizer comparison across 73 languages
e7c19f5 verified
Raw History Contribute Delete
9.72 MB
File too large to display, you can check the raw version instead.