BeardedMonster
/

sabiyarn-3B

african-languages

nigerian-languages

Model card Files Files and versions

9.75 MB

Ctrl+K

Ctrl+K

1 contributor

History: 2 commits

BeardedMonster's picture

SabiYarn 3B tokenizer: 131,072-token byte-level BPE with NFC normalizer, special ids 0-179 unchanged, tokenizer comparison across 73 languages

e7c19f5 verified 1 day ago

.gitattributes

1.52 kB
initial commit 1 day ago
README.md

23.8 kB
SabiYarn 3B tokenizer: 131,072-token byte-level BPE with NFC normalizer, special ids 0-179 unchanged, tokenizer comparison across 73 languages 1 day ago
chat_template.jinja

4.5 kB
SabiYarn 3B tokenizer: 131,072-token byte-level BPE with NFC normalizer, special ids 0-179 unchanged, tokenizer comparison across 73 languages 1 day ago
tokenizer.json

9.72 MB
SabiYarn 3B tokenizer: 131,072-token byte-level BPE with NFC normalizer, special ids 0-179 unchanged, tokenizer comparison across 73 languages 1 day ago
tokenizer_config.json

1.48 kB
SabiYarn 3B tokenizer: 131,072-token byte-level BPE with NFC normalizer, special ids 0-179 unchanged, tokenizer comparison across 73 languages 1 day ago