tokenizer_v6 / tokenizer.json
sujitpandey's picture
Add tokenizer.json (vocab 32000, BPE, fit on fineweb-edu-dedup 1.6M + k-10-12-mixed)
acbb711 verified
Raw History Contribute Delete
1.13 MB
File too large to display, you can check the raw version instead.