tokenizer_v6 / tokenizer.json

Commit History

  1. Add tokenizer.json (vocab 32000, BPE, fit on fineweb-edu-dedup 1.6M + k-10-12-mixed)

    Verified signature
    acbb711