banana-idk-3m / tokenizer.json

Commit History

BPE tokenizer (8192 vocab, 5922 merges) trained on FineWeb-Edu
d184854
verified

Compactbot commited on

Add tokenizer files
f26570e
verified

Compactbot commited on