ldt-10m / tokenizer.json
Compactbot's picture
Byte-level BPE tokenizer (vocab 12288)
9049741 verified
Raw History
830 kB
File too large to display, you can check the raw version instead.