Byte-level BPE tokenizer (tokenizers JSON format).
- vocab size: 32000
- corpus: HuggingFaceFW/fineweb-2 (kat_Geor)
- trained by: lm.py --tokenize-to / training run
from tokenizers import Tokenizer
tok = Tokenizer.from_pretrained("ZurabDz/ka-bpe-32k")
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support