compacttest-5m / tokenizer_config.json
Compactbot's picture
Add tokenizer_config.json, model.py, .gitattributes and README (renamed copy of gpt-s2.5-5m) (#5)
9014085
Raw
History Blame Contribute Delete
253 Bytes
{
"tokenizer_class": "GPTS25BPE",
"model_type": "gpt-s2.5",
"vocab_size": 8192,
"add_prefix_space": false,
"note": "8192-vocab BPE (byte-level). This is a custom BPE, not a standard HF tokenizer; load bpe_8k.json / tokenizer.json directly."
}