SBDO2 / tokenizer.json
SBDO1's picture
Upload folder using huggingface_hub
006d174 verified
Raw
History Blame Contribute Delete
153 Bytes
{
"model": {
"type": "BPE",
"prefix": ""
},
"vocab": {},
"merges": [],
"special_tokens": {
"<|endoftext|>": 50256
}
}