mini-code-generator-v6 / v6_byte_tokenizer.json
quachtohai's picture
Release frozen V6 experiment
a2cb92a verified
Raw History Blame Contribute Delete
111 Bytes
{
"type": "utf8_byte",
"pad_id": 0,
"bos_id": 1,
"eos_id": 2,
"byte_offset": 3,
"vocab_size": 259
}