PP-Hydrogen / tokenizer /meta.json
AdminHugging's picture
Upload tokenizer/meta.json with huggingface_hub
f890162 verified
Raw History Blame Contribute Delete
183 Bytes
{
"vocab_size": 24576,
"special_tokens": [
"<|pad|>",
"<|bos|>",
"<|eos|>"
],
"sample_chars": 40062469,
"chars_per_token": {
"en": 3.18,
"py": 3.11
}
}