Ivme-Conversate-v3.1-Base / tokenizer_config.json
ereniko's picture
Add Ivme-Conversate-v3.1-Base (CPT of v3 on 31.4B dense tokens)
07d8f8e verified
Raw History Blame Contribute Delete
181 Bytes
{
"tokenizer_class": "PreTrainedTokenizerFast",
"eos_token": "<|endoftext|>",
"bos_token": "<|endoftext|>",
"pad_token": "<|pad|>",
"clean_up_tokenization_spaces": false
}