Aion-1.0-Instruct-2.3B / tokenizer_config.json
ersomme-msft's picture msalvaris's picture
Duplicate from asgmodels/Aion-Instruct-v1
1423a07
Raw History Blame Contribute Delete
341 Bytes
{
"add_prefix_space": false,
"backend": "tokenizers",
"bos_token": "<|endoftext|>",
"clean_up_tokenization_spaces": false,
"eos_token": "<|end|>",
"is_local": true,
"local_files_only": true,
"model_max_length": 32768,
"pad_token": "<|endoftext|>",
"tokenizer_class": "TokenizersBackend",
"unk_token": "<|endoftext|>"
}