PhiloLM / tokenizer_config.json
gustavecortal's picture
Unix line endings
1b603cd verified
Raw History Blame Contribute Delete
234 Bytes
{
"tokenizer_class": "PreTrainedTokenizerFast",
"bos_token": "<|endoftext|>",
"eos_token": "<|endoftext|>",
"pad_token": "<|endoftext|>",
"padding_side": "left",
"model_max_length": 4096,
"clean_up_tokenization_spaces": false
}