grt-20m-curriculum / tokenizer_config.json
davidhouseholter's picture
Initial release of MatterWaveAI/grt-20m-curriculum (weights and SLM-Bench curriculum evaluation)
1ec5707
Raw History Blame Contribute Delete
175 Bytes
{
"tokenizer_class": "PreTrainedTokenizerFast",
"bos_token": "<bos>",
"eos_token": "<eos>",
"unk_token": "<unk>",
"pad_token": "<pad>",
"model_max_length": 1024
}