privacy-filter-mic / tokenizer_config.json
tarekziade's picture
tarekziade HF Staff
Publish MIC usage, slim fixture and reviewed Transformers expectations
259b21a verified
Raw History Blame Contribute Delete
234 Bytes
{
"backend": "tokenizers",
"eos_token": "<|endoftext|>",
"model_input_names": [
"input_ids",
"attention_mask"
],
"model_max_length": 128000,
"pad_token": "<|endoftext|>",
"tokenizer_class": "TokenizersBackend"
}