andrew3d
/

qwen2-tiny-probe

qwen2-tiny-probe / tokenizer_config.json

add probe tokenizer and config

1db111a verified about 1 month ago

132 Bytes

{"tokenizer_class":"ProbeTokenizer","auto_map":{"AutoTokenizer":["tokenization_probe.ProbeTokenizer",null]},"model_max_length":512}

	{"tokenizer_class":"ProbeTokenizer","auto_map":{"AutoTokenizer":["tokenization_probe.ProbeTokenizer",null]},"model_max_length":512}