fasthypernet-11 / tokenizer_config.json

FastHyperNet-125M Hybrid Multilingual Model Complete (Trie Accelerated, 8K Semantically Aligned Space, Grammar Protective)

f1884ad verified 2 months ago

199 Bytes

	{
	"tokenizer_class": "HFCompatibleHybridTokenizer",
	"vocab_size": 8192,
	"model_max_length": 512,
	"bos_token": "<bos>",
	"eos_token": "<eos>",
	"unk_token": "<unk>",
	"pad_token": "<pad>"
	}