sat-12l-sm-coreml / tokenizer_config.json

Core ML conversion of segment-any-text/sat-12l-sm (int8 embeddings) with the XLM-R tokenizer

62c68b3 verified 3 days ago

153 Bytes

{"tokenizer_class": "XLMRobertaTokenizer", "model_max_length": 512, "bos_token": "<s>", "eos_token": "</s>", "unk_token": "<unk>", "pad_token": "<pad>"}

	{"tokenizer_class": "XLMRobertaTokenizer", "model_max_length": 512, "bos_token": "<s>", "eos_token": "</s>", "unk_token": "<unk>", "pad_token": "<pad>"}