tokenizer_config.json · rpg67/Affine.Earth at main

Affine.Earth / tokenizer_config.json

Add config.json and tokenizer_config.json for HF download counter tracking

13d97f3 verified 1 day ago

154 Bytes

{
  "tokenizer_class": "PreTrainedTokenizerFast",
  "model_max_length": 262144,
  "study51_iso_languages": 160,
  "clean_up_tokenization_spaces": false
}

	{
	"tokenizer_class": "PreTrainedTokenizerFast",
	"model_max_length": 262144,
	"study51_iso_languages": 160,
	"clean_up_tokenization_spaces": false
	}