shangkuns
/

b1k-eval-utils

b1k-eval-utils / tokenizer_config.json

Upload folder using huggingface_hub

9bca14a verified 8 months ago

149 Bytes

{
  "tokenizer_class": "PaligemmaTokenizer",
  "max_len": 200,
  "auto_map": {
    "AutoTokenizer": ["tokenizer_pi05.PaligemmaTokenizer", null]
  }
}

	{
	"tokenizer_class": "PaligemmaTokenizer",
	"max_len": 200,
	"auto_map": {
	"AutoTokenizer": ["tokenizer_pi05.PaligemmaTokenizer", null]
	}
	}