Buckets:

omaryasser95
/

model_ocr-bucket

Files

xet

omaryasser95/model_ocr-bucket / tokenizer_config.json

omaryasser95

3 months ago

download

raw

329 Bytes

{
  "added_tokens_decoder": {},
  "auto_map": {
    "AutoTokenizer": [
      "tokenization_qwen.QWenTokenizer",
      null
    ]
  },
  "clean_up_tokenization_spaces": true,
  "eos_token": "<|im_end|>",
  "model_max_length": 8000,
  "pad_token": "<|endoftext|>",
  "padding_side": "right",
  "tokenizer_class": "QWenTokenizer"
}

Xet Storage Details

Size:: 329 Bytes
Xet hash:: c046328544138b1f9e87538632f869c0bb8f26fe5018b4a61d615514ebf783b5

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.

	{
	"added_tokens_decoder": {},
	"auto_map": {
	"AutoTokenizer": [
	"tokenization_qwen.QWenTokenizer",
	null
	]
	},
	"clean_up_tokenization_spaces": true,
	"eos_token": "<\|im_end\|>",
	"model_max_length": 8000,
	"pad_token": "<\|endoftext\|>",
	"padding_side": "right",
	"tokenizer_class": "QWenTokenizer"
	}