DeadBranches/text-generation / inbox /tokenizer_config.json
DeadBranches's picture
download
raw
243 Bytes
{
"backend": "tokenizers",
"bos_token": "<bos>",
"eos_token": "<eos>",
"is_local": true,
"model_max_length": 1000000000000000019884624838656,
"pad_token": "<pad>",
"tokenizer_class": "TokenizersBackend",
"unk_token": "<unk>"
}

Xet Storage Details

Size:
243 Bytes
·
Xet hash:
72f79875edb3e1dac8db2e4e2e2e9f58e5f0028c5c3eeda05e6a00d6c2f8d39d

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.