smitathkr1
/

ord-tokenizer

ord-tokenizer / tokenizer_config.json

Upload tokenizer_config.json with huggingface_hub

f6b4285 verified about 1 year ago

141 Bytes

{
  "tokenizer_class": "ByteLevelBPETokenizer",
  "unk_token": "[UNK]",
  "bos_token": "<s>",
  "eos_token": "</s>",
  "pad_token": "[PAD]"
}

	{
	"tokenizer_class": "ByteLevelBPETokenizer",
	"unk_token": "[UNK]",
	"bos_token": "<s>",
	"eos_token": "</s>",
	"pad_token": "[PAD]"
	}