ocolegro
/

custom-tokenizer

custom-tokenizer / special_tokens_map.json

Upload tokenizer

753d073 over 3 years ago

123 Bytes

{
  "bos_token": "<|endoftext|>",
  "eos_token": "<|endoftext|>",
  "pad_token": "[PAD]",
  "unk_token": "<|endoftext|>"
}