Xenova
/

CodeGPT-tokenizer

CodeGPT-tokenizer / added_tokens.json

Xenova HF Staff

Upload 6 files

d496768 verified about 2 years ago

45 Bytes

{
  "<EOL>": 50000,
  "<|UNKNOWN|>": 50001
}