First5M / tokenizer_config.json
kaafivikrant's picture
Upload folder using huggingface_hub
1be11eb verified
Raw History Blame Contribute Delete
183 Bytes
{
"tokenizer_type": "tiktoken",
"encoding": "gpt2",
"vocab_size": 50257,
"install": "pip install tiktoken",
"usage": "import tiktoken; enc = tiktoken.get_encoding('gpt2')"
}