How to use adkhamboy/custom_tokenizer_large_python with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("adkhamboy/custom_tokenizer_large_python", device_map="auto")
https://huggingface.co/adkhamboy/custom_tokenizer_large_python/resolve/2e2f1cc1fb8ea077e362b3a2a62693461e0646dd/tokenizer.json
hf download hf://adkhamboy/custom_tokenizer_large_python@2e2f1cc1fb8ea077e362b3a2a62693461e0646dd/tokenizer.json
curl -L -o tokenizer.json https://huggingface.co/adkhamboy/custom_tokenizer_large_python/resolve/2e2f1cc1fb8ea077e362b3a2a62693461e0646dd/tokenizer.json
