How to use datapaf/starcoder2_racket_tokenizer_expanded with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("datapaf/starcoder2_racket_tokenizer_expanded", device_map="auto")
https://huggingface.co/datapaf/starcoder2_racket_tokenizer_expanded/resolve/ae8c8d3327d7451ca8f565c7c64cae4852fbaee5/tokenizer.json
hf download hf://datapaf/starcoder2_racket_tokenizer_expanded@ae8c8d3327d7451ca8f565c7c64cae4852fbaee5/tokenizer.json
curl -L -o tokenizer.json https://huggingface.co/datapaf/starcoder2_racket_tokenizer_expanded/resolve/ae8c8d3327d7451ca8f565c7c64cae4852fbaee5/tokenizer.json
