How to use SmallDoge/Doge-tokenizer with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("SmallDoge/Doge-tokenizer", device_map="auto")
https://huggingface.co/SmallDoge/Doge-tokenizer/resolve/8626dcc495dcafe7d2bc076ae428c408988c77fe/tokenizer.json
hf download hf://SmallDoge/Doge-tokenizer@8626dcc495dcafe7d2bc076ae428c408988c77fe/tokenizer.json
curl -L -o tokenizer.json https://huggingface.co/SmallDoge/Doge-tokenizer/resolve/8626dcc495dcafe7d2bc076ae428c408988c77fe/tokenizer.json
