How to use M4rt0no/Tokenizer-Priorizacion-v2_batch32 with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("M4rt0no/Tokenizer-Priorizacion-v2_batch32", device_map="auto")
https://huggingface.co/M4rt0no/Tokenizer-Priorizacion-v2_batch32/resolve/main/tokenizer.json
hf download hf://M4rt0no/Tokenizer-Priorizacion-v2_batch32/tokenizer.json
curl -L -o tokenizer.json https://huggingface.co/M4rt0no/Tokenizer-Priorizacion-v2_batch32/resolve/main/tokenizer.json
