How to use p1atdev/multi-tokenizers-processor-sample with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("p1atdev/multi-tokenizers-processor-sample", device_map="auto")
https://huggingface.co/p1atdev/multi-tokenizers-processor-sample/resolve/035c85dfbc32d1bc3ff0f4f424c7d6f64463fce5/tokenizer_1/tokenizer.json
hf download hf://p1atdev/multi-tokenizers-processor-sample@035c85dfbc32d1bc3ff0f4f424c7d6f64463fce5/tokenizer_1/tokenizer.json
curl -L -o tokenizer.json https://huggingface.co/p1atdev/multi-tokenizers-processor-sample/resolve/035c85dfbc32d1bc3ff0f4f424c7d6f64463fce5/tokenizer_1/tokenizer.json
