How to use schreon/xnext-lhm_passages_encoder with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, DPRContextEncoder tokenizer = AutoTokenizer.from_pretrained("schreon/xnext-lhm_passages_encoder") model = DPRContextEncoder.from_pretrained("schreon/xnext-lhm_passages_encoder", device_map="auto")
https://huggingface.co/schreon/xnext-lhm_passages_encoder/resolve/main/tokenizer.json
hf download hf://schreon/xnext-lhm_passages_encoder/tokenizer.json
curl -L -o tokenizer.json https://huggingface.co/schreon/xnext-lhm_passages_encoder/resolve/main/tokenizer.json
