Instructions to use syntriq/lupe-multilingual-reranker-large with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- sentence-transformers
How to use syntriq/lupe-multilingual-reranker-large with sentence-transformers:
from sentence_transformers import CrossEncoder model = CrossEncoder("syntriq/lupe-multilingual-reranker-large") query = "Which planet is known as the Red Planet?" passages = [ "Venus is often called Earth's twin because of its similar size and proximity.", "Mars, known for its reddish appearance, is often referred to as the Red Planet.", "Jupiter, the largest planet in our solar system, has a prominent red spot.", "Saturn, famous for its rings, is sometimes mistaken for the Red Planet." ] scores = model.predict([(query, passage) for passage in passages]) print(scores) - Notebooks
- Google Colab
- Kaggle
Lupe Multilingual Reranker Large
Deutsch: Lupe Multilingual Reranker Large sortiert Suchtreffer nach Relevanz, auf Deutsch, Englisch und 34 weiteren Sprachen, etwa als zweiter Schritt in Suche und RAG. In unserer Auswertung führt es auf MIRACL im Mittel, auf Deutsch und auf Englisch; auf NanoBEIR und GermanDPR liegt das fast dreimal so große mxbai-rerank-large-v2 vorn. Weitertrainiert aus bge-reranker-v2-m3 durch Destillation aus einem größeren offenen Reranker, ohne echte personenbezogene Daten.
A 568M-parameter multilingual cross-encoder for reranking search results in German, English and 34 more languages, built on BAAI/bge-reranker-v2-m3. In our evaluation it improves on bge-reranker-v2-m3 on every set and leads on MIRACL.
Quickstart
from sentence_transformers import CrossEncoder
model = CrossEncoder("syntriq/lupe-multilingual-reranker-large")
query = "Wie hoch ist die Zugspitze?"
passages = ["Die Zugspitze ist mit 2962 Metern der höchste Berg Deutschlands.", "Der Rhein ist ein Fluss in Mitteleuropa."]
print(model.rank(query, passages))
ONNX
ONNX exports for CPU inference are in onnx/: onnx/model.onnx with its weights in onnx/model.onnx_data (fp32) and onnx/model_qint8_avx512_vnni.onnx (int8, dynamic quantization). Install with pip install "sentence-transformers[onnx]".
from sentence_transformers import CrossEncoder
model = CrossEncoder("syntriq/lupe-multilingual-reranker-large", backend="onnx", model_kwargs={"file_name": "onnx/model_qint8_avx512_vnni.onnx", "provider": "CPUExecutionProvider"})
query = "Wie hoch ist die Zugspitze?"
passages = ["Die Zugspitze ist mit 2962 Metern der höchste Berg Deutschlands.", "Der Rhein ist ein Fluss in Mitteleuropa."]
print(model.rank(query, passages))
Use "file_name": "onnx/model.onnx" for fp32. "provider": "CPUExecutionProvider" keeps ONNX Runtime on the CPU on every platform.
Agreement with the PyTorch model on CPU: 100 query-passage pairs (50 German, 50 English) and top-1 passage on 20 ranking lists of 16 passages, all from our validation split.
| Format | Max absolute score difference (scores 0 to 1) | Spearman | Top-1 agreement |
|---|---|---|---|
| ONNX fp32 | 5.7e-06 | 1.0000 | 100% |
| ONNX int8 | 0.117 | 0.9947 | 100% |
Speed on an Apple M5 CPU (default thread settings, batch 32, 512 tokens per pair):
| Backend | Pairs per second |
|---|---|
| PyTorch fp32 | 4.3 |
| ONNX fp32 | 3.2 |
| ONNX int8 | 4.3 |
On this ARM CPU, ONNX is not faster than PyTorch. The int8 file is quantized for x86 CPUs with AVX-512 VNNI; we did not measure its speed there.
Results
nDCG@10. Every model ran through the same evaluation harness on the same queries: up to 200 queries per set, the first 200 of each MIRACL language and of GermanQuAD and GermanDPR (the MIRACL sets for Korean, Telugu and Yoruba have 186, 74 and 89). MIRACL reranks the candidates of the MTEB MIRACLReranking task (dev split) in 18 languages, GermanQuAD and GermanDPR rerank the top 100 passages of intfloat/multilingual-e5-base, NanoBEIR is the English sentence-transformers CrossEncoderNanoBEIREvaluator (13 sets, top 100 BM25 candidates).
| Model | MIRACL mean (18 languages) | MIRACL-de | MIRACL-en | NanoBEIR | GermanQuAD | GermanDPR | Parameters |
|---|---|---|---|---|---|---|---|
| syntriq/lupe-multilingual-reranker-large | 0.829 | 0.794 | 0.767 | 0.712 | 0.990 | 0.868 | 568M |
| BAAI/bge-reranker-v2-m3 | 0.821 | 0.774 | 0.748 | 0.694 | 0.989 | 0.863 | 568M |
| Qwen/Qwen3-Reranker-0.6B | 0.790 | 0.766 | 0.709 | not measured | 0.990 | 0.852 | 596M |
| mixedbread-ai/mxbai-rerank-large-v2 | not measured | 0.702¹ | 0.730¹ | 0.731 | 0.990 | 0.876 | 1.54B |
mixedbread-ai/mxbai-rerank-large-v2, almost three times the size, wins NanoBEIR (0.731 vs 0.712) and GermanDPR (0.876 vs 0.868). GermanQuAD is level between this model, Qwen/Qwen3-Reranker-0.6B and mxbai-rerank-large-v2. This model leads on MIRACL-de, MIRACL-en and, among the models where it was measured, the MIRACL mean.
¹ MIRACL-de and MIRACL-en of the mxbai v2 models come from the evaluation log of the same run: the process that held these languages ran out of GPU memory before it wrote its result file. Their MIRACL mean is not measured because Hindi did not finish.
Full MIRACL dev sets, all queries:
| Model | MIRACL-de (221 queries) | MIRACL-en (705 queries) |
|---|---|---|
| syntriq/lupe-multilingual-reranker-large | not measured | not measured |
| BAAI/bge-reranker-v2-m3 | 0.780 | 0.748 |
This model was not run on the full sets; bge-reranker-v2-m3 is shown for reference.
Caveats: 200 queries per set and no significance test, so differences of a few thousandths are not meaningful. Checkpoints were selected on our own validation split, and the choice between training stages and averaged variants also took these public sets into account. The English training data contains training splits of datasets whose test splits are part of NanoBEIR (for example NQ, HotpotQA, FEVER, FiQA).
Intended use and limitations
- Reranking only: a cross-encoder that scores query-passage pairs. Retrieve 50 to 200 candidates with an embedding model or BM25 first, then rerank them with this model. It does not produce embeddings.
- Trained with a maximum input length of 512 tokens for query plus passage. Longer inputs are accepted by the architecture but were not trained or tested; split long documents into passages.
- Trained on 36 languages, evaluated on the 18 MIRACL languages plus the German (GermanQuAD, GermanDPR) and English (NanoBEIR) sets. Languages outside MIRACL (for example Dutch, Polish or Turkish) were not evaluated.
- The teacher is weaker in some languages: in the reviewed disagreements the judge sided against the teacher most often for Yoruba and Swahili, so quality there is less certain.
- Scores are meant for ranking within one query. They are not calibrated probabilities; check any fixed threshold on your own data.
- Results come from 200-query evaluations without a significance test.
Training data
Query-passage pairs in 36 languages: ar, bg, bn, ca, cs, da, de, el, en, es, fa, fi, fr, he, hi, hu, id, it, ja, ko, nl, no, pl, pt, ro, ru, sk, sv, sw, te, th, tr, uk, vi, yo, zh. Every pair carries a teacher score.
| Source | Used for | License |
|---|---|---|
| MIRACL train (sentence-transformers/miracl, from miracl/miracl) | Real queries with passages, 16 languages | Apache-2.0 |
| Wikipedia (wikimedia/wikipedia, 2023-11-01) | Passages | CC BY-SA 3.0 and GFDL |
| FineWeb-2 (HuggingFaceFW/fineweb-2) | Web passages | ODC-By 1.0, subject to the Common Crawl terms of use |
| FinePDFs (HuggingFaceFW/finepdfs), German part | German passages | ODC-By 1.0, subject to the Common Crawl terms of use |
| WebFAQ (PaDaS-Lab/webfaq-retrieval) | Real FAQ questions and answers | CC BY 4.0 |
| English pairs (cross-encoder/ettin-reranker-v1-data, MS MARCO parts removed) | 7,000,000 English pairs | Apache-2.0 for the collection; the upstream sources keep their own terms |
| Synthetic queries | About 728,000 multilingual and about 198,000 German queries (a question, a keyword query and an English question per passage), written by an open-weight large language model released under the MIT license | Generated by Syntriq |
| Synthetic German business documents | German passages from the synthetic training documents of the Syntriq Tarn PII models | Generated by Syntriq |
- Multilingual part (35 languages): 717K passages and 253K real queries (WebFAQ questions, MIRACL train questions) plus the synthetic queries. Each query has its positive passage and up to seven hard negatives mined with Snowflake/snowflake-arctic-embed-l-v2.0 (Apache-2.0).
- German part: the training data of syntriq/lupe-german-reranker-small, 4.0M pairs.
- Leakage control: Wikipedia articles and MIRACL train passages whose title starts a relevant passage of a MIRACL dev query were left out; German passages that share text with GermanQuAD, GermanDPR or relevant MIRACL-de passages were removed.
- No real personal data was used on purpose. Public web, FAQ and forum text in the sources can contain incidental names; the German business documents are synthetic.
- Data safety: 40 passages from the public sources (FineWeb-2, WebFAQ, MIRACL train and Wikipedia) matched spam patterns for abusive content. They were found and removed from the training data of the final training stage. The earlier training stage of the small and base models ran before this filter existed and included them; the large model was trained only on the filtered data.
Training
Distilled from a larger open reranker teacher (Apache-2.0). For every query-passage pair the teacher's score, the logit of "yes" minus the logit of "no", is the regression target, and the student learns it with mean squared error on single pairs (sentence-transformers MSELoss). One pass over the chosen pairs, maximum length 512 tokens, bf16, warmup 3 %, weight decay 0.01, seed 0; five validation checks per run on 2,160 held-out queries in 36 languages, the best checkpoint kept. All 7,000,000 English pairs carry teacher scores.
Label corrections: about 15,000 cases where the teacher ranked a mined negative above the labeled positive were reviewed by a large-language-model judge, and its verdicts corrected the labels before the final stage: 7,271 positives were raised above their negatives, 5,718 negatives judged relevant were raised to the positive's score, and in 1,984 cases the teacher's score was kept.
Lineage: BAAI/bge-reranker-v2-m3 → one multilingual stage: 11,624,580 pairs, learning rate 5e-6, 1,152 pairs per step (192 on each of 6 GPUs), 96.2 minutes, started from BAAI/bge-reranker-v2-m3, on the first part of the shuffled 18,582,780-pair data set (11,582,780 multilingual and German, 7,000,000 English).
The released weights are the element-wise average of the fine-tuned model and the original BAAI/bge-reranker-v2-m3 (0.5 each), with no further training. The fine-tuned model alone was slightly below bge-reranker-v2-m3 on the MIRACL mean (0.819 vs 0.821) and level on MIRACL-en (0.748 vs 0.748); the average is above both on the MIRACL mean (0.829), MIRACL-de (0.794) and MIRACL-en (0.767). A 0.7/0.3 average was also measured and not chosen.
Hardware: one node with 8× NVIDIA B200 GPUs; each run used several of them in parallel (distributed data parallel).
License and citation
Apache-2.0. Base models: BAAI/bge-reranker-v2-m3 (Apache-2.0). Teacher: Apache-2.0. Data licenses as listed above.
@misc{syntriq2026lupemultilinguallarge,
title = {Lupe Multilingual Reranker Large},
author = {Syntriq Labs},
year = {2026},
url = {https://huggingface.co/syntriq/lupe-multilingual-reranker-large}
}
- Downloads last month
- 17
Model tree for syntriq/lupe-multilingual-reranker-large
Base model
BAAI/bge-reranker-v2-m3