Instructions to use syntriq/lupe-multilingual-reranker-small with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- sentence-transformers
How to use syntriq/lupe-multilingual-reranker-small with sentence-transformers:
from sentence_transformers import CrossEncoder model = CrossEncoder("syntriq/lupe-multilingual-reranker-small") query = "Which planet is known as the Red Planet?" passages = [ "Venus is often called Earth's twin because of its similar size and proximity.", "Mars, known for its reddish appearance, is often referred to as the Red Planet.", "Jupiter, the largest planet in our solar system, has a prominent red spot.", "Saturn, famous for its rings, is sometimes mistaken for the Red Planet." ] scores = model.predict([(query, passage) for passage in passages]) print(scores) - Notebooks
- Google Colab
- Kaggle
Lupe Multilingual Reranker Small
Deutsch: Lupe Multilingual Reranker Small sortiert Suchtreffer nach Relevanz, auf Deutsch, Englisch und 34 weiteren Sprachen, etwa als zweiter Schritt in Suche und RAG. Mit 141 Millionen Parametern liegt es in unserer Auswertung auf allen sechs Testsets vor den verglichenen Modellen ähnlicher Größe, auf MIRACL-en nur knapp. Destilliert aus einem größeren offenen Reranker, ohne echte personenbezogene Daten trainiert. Nachfolger von syntriq/lupe-german-reranker-small, auf allen deutschen Testsets gleich gut oder besser.
A 141M-parameter multilingual cross-encoder for reranking search results in German, English and 34 more languages, a drop-in second stage for search and RAG. In our evaluation it leads its size class on MIRACL and the German sets. It replaces syntriq/lupe-german-reranker-small: same size, trained further from it, equal or better on every German set we measured.
Quickstart
from sentence_transformers import CrossEncoder
model = CrossEncoder("syntriq/lupe-multilingual-reranker-small")
query = "Wie hoch ist die Zugspitze?"
passages = ["Die Zugspitze ist mit 2962 Metern der höchste Berg Deutschlands.", "Der Rhein ist ein Fluss in Mitteleuropa."]
print(model.rank(query, passages))
ONNX
ONNX exports for CPU inference are in onnx/: onnx/model.onnx (fp32) and onnx/model_qint8_avx512_vnni.onnx (int8, dynamic quantization). Install with pip install "sentence-transformers[onnx]".
from sentence_transformers import CrossEncoder
model = CrossEncoder("syntriq/lupe-multilingual-reranker-small", backend="onnx", model_kwargs={"file_name": "onnx/model_qint8_avx512_vnni.onnx", "provider": "CPUExecutionProvider"})
query = "Wie hoch ist die Zugspitze?"
passages = ["Die Zugspitze ist mit 2962 Metern der höchste Berg Deutschlands.", "Der Rhein ist ein Fluss in Mitteleuropa."]
print(model.rank(query, passages))
Use "file_name": "onnx/model.onnx" for fp32. Keep "provider": "CPUExecutionProvider" on macOS: the CoreML provider that ONNX Runtime picks there by default failed on this model in our test.
Agreement with the PyTorch model on CPU: 100 query-passage pairs (50 German, 50 English) and top-1 passage on 20 ranking lists of 16 passages, all from our validation split.
| Format | Max absolute score difference (scores 0 to 1) | Spearman | Top-1 agreement |
|---|---|---|---|
| ONNX fp32 | 2.1e-06 | 1.0000 | 100% |
| ONNX int8 | 0.136 | 0.9973 | 100% |
Speed on an Apple M5 CPU (default thread settings, batch 32, 512 tokens per pair):
| Backend | Pairs per second |
|---|---|
| PyTorch fp32 | 19.2 |
| ONNX fp32 | 12.9 |
| ONNX int8 | 14.2 |
On this ARM CPU, ONNX is not faster than PyTorch. The int8 file is quantized for x86 CPUs with AVX-512 VNNI; we did not measure its speed there.
Results
nDCG@10. Every model ran through the same evaluation harness on the same queries: up to 200 queries per set, the first 200 of each MIRACL language and of GermanQuAD and GermanDPR (the MIRACL sets for Korean, Telugu and Yoruba have 186, 74 and 89). MIRACL reranks the candidates of the MTEB MIRACLReranking task (dev split) in 18 languages, GermanQuAD and GermanDPR rerank the top 100 passages of intfloat/multilingual-e5-base, NanoBEIR is the English sentence-transformers CrossEncoderNanoBEIREvaluator (13 sets, top 100 BM25 candidates).
| Model | MIRACL mean (18 languages) | MIRACL-de | MIRACL-en | NanoBEIR | GermanQuAD | GermanDPR | Parameters |
|---|---|---|---|---|---|---|---|
| syntriq/lupe-multilingual-reranker-small | 0.787 | 0.739 | 0.714 | 0.707 | 0.991 | 0.858 | 141M |
| Horizon-Labs/multilingual-reranker-small | 0.753 | 0.712 | 0.691 | 0.669 | 0.986 | 0.843 | 141M |
| cross-encoder/mmarco-mMiniLMv2-L12-H384-v1 | 0.736 | 0.701 | 0.656 | 0.638 | 0.925 | 0.804 | 118M |
| mixedbread-ai/mxbai-rerank-xsmall-v1 | 0.511 | 0.547 | 0.712 | 0.640 | 0.907 | 0.743 | 71M |
| jinaai/jina-reranker-v1-turbo-en | 0.354 | 0.307 | 0.613 | 0.653 | 0.768 | 0.624 | 38M |
| cross-encoder/ms-marco-MiniLM-L12-v2 | 0.307 | 0.302 | 0.660 | 0.665 | 0.844 | 0.592 | 33M |
This model scores highest on every set. On MIRACL-en its margin over mixedbread-ai/mxbai-rerank-xsmall-v1 (0.714 vs 0.712) is too small to call.
Full MIRACL dev sets, all queries:
| Model | MIRACL-de (221 queries) | MIRACL-en (705 queries) |
|---|---|---|
| syntriq/lupe-multilingual-reranker-small | 0.747 | 0.726 |
The competitors in this class were not run on the full sets. On the full German sets this model scores 0.989 on GermanQuAD (2,204 queries) and 0.870 on GermanDPR (1,025 queries).
Caveats: 200 queries per set and no significance test, so differences of a few thousandths are not meaningful. Checkpoints were selected on our own validation split, and the choice between training stages and averaged variants also took these public sets into account. The English training data contains training splits of datasets whose test splits are part of NanoBEIR (for example NQ, HotpotQA, FEVER, FiQA).
Intended use and limitations
- Reranking only: a cross-encoder that scores query-passage pairs. Retrieve 50 to 200 candidates with an embedding model or BM25 first, then rerank them with this model. It does not produce embeddings.
- Trained with a maximum input length of 512 tokens for query plus passage. Longer inputs are accepted by the architecture but were not trained or tested; split long documents into passages.
- Trained on 36 languages, evaluated on the 18 MIRACL languages plus the German (GermanQuAD, GermanDPR) and English (NanoBEIR) sets. Languages outside MIRACL (for example Dutch, Polish or Turkish) were not evaluated.
- The teacher is weaker in some languages: in the reviewed disagreements the judge sided against the teacher most often for Yoruba and Swahili, so quality there is less certain.
- Scores are meant for ranking within one query. They are not calibrated probabilities; check any fixed threshold on your own data.
- Results come from 200-query evaluations without a significance test.
Training data
Query-passage pairs in 36 languages: ar, bg, bn, ca, cs, da, de, el, en, es, fa, fi, fr, he, hi, hu, id, it, ja, ko, nl, no, pl, pt, ro, ru, sk, sv, sw, te, th, tr, uk, vi, yo, zh. Every pair carries a teacher score.
| Source | Used for | License |
|---|---|---|
| MIRACL train (sentence-transformers/miracl, from miracl/miracl) | Real queries with passages, 16 languages | Apache-2.0 |
| Wikipedia (wikimedia/wikipedia, 2023-11-01) | Passages | CC BY-SA 3.0 and GFDL |
| FineWeb-2 (HuggingFaceFW/fineweb-2) | Web passages | ODC-By 1.0, subject to the Common Crawl terms of use |
| FinePDFs (HuggingFaceFW/finepdfs), German part | German passages | ODC-By 1.0, subject to the Common Crawl terms of use |
| WebFAQ (PaDaS-Lab/webfaq-retrieval) | Real FAQ questions and answers | CC BY 4.0 |
| English pairs (cross-encoder/ettin-reranker-v1-data, MS MARCO parts removed) | 7,000,000 English pairs | Apache-2.0 for the collection; the upstream sources keep their own terms |
| Synthetic queries | About 728,000 multilingual and about 198,000 German queries (a question, a keyword query and an English question per passage), written by an open-weight large language model released under the MIT license | Generated by Syntriq |
| Synthetic German business documents | German passages from the synthetic training documents of the Syntriq Tarn PII models | Generated by Syntriq |
- Multilingual part (35 languages): 717K passages and 253K real queries (WebFAQ questions, MIRACL train questions) plus the synthetic queries. Each query has its positive passage and up to seven hard negatives mined with Snowflake/snowflake-arctic-embed-l-v2.0 (Apache-2.0).
- German part: the training data of syntriq/lupe-german-reranker-small, 4.0M pairs.
- Leakage control: Wikipedia articles and MIRACL train passages whose title starts a relevant passage of a MIRACL dev query were left out; German passages that share text with GermanQuAD, GermanDPR or relevant MIRACL-de passages were removed.
- No real personal data was used on purpose. Public web, FAQ and forum text in the sources can contain incidental names; the German business documents are synthetic.
- Data safety: 40 passages from the public sources (FineWeb-2, WebFAQ, MIRACL train and Wikipedia) matched spam patterns for abusive content. They were found and removed from the training data of the final training stage. The earlier training stage of the small and base models ran before this filter existed and included them; the large model was trained only on the filtered data.
Training
Distilled from a larger open reranker teacher (Apache-2.0). For every query-passage pair the teacher's score, the logit of "yes" minus the logit of "no", is the regression target, and the student learns it with mean squared error on single pairs (sentence-transformers MSELoss). One pass over the chosen pairs, maximum length 512 tokens, bf16, warmup 3 %, weight decay 0.01, seed 0; five validation checks per run on 2,160 held-out queries in 36 languages, the best checkpoint kept. In the first stage, most English labels were the scores shipped with the English data set, mapped onto the teacher's scale with a monotone fit; in the final stage all 7,000,000 English pairs carry teacher scores.
Label corrections: about 15,000 cases where the teacher ranked a mined negative above the labeled positive were reviewed by a large-language-model judge, and its verdicts corrected the labels before the final stage: 7,271 positives were raised above their negatives, 5,718 negatives judged relevant were raised to the positive's score, and in 1,984 cases the teacher's score was kept. The first stage used 1,745 such verdicts.
Lineage: jhu-clsp/mmBERT-small → Horizon-Labs/multilingual-reranker-small → syntriq/lupe-german-reranker-small (German fine-tune) → two multilingual stages:
- Stage 1: 12,000,000 pairs, learning rate 2e-5, 2,048 pairs per step (512 on each of 4 GPUs), 49.1 minutes, started from syntriq/lupe-german-reranker-small. Data set of 13,974,199 pairs (3.0M multilingual, 4.0M German, 7.0M English).
- Stage 2: 18,582,780 pairs, learning rate 1e-5, 2,048 pairs per step (512 on each of 4 GPUs), 75.1 minutes, started from stage 1. Data set of 18,582,780 pairs (11,582,780 multilingual and German, 7,000,000 English).
The released weights are the element-wise average of stage 1 and stage 2 (0.5 each), with no further training. Stage 2 alone gained English (MIRACL-en 0.706 → 0.718, NanoBEIR 0.690 → 0.700) but lost German (MIRACL-de 0.742 → 0.736, GermanQuAD 0.991 → 0.987); the average scores 0.739 on MIRACL-de, 0.991 on GermanQuAD and 0.707 on NanoBEIR.
Hardware: one node with 8× NVIDIA B200 GPUs; each run used several of them in parallel (distributed data parallel).
License and citation
Apache-2.0. Base models: jhu-clsp/mmBERT-small (MIT), Horizon-Labs/multilingual-reranker-small (Apache-2.0), syntriq/lupe-german-reranker-small (Apache-2.0). Teacher: Apache-2.0. Data licenses as listed above.
@misc{syntriq2026lupemultilingualsmall,
title = {Lupe Multilingual Reranker Small},
author = {Syntriq Labs},
year = {2026},
url = {https://huggingface.co/syntriq/lupe-multilingual-reranker-small}
}
- Downloads last month
- 36
Model tree for syntriq/lupe-multilingual-reranker-small
Base model
jhu-clsp/mmBERT-small