Lupe Multilingual Reranker Large

Deutsch: Lupe Multilingual Reranker Large sortiert Suchtreffer nach Relevanz, auf Deutsch, Englisch und 34 weiteren Sprachen, etwa als zweiter Schritt in Suche und RAG. In unserer Auswertung führt es auf MIRACL im Mittel, auf Deutsch und auf Englisch; auf NanoBEIR und GermanDPR liegt das fast dreimal so große mxbai-rerank-large-v2 vorn. Weitertrainiert aus bge-reranker-v2-m3 durch Destillation aus einem größeren offenen Reranker, ohne echte personenbezogene Daten.

A 568M-parameter multilingual cross-encoder for reranking search results in German, English and 34 more languages, built on BAAI/bge-reranker-v2-m3. In our evaluation it improves on bge-reranker-v2-m3 on every set and leads on MIRACL.

Quickstart

from sentence_transformers import CrossEncoder

model = CrossEncoder("syntriq/lupe-multilingual-reranker-large")
query = "Wie hoch ist die Zugspitze?"
passages = ["Die Zugspitze ist mit 2962 Metern der höchste Berg Deutschlands.", "Der Rhein ist ein Fluss in Mitteleuropa."]
print(model.rank(query, passages))

ONNX

ONNX exports for CPU inference are in onnx/: onnx/model.onnx with its weights in onnx/model.onnx_data (fp32) and onnx/model_qint8_avx512_vnni.onnx (int8, dynamic quantization). Install with pip install "sentence-transformers[onnx]".

from sentence_transformers import CrossEncoder

model = CrossEncoder("syntriq/lupe-multilingual-reranker-large", backend="onnx", model_kwargs={"file_name": "onnx/model_qint8_avx512_vnni.onnx", "provider": "CPUExecutionProvider"})
query = "Wie hoch ist die Zugspitze?"
passages = ["Die Zugspitze ist mit 2962 Metern der höchste Berg Deutschlands.", "Der Rhein ist ein Fluss in Mitteleuropa."]
print(model.rank(query, passages))

Use "file_name": "onnx/model.onnx" for fp32. "provider": "CPUExecutionProvider" keeps ONNX Runtime on the CPU on every platform.

Agreement with the PyTorch model on CPU: 100 query-passage pairs (50 German, 50 English) and top-1 passage on 20 ranking lists of 16 passages, all from our validation split.

Format Max absolute score difference (scores 0 to 1) Spearman Top-1 agreement
ONNX fp32 5.7e-06 1.0000 100%
ONNX int8 0.117 0.9947 100%

Speed on an Apple M5 CPU (default thread settings, batch 32, 512 tokens per pair):

Backend Pairs per second
PyTorch fp32 4.3
ONNX fp32 3.2
ONNX int8 4.3

On this ARM CPU, ONNX is not faster than PyTorch. The int8 file is quantized for x86 CPUs with AVX-512 VNNI; we did not measure its speed there.

Results

nDCG@10. Every model ran through the same evaluation harness on the same queries: up to 200 queries per set, the first 200 of each MIRACL language and of GermanQuAD and GermanDPR (the MIRACL sets for Korean, Telugu and Yoruba have 186, 74 and 89). MIRACL reranks the candidates of the MTEB MIRACLReranking task (dev split) in 18 languages, GermanQuAD and GermanDPR rerank the top 100 passages of intfloat/multilingual-e5-base, NanoBEIR is the English sentence-transformers CrossEncoderNanoBEIREvaluator (13 sets, top 100 BM25 candidates).

Model MIRACL mean (18 languages) MIRACL-de MIRACL-en NanoBEIR GermanQuAD GermanDPR Parameters
syntriq/lupe-multilingual-reranker-large 0.829 0.794 0.767 0.712 0.990 0.868 568M
BAAI/bge-reranker-v2-m3 0.821 0.774 0.748 0.694 0.989 0.863 568M
Qwen/Qwen3-Reranker-0.6B 0.790 0.766 0.709 not measured 0.990 0.852 596M
mixedbread-ai/mxbai-rerank-large-v2 not measured 0.702¹ 0.730¹ 0.731 0.990 0.876 1.54B

mixedbread-ai/mxbai-rerank-large-v2, almost three times the size, wins NanoBEIR (0.731 vs 0.712) and GermanDPR (0.876 vs 0.868). GermanQuAD is level between this model, Qwen/Qwen3-Reranker-0.6B and mxbai-rerank-large-v2. This model leads on MIRACL-de, MIRACL-en and, among the models where it was measured, the MIRACL mean.

¹ MIRACL-de and MIRACL-en of the mxbai v2 models come from the evaluation log of the same run: the process that held these languages ran out of GPU memory before it wrote its result file. Their MIRACL mean is not measured because Hindi did not finish.

Full MIRACL dev sets, all queries:

Model MIRACL-de (221 queries) MIRACL-en (705 queries)
syntriq/lupe-multilingual-reranker-large not measured not measured
BAAI/bge-reranker-v2-m3 0.780 0.748

This model was not run on the full sets; bge-reranker-v2-m3 is shown for reference.

Caveats: 200 queries per set and no significance test, so differences of a few thousandths are not meaningful. Checkpoints were selected on our own validation split, and the choice between training stages and averaged variants also took these public sets into account. The English training data contains training splits of datasets whose test splits are part of NanoBEIR (for example NQ, HotpotQA, FEVER, FiQA).

Intended use and limitations

  • Reranking only: a cross-encoder that scores query-passage pairs. Retrieve 50 to 200 candidates with an embedding model or BM25 first, then rerank them with this model. It does not produce embeddings.
  • Trained with a maximum input length of 512 tokens for query plus passage. Longer inputs are accepted by the architecture but were not trained or tested; split long documents into passages.
  • Trained on 36 languages, evaluated on the 18 MIRACL languages plus the German (GermanQuAD, GermanDPR) and English (NanoBEIR) sets. Languages outside MIRACL (for example Dutch, Polish or Turkish) were not evaluated.
  • The teacher is weaker in some languages: in the reviewed disagreements the judge sided against the teacher most often for Yoruba and Swahili, so quality there is less certain.
  • Scores are meant for ranking within one query. They are not calibrated probabilities; check any fixed threshold on your own data.
  • Results come from 200-query evaluations without a significance test.

Training data

Query-passage pairs in 36 languages: ar, bg, bn, ca, cs, da, de, el, en, es, fa, fi, fr, he, hi, hu, id, it, ja, ko, nl, no, pl, pt, ro, ru, sk, sv, sw, te, th, tr, uk, vi, yo, zh. Every pair carries a teacher score.

Source Used for License
MIRACL train (sentence-transformers/miracl, from miracl/miracl) Real queries with passages, 16 languages Apache-2.0
Wikipedia (wikimedia/wikipedia, 2023-11-01) Passages CC BY-SA 3.0 and GFDL
FineWeb-2 (HuggingFaceFW/fineweb-2) Web passages ODC-By 1.0, subject to the Common Crawl terms of use
FinePDFs (HuggingFaceFW/finepdfs), German part German passages ODC-By 1.0, subject to the Common Crawl terms of use
WebFAQ (PaDaS-Lab/webfaq-retrieval) Real FAQ questions and answers CC BY 4.0
English pairs (cross-encoder/ettin-reranker-v1-data, MS MARCO parts removed) 7,000,000 English pairs Apache-2.0 for the collection; the upstream sources keep their own terms
Synthetic queries About 728,000 multilingual and about 198,000 German queries (a question, a keyword query and an English question per passage), written by an open-weight large language model released under the MIT license Generated by Syntriq
Synthetic German business documents German passages from the synthetic training documents of the Syntriq Tarn PII models Generated by Syntriq
  • Multilingual part (35 languages): 717K passages and 253K real queries (WebFAQ questions, MIRACL train questions) plus the synthetic queries. Each query has its positive passage and up to seven hard negatives mined with Snowflake/snowflake-arctic-embed-l-v2.0 (Apache-2.0).
  • German part: the training data of syntriq/lupe-german-reranker-small, 4.0M pairs.
  • Leakage control: Wikipedia articles and MIRACL train passages whose title starts a relevant passage of a MIRACL dev query were left out; German passages that share text with GermanQuAD, GermanDPR or relevant MIRACL-de passages were removed.
  • No real personal data was used on purpose. Public web, FAQ and forum text in the sources can contain incidental names; the German business documents are synthetic.
  • Data safety: 40 passages from the public sources (FineWeb-2, WebFAQ, MIRACL train and Wikipedia) matched spam patterns for abusive content. They were found and removed from the training data of the final training stage. The earlier training stage of the small and base models ran before this filter existed and included them; the large model was trained only on the filtered data.

Training

Distilled from a larger open reranker teacher (Apache-2.0). For every query-passage pair the teacher's score, the logit of "yes" minus the logit of "no", is the regression target, and the student learns it with mean squared error on single pairs (sentence-transformers MSELoss). One pass over the chosen pairs, maximum length 512 tokens, bf16, warmup 3 %, weight decay 0.01, seed 0; five validation checks per run on 2,160 held-out queries in 36 languages, the best checkpoint kept. All 7,000,000 English pairs carry teacher scores.

Label corrections: about 15,000 cases where the teacher ranked a mined negative above the labeled positive were reviewed by a large-language-model judge, and its verdicts corrected the labels before the final stage: 7,271 positives were raised above their negatives, 5,718 negatives judged relevant were raised to the positive's score, and in 1,984 cases the teacher's score was kept.

Lineage: BAAI/bge-reranker-v2-m3 → one multilingual stage: 11,624,580 pairs, learning rate 5e-6, 1,152 pairs per step (192 on each of 6 GPUs), 96.2 minutes, started from BAAI/bge-reranker-v2-m3, on the first part of the shuffled 18,582,780-pair data set (11,582,780 multilingual and German, 7,000,000 English).

The released weights are the element-wise average of the fine-tuned model and the original BAAI/bge-reranker-v2-m3 (0.5 each), with no further training. The fine-tuned model alone was slightly below bge-reranker-v2-m3 on the MIRACL mean (0.819 vs 0.821) and level on MIRACL-en (0.748 vs 0.748); the average is above both on the MIRACL mean (0.829), MIRACL-de (0.794) and MIRACL-en (0.767). A 0.7/0.3 average was also measured and not chosen.

Hardware: one node with 8× NVIDIA B200 GPUs; each run used several of them in parallel (distributed data parallel).

License and citation

Apache-2.0. Base models: BAAI/bge-reranker-v2-m3 (Apache-2.0). Teacher: Apache-2.0. Data licenses as listed above.

@misc{syntriq2026lupemultilinguallarge,
  title  = {Lupe Multilingual Reranker Large},
  author = {Syntriq Labs},
  year   = {2026},
  url    = {https://huggingface.co/syntriq/lupe-multilingual-reranker-large}
}
Downloads last month
17
Safetensors
Model size
0.6B params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for syntriq/lupe-multilingual-reranker-large

Quantized
(76)
this model

Datasets used to train syntriq/lupe-multilingual-reranker-large