---
license: apache-2.0
language:
- en
base_model: cross-encoder/ms-marco-MiniLM-L-6-v2
pipeline_tag: text-ranking
library_name: fastembed
tags:
- cross-encoder
- reranker
- onnx
- int8
- distillation
---

# Context reranker (MiniLM-L6, int8 ONNX)

A small cross-encoder that scores how well a passage answers a question. In a
retrieval-augmented system it orders the parts of the retrieved documents the
answer model reads, so the passage with the answer gets into a limited context
budget first.

- **Base model:** [cross-encoder/ms-marco-MiniLM-L-6-v2](https://huggingface.co/cross-encoder/ms-marco-MiniLM-L-6-v2) (22M parameters)
- **Teacher:** [BAAI/bge-reranker-v2-m3](https://huggingface.co/BAAI/bge-reranker-v2-m3) — strong, but seconds per question on CPU
- **Training:** listwise distillation of the teacher's scores (KL divergence), 2 epochs, on about 9,000
  English questions that a locally hosted LLM wrote from 3,000 passages of a public-finance document
  corpus. For each question, the teacher scored the passages of the top-10 sections the retrieval
  system returned for it. No evaluation questions were used for training.
- **Format:** ONNX (exported with classic attention, opset 14), dynamic int8 quantization, 23 MB.
  About 65 ms to score 16 passages on a desktop CPU.

## Results

| Test | Base MiniLM | This model | Teacher |
|---|---|---|---|
| In-domain: answer passage inside an 18,000-character context (475 questions; 396 with no reranker; this model scored by copies that never saw the question's document) | 398 | **409** | 415 |
| SciFact, 100 questions, BM25 top-30 reranked (nDCG@10) | 0.723 | **0.757** | 0.793 |
| 20 everyday questions, answer ranked first of 4 | 17/20 | **17/20** | 18/20 |

## Usage

With [fastembed](https://github.com/qdrant/fastembed):

```python
from huggingface_hub import snapshot_download
from fastembed.rerank.cross_encoder import TextCrossEncoder
from fastembed.common.model_description import ModelSource

path = snapshot_download("Badawi-a/context-reranker", revision="v1")
TextCrossEncoder.add_custom_model(model="local/context-reranker", sources=ModelSource(hf="local/context-reranker"),
                                  model_file="onnx/model.onnx")
reranker = TextCrossEncoder("local/context-reranker", specific_model_path=path)
scores = list(reranker.rerank("What is the VAT rate?", ["The VAT rate is 14 percent.", "It rained in Cairo."], batch_size=8))
```

Higher scores mean more relevant.

## Limitations

- English only. Questions in other languages should be translated first.
- Trained for one domain (public finance); it generalizes (see SciFact), but less well than the teacher.
- Like other small rerankers, it can prefer a passage that repeats the question's words over one that answers it.