context-reranker / README.md
Badawi-a's picture
Context reranker v1: MiniLM-L6 distilled from bge-reranker-v2-m3, int8 ONNX
a7b27c9 verified
|
Raw History Blame Contribute Delete
2.78 kB
---
license: apache-2.0
language:
- en
base_model: cross-encoder/ms-marco-MiniLM-L-6-v2
pipeline_tag: text-ranking
library_name: fastembed
tags:
- cross-encoder
- reranker
- onnx
- int8
- distillation
---
# Context reranker (MiniLM-L6, int8 ONNX)
A small cross-encoder that scores how well a passage answers a question. In a
retrieval-augmented system it orders the parts of the retrieved documents the
answer model reads, so the passage with the answer gets into a limited context
budget first.
- **Base model:** [cross-encoder/ms-marco-MiniLM-L-6-v2](https://huggingface.co/cross-encoder/ms-marco-MiniLM-L-6-v2) (22M parameters)
- **Teacher:** [BAAI/bge-reranker-v2-m3](https://huggingface.co/BAAI/bge-reranker-v2-m3) — strong, but seconds per question on CPU
- **Training:** listwise distillation of the teacher's scores (KL divergence), 2 epochs, on about 9,000
English questions that a locally hosted LLM wrote from 3,000 passages of a public-finance document
corpus. For each question, the teacher scored the passages of the top-10 sections the retrieval
system returned for it. No evaluation questions were used for training.
- **Format:** ONNX (exported with classic attention, opset 14), dynamic int8 quantization, 23 MB.
About 65 ms to score 16 passages on a desktop CPU.
## Results
| Test | Base MiniLM | This model | Teacher |
|---|---|---|---|
| In-domain: answer passage inside an 18,000-character context (475 questions; 396 with no reranker; this model scored by copies that never saw the question's document) | 398 | **409** | 415 |
| SciFact, 100 questions, BM25 top-30 reranked (nDCG@10) | 0.723 | **0.757** | 0.793 |
| 20 everyday questions, answer ranked first of 4 | 17/20 | **17/20** | 18/20 |
## Usage
With [fastembed](https://github.com/qdrant/fastembed):
```python
from huggingface_hub import snapshot_download
from fastembed.rerank.cross_encoder import TextCrossEncoder
from fastembed.common.model_description import ModelSource
path = snapshot_download("Badawi-a/context-reranker", revision="v1")
TextCrossEncoder.add_custom_model(model="local/context-reranker", sources=ModelSource(hf="local/context-reranker"),
model_file="onnx/model.onnx")
reranker = TextCrossEncoder("local/context-reranker", specific_model_path=path)
scores = list(reranker.rerank("What is the VAT rate?", ["The VAT rate is 14 percent.", "It rained in Cairo."], batch_size=8))
```
Higher scores mean more relevant.
## Limitations
- English only. Questions in other languages should be translated first.
- Trained for one domain (public finance); it generalizes (see SciFact), but less well than the teacher.
- Like other small rerankers, it can prefer a passage that repeats the question's words over one that answers it.