cbx-rag-context-relevance-68M
Judges which retrieved context sentences are relevant to the question. Give it the question and the passages. It returns:
- a relevance score and flag for every context sentence (it reads each sentence with two neighbours on each side);
- the relevance fraction: the share of context sentences that are relevant;
- context precision, when there are two or more passages: are the relevant passages ranked first?
It is one of four CBX RAG evaluator models.
How to use
pip install -U huggingface_hub
hf download CobuildX-AI/cbx-rag-context-relevance-68M --revision v0.1 --local-dir cbx-rag-context-relevance-68M
cd cbx-rag-context-relevance-68M
pip install -r requirements.txt
python example.py
from cbx_rag_eval import ContextRelevance
judge = ContextRelevance(".")
out = judge(question, passages) # passages in retrieval order
out["relevance"], out.get("context_precision")
[s for s in out["sentences"] if s["relevant"]]
Outputs:
relevance: the answer-level fraction, with the same convention as RAGBench (relevant sentences / all sentences).context_precision: average precision over the passages in the order given. A passage counts as relevant when any of its sentences is flagged.
On long single documents (contract profile), use the per-sentence scores. There the answer-level fraction is
close to a constant: few sentences are relevant to any one question, so a fixed guess is hard to beat.
Speed (seconds per call, model loaded; GPU NVIDIA TITAN RTX fp16, CPU 8 threads fp32):
| Input | GPU | CPU |
|---|---|---|
| one contract (about 414 sentences) | 1.02 s | 43.87 s |
| several short passages (about 29 sentences) | 0.10 s | 1.71 s |
The four CBX RAG evaluator models
| Model | Judges | Output |
|---|---|---|
| cbx-rag-faithfulness-68M | each answer sentence against the passages | support score per sentence, answer score, hallucination flag |
| cbx-rag-response-relevance-68M | the answer against the question | direct / partial / irrelevant |
| cbx-rag-context-relevance-68M | each context sentence against the question | relevance per sentence, relevance fraction, context precision |
| cbx-rag-context-utilization-68M | each context sentence against the question and answer | use per sentence, utilization fraction |
All four share the wrapper cbx_rag_eval.py. Every model is fine-tuned from
jhu-clsp/ettin-encoder-68m (68M parameters). Current version:
v0.1 (use --revision v0.1 to pin it).
Sentences and profiles
Sentences: the models judge sentences, and they were trained on RAGBench's sentence splits. The wrapper splits raw text by profile (
split_sentences; chosen on validation):contract: line breaks, then NLTK punkt;financial: blank lines, then pysbd.
On validation, the wrapper's splitting instead of RAGBench's own sentence splits changed sentence AUROC 0.942 β 0.920 (CUAD); sentence AUROC 0.965 β 0.962 (FinQA); answer RMSE 0.034 β 0.034 (CUAD); answer RMSE 0.066 β 0.070 (FinQA). For the best accuracy, pass your own sentences (a list of strings per passage, and
answer_sentences=for the answer); the splitter is then skipped.Profiles: the settings were fixed on two kinds of validation data, so every call takes a
profile."contract": one document, such as a contract (CUAD-like)."financial": several short passages with numbers and tables (FinQA-like). Faithfulness adds a numeric check here."auto"(default):"contract"for a single passage,"financial"for two or more.
For other kinds of text, choose the profile whose shape is closest and check the results on your own data.
Training data and labels
- Source: the CUAD (contracts) and FinQA (financial reports) subsets of
RAGBench (CC BY 4.0). Its answers were written by
claude-3-haiku and gpt-3.5-turbo; its labels came from gpt-4-turbo.
- Train: 1,530 CUAD and 12,502 FinQA answers.
- Test: 510 CUAD (102 contracts) and 2,294 FinQA answers.
- Labels were reviewed by LLMs, not humans:
- Faithfulness: majority vote of RAGBench and two LLM reviewers: Claude Haiku and gpt-5.4-mini, or two Haiku passes for CUAD train.
- CUAD context relevance / utilization: majority vote of RAGBench and two Claude Haiku passes.
- FinQA context labels: kept as RAGBench made them.
- Audits: samples were checked blind by Claude Opus and Claude Sonnet.
- Response relevance: uses its own set (rr-v0.1). Claude Haiku wrote varied answers to the RAGBench questions and labelled them.
- Test sets: the RAGBench test splits, with the same label views.
- Baselines: RAGAS, GPT-3.5 and TruLens scores are the ones stored in RAGBench, not re-runs.
- Rule: every threshold and setting was fixed on validation.
Licence
Apache-2.0 for the model weights and code.
Data attribution:
- RAGBench (Friel et al., 2024), CC BY 4.0;
- CUAD (The Atticus Project), CC BY 4.0;
- FinQA (Chen et al., 2021), MIT.
Labels were produced with Claude (Anthropic) and gpt-5.4-mini (OpenAI) models.
Disclaimer: the scores are statistical estimates for evaluating RAG systems. They are not a guarantee that an answer is correct, complete or safe. Do not use them as the sole basis for legal, financial or other consequential decisions.
Results (test sets)
- Labels: CUAD uses reviewed labels: RAGBench's long clause-block labels were rejected by audit in most cases and replaced by the majority of RAGBench and two Claude Haiku passes. FinQA uses RAGBench labels.
- Constant: predicts the validation average.
Sentence level:
| Test set | AUROC [95% CI] | PR-AUC [95% CI] | Precision / recall at the threshold | Positives |
|---|---|---|---|---|
| CUAD contracts | 0.933 [0.920, 0.945] | 0.372 [0.321, 0.421] | 0.40 / 0.42 | 1.5% of 104,725 sentences |
| FinQA reports | 0.977 [0.972, 0.982] | 0.858 [0.839, 0.877] | 0.81 / 0.79 | 6.1% of 53,640 sentences |
Answer-level relevance fraction: RMSE, lower is better. Baselines are RAGBench's stored scores, on the answers they cover.
| Test set | This model | RAGAS | GPT-3.5 | TruLens | Constant |
|---|---|---|---|---|---|
| CUAD contracts | 0.047 | 0.057 | 0.118 | 0.666 | 0.050 |
| FinQA reports | 0.054 | 0.106 | 0.103 | 0.793 | 0.081 |
On CUAD the model beats RAGAS and GPT-3.5 significantly, but not the constant (0.044 vs 0.047 over all answers, a difference within noise).
Context precision (FinQA, 2,281 answers with a relevant passage): RMSE 0.164 [0.143, 0.184], against 0.209 for a constant. RAGBench stores no context-precision baselines. It is not available for single-document contexts.
Limitations
- CUAD sentence precision and recall are only about 0.4. A blind Claude Sonnet audit found that about 3 in 4 of the model's misses, and 1 in 4 of its false alarms, were label errors. So the true quality is probably higher than measured, but this is not quantified.
- It disagrees with RAGBench's CUAD labels on purpose. On those block labels its AUROC is 0.72.
- Not reported: evidence completeness (relevant sentences the answer used). Combined with cbx-rag-context-utilization it did not beat a constant.
- Cost: every context sentence is scored, about 0.5 s per 280-sentence contract on a GPU.
- Coverage: two domains (contracts, financial reports), English, LLM-made labels.
- Downloads last month
- 22
Model tree for CobuildX-AI/cbx-rag-context-relevance-68M
Base model
jhu-clsp/ettin-encoder-68m