cbx-rag-faithfulness-68M
Checks whether each sentence of a RAG answer is supported by the retrieved passages. Give it the question, the passages and the answer. It returns:
- a support score and a flag for every answer sentence;
- an answer score;
- a hallucination flag, set when any sentence is flagged.
It is one of four CBX RAG evaluator models. See also cbx-hallucination-detector, which marks unsupported spans in any grounded answer. This model scores whole sentences against retrieved passages and was trained on contract and financial-report RAG data.
How to use
pip install -U huggingface_hub
hf download CobuildX-AI/cbx-rag-faithfulness-68M --revision v0.1 --local-dir cbx-rag-faithfulness-68M
cd cbx-rag-faithfulness-68M
pip install -r requirements.txt
python example.py
from cbx_rag_eval import Faithfulness
judge = Faithfulness(".") # GPU when available, else CPU
out = judge(question, passages, answer) # passages: a string or a list of strings
out["hallucination"], out["answer_score"]
for s in out["sentences"]:
print(s["flag"], s["score"], s["text"])
Arguments:
profile:"auto","contract"or"financial"(see below).operating_point:"default"or"high_recall".answer_sentences: pre-split answer sentences.
How it scores a sentence:
- Evidence: the passages, or for long documents the sentences closest to the question and the answer sentence (BM25, up to 2,048 tokens).
- Number check (
financialprofile only): every number in the answer sentence is checked against the passage numbers, directly or as one arithmetic step (difference, ratio, percent change, sum). A logistic combiner merges that check with the model's score. The verdict is reported asnumbers. - Flag: set when the score passes the profile threshold, which flags about 4.5% of validation sentences.
- Answer score: the highest sentence score.
Speed (seconds per call, model loaded; GPU NVIDIA TITAN RTX fp16, CPU 8 threads fp32):
| Input | GPU | CPU |
|---|---|---|
| one contract (about 414 sentences) | 0.73 s | 15.18 s |
| several short passages (about 29 sentences) | 0.13 s | 2.13 s |
The four CBX RAG evaluator models
| Model | Judges | Output |
|---|---|---|
| cbx-rag-faithfulness-68M | each answer sentence against the passages | support score per sentence, answer score, hallucination flag |
| cbx-rag-response-relevance-68M | the answer against the question | direct / partial / irrelevant |
| cbx-rag-context-relevance-68M | each context sentence against the question | relevance per sentence, relevance fraction, context precision |
| cbx-rag-context-utilization-68M | each context sentence against the question and answer | use per sentence, utilization fraction |
All four share the wrapper cbx_rag_eval.py. Every model is fine-tuned from
jhu-clsp/ettin-encoder-68m (68M parameters). Current version:
v0.1 (use --revision v0.1 to pin it).
Sentences and profiles
Sentences: the models judge sentences, and they were trained on RAGBench's sentence splits. The wrapper splits raw text by profile (
split_sentences; chosen on validation):contract: line breaks, then NLTK punkt;financial: blank lines, then pysbd.
On validation, the wrapper's splitting instead of RAGBench's own sentence splits changed whole-answer AUROC 0.797 β 0.797 (CUAD); whole-answer AUROC 0.744 β 0.739 (FinQA). For the best accuracy, pass your own sentences (a list of strings per passage, and
answer_sentences=for the answer); the splitter is then skipped.Profiles: the settings were fixed on two kinds of validation data, so every call takes a
profile."contract": one document, such as a contract (CUAD-like)."financial": several short passages with numbers and tables (FinQA-like). Faithfulness adds a numeric check here."auto"(default):"contract"for a single passage,"financial"for two or more.
For other kinds of text, choose the profile whose shape is closest and check the results on your own data.
Training data and labels
- Source: the CUAD (contracts) and FinQA (financial reports) subsets of
RAGBench (CC BY 4.0). Its answers were written by
claude-3-haiku and gpt-3.5-turbo; its labels came from gpt-4-turbo.
- Train: 1,530 CUAD and 12,502 FinQA answers.
- Test: 510 CUAD (102 contracts) and 2,294 FinQA answers.
- Labels were reviewed by LLMs, not humans:
- Faithfulness: majority vote of RAGBench and two LLM reviewers: Claude Haiku and gpt-5.4-mini, or two Haiku passes for CUAD train.
- CUAD context relevance / utilization: majority vote of RAGBench and two Claude Haiku passes.
- FinQA context labels: kept as RAGBench made them.
- Audits: samples were checked blind by Claude Opus and Claude Sonnet.
- Response relevance: uses its own set (rr-v0.1). Claude Haiku wrote varied answers to the RAGBench questions and labelled them.
- Test sets: the RAGBench test splits, with the same label views.
- Baselines: RAGAS, GPT-3.5 and TruLens scores are the ones stored in RAGBench, not re-runs.
- Rule: every threshold and setting was fixed on validation.
Licence
Apache-2.0 for the model weights and code.
Data attribution:
- RAGBench (Friel et al., 2024), CC BY 4.0;
- CUAD (The Atticus Project), CC BY 4.0;
- FinQA (Chen et al., 2021), MIT.
Labels were produced with Claude (Anthropic) and gpt-5.4-mini (OpenAI) models.
Disclaimer: the scores are statistical estimates for evaluating RAG systems. They are not a guarantee that an answer is correct, complete or safe. Do not use them as the sole basis for legal, financial or other consequential decisions.
Results (test sets, labels reviewed by LLMs)
Whole answer: does the answer contain an unsupported sentence? AUROC; RAGAS, GPT-3.5 and TruLens are RAGBench's stored scores (81β85% of CUAD answers).
| Test set (labels) | This model [95% CI] | RAGAS | GPT-3.5 | TruLens |
|---|---|---|---|---|
| CUAD contracts (reviewed) | 0.776 [0.71, 0.83] | 0.614 | 0.494 | 0.424 |
| FinQA reports (reviewed) | 0.735 [0.71, 0.76] | 0.532 | 0.526 | 0.553 |
| CUAD (RAGBench labels) | 0.724 [0.64, 0.80] | 0.660 | 0.514 | 0.399 |
| FinQA (RAGBench labels) | 0.576 [0.53, 0.62] | 0.574 | 0.567 | 0.528 |
Hallucination flag at the default operating point (whole answers, reviewed labels):
| Test set | Answers flagged | Precision | Recall | False-positive rate |
|---|---|---|---|---|
| CUAD (510 answers, 77 with an unsupported sentence) | 13.5% | 0.43 | 0.39 | 0.09 |
| FinQA (2,294 answers, 501) | 15.6% | 0.56 | 0.40 | 0.09 |
Against the baselines at the same flag rate. Each baseline flags as many answers as this model on the same answers. Precision / recall:
| Test set | This model | RAGAS | GPT-3.5 | TruLens |
|---|---|---|---|---|
| CUAD (β15% flagged) | 0.44 / 0.45 | 0.27 / 0.28 | 0.14 / 0.14 | 0.05 / 0.05 |
| FinQA (β16% flagged) | 0.56 / 0.40 | 0.30 / 0.21 | 0.27 / 0.19 | 0.24 / 0.17 |
All differences are significant (context-clustered bootstrap). On RAGBench's own FinQA labels, which rarely mark arithmetic errors, the model only ties the baselines.
Sentence level, default operating point (reviewed labels):
- CUAD: flags 3.9% of sentences, precision 0.38, recall 0.25.
- FinQA: flags 4.4%, precision 0.53, recall 0.33. Without the number check it would be 0.33 / 0.23.
Limitations
- Misses many hallucinations. At the default point it finds about 40% of the answers with an unsupported
sentence. On CUAD, about half of its flags are false alarms. Use
"high_recall"to catch more at the cost of more false alarms. It is a screening signal, not a verdict. - Right number, wrong context passes: a number copied correctly but attached to the wrong year or entity is still accepted. About 6% of sentences whose numbers all verify are unsupported.
- Two domains only: contracts and financial reports in English. Other domains, conversational answers and code were not tested.
- LLM labels: labels come from LLM review; no human annotation.
- Only the given passages count: a statement that is true but not in the passages is flagged.
- Downloads last month
- 22
Model tree for CobuildX-AI/cbx-rag-faithfulness-68M
Base model
jhu-clsp/ettin-encoder-68m