--- license: apache-2.0 base_model: jhu-clsp/ettin-encoder-68m pipeline_tag: text-classification language: - en tags: - rag - rag-evaluation - context-utilization --- # cbx-rag-context-utilization-68M Judges which **context sentences the answer actually used**. Give it the question, the passages and the answer. It returns: - a score and flag for every context sentence; - the **utilization fraction**: the share of context sentences the answer used, with the RAGBench convention. It is one of four CBX RAG evaluator models. ## How to use ```bash pip install -U huggingface_hub hf download CobuildX-AI/cbx-rag-context-utilization-68M --revision v0.1 --local-dir cbx-rag-context-utilization-68M cd cbx-rag-context-utilization-68M pip install -r requirements.txt python example.py ``` ```python from cbx_rag_eval import ContextUtilization judge = ContextUtilization(".") out = judge(question, passages, answer) out["utilization"] [s for s in out["sentences"] if s["utilized"]] ``` **How the fraction is computed:** each sentence probability is calibrated (isotonic, fitted on validation), and the fraction is their average. **Prefilter (`contract` profile):** only the 64 sentences closest to the question and answer (BM25), plus their neighbours, are scored; the rest count as unused. On validation this kept 98.8% of used sentences while scoring about 39% of all sentences, with the same RMSE. Pass `prefilter=False` to score everything. The test numbers below are without the prefilter. **Speed** (seconds per call, model loaded; GPU NVIDIA TITAN RTX fp16, CPU 8 threads fp32): | Input | GPU | CPU | |---|---|---| | one contract (about 414 sentences) | 0.61 s | 23.43 s | | several short passages (about 29 sentences) | 0.12 s | 3.04 s | ## The four CBX RAG evaluator models | Model | Judges | Output | |---|---|---| | [cbx-rag-faithfulness-68M](https://huggingface.co/CobuildX-AI/cbx-rag-faithfulness-68M) | each answer sentence against the passages | support score per sentence, answer score, hallucination flag | | [cbx-rag-response-relevance-68M](https://huggingface.co/CobuildX-AI/cbx-rag-response-relevance-68M) | the answer against the question | direct / partial / irrelevant | | [cbx-rag-context-relevance-68M](https://huggingface.co/CobuildX-AI/cbx-rag-context-relevance-68M) | each context sentence against the question | relevance per sentence, relevance fraction, context precision | | [cbx-rag-context-utilization-68M](https://huggingface.co/CobuildX-AI/cbx-rag-context-utilization-68M) | each context sentence against the question and answer | use per sentence, utilization fraction | All four share the wrapper `cbx_rag_eval.py`. Every model is fine-tuned from [jhu-clsp/ettin-encoder-68m](https://huggingface.co/jhu-clsp/ettin-encoder-68m) (68M parameters). Current version: **v0.1** (use `--revision v0.1` to pin it). ## Sentences and profiles - **Sentences:** the models judge sentences, and they were trained on RAGBench's sentence splits. The wrapper splits raw text by profile (`split_sentences`; chosen on validation): - `contract`: line breaks, then NLTK punkt; - `financial`: blank lines, then pysbd. On validation, the wrapper's splitting instead of RAGBench's own sentence splits changed sentence AUROC 0.990 → 0.981 (CUAD); sentence AUROC 0.996 → 0.995 (FinQA); answer RMSE 0.040 → 0.043 (CUAD); answer RMSE 0.036 → 0.040 (FinQA). For the best accuracy, pass your own sentences (a list of strings per passage, and `answer_sentences=` for the answer); the splitter is then skipped. - **Profiles:** the settings were fixed on two kinds of validation data, so every call takes a `profile`. - `"contract"`: one document, such as a contract (CUAD-like). - `"financial"`: several short passages with numbers and tables (FinQA-like). Faithfulness adds a numeric check here. - `"auto"` (default): `"contract"` for a single passage, `"financial"` for two or more. For other kinds of text, choose the profile whose shape is closest and check the results on your own data. ## Training data and labels - **Source:** the CUAD (contracts) and FinQA (financial reports) subsets of [RAGBench](https://huggingface.co/datasets/galileo-ai/ragbench) (CC BY 4.0). Its answers were written by claude-3-haiku and gpt-3.5-turbo; its labels came from gpt-4-turbo. - Train: 1,530 CUAD and 12,502 FinQA answers. - Test: 510 CUAD (102 contracts) and 2,294 FinQA answers. - **Labels were reviewed by LLMs, not humans:** - **Faithfulness:** majority vote of RAGBench and two LLM reviewers: Claude Haiku and gpt-5.4-mini, or two Haiku passes for CUAD train. - **CUAD context relevance / utilization:** majority vote of RAGBench and two Claude Haiku passes. - **FinQA context labels:** kept as RAGBench made them. - **Audits:** samples were checked blind by Claude Opus and Claude Sonnet. - **Response relevance:** uses its own set (rr-v0.1). Claude Haiku wrote varied answers to the RAGBench questions and labelled them. - **Test sets:** the RAGBench test splits, with the same label views. - **Baselines:** RAGAS, GPT-3.5 and TruLens scores are the ones stored in RAGBench, not re-runs. - **Rule:** every threshold and setting was fixed on validation. ## Licence Apache-2.0 for the model weights and code. **Data attribution:** - RAGBench (Friel et al., 2024), CC BY 4.0; - CUAD (The Atticus Project), CC BY 4.0; - FinQA (Chen et al., 2021), MIT. Labels were produced with Claude (Anthropic) and gpt-5.4-mini (OpenAI) models. **Disclaimer:** the scores are statistical estimates for evaluating RAG systems. They are not a guarantee that an answer is correct, complete or safe. Do not use them as the sole basis for legal, financial or other consequential decisions. ## Results (test sets) - **Labels:** CUAD uses reviewed labels (majority of RAGBench and two Claude Haiku passes); FinQA uses RAGBench labels. - **Constant:** predicts the validation average. **Sentence level:** | Test set | AUROC [95% CI] | PR-AUC [95% CI] | Precision / recall at the threshold | |---|---|---|---| | CUAD contracts | 0.981 [0.973, 0.987] | 0.737 [0.697, 0.774] | 0.78 / 0.60 | | FinQA reports | 0.995 [0.994, 0.996] | 0.945 [0.933, 0.956] | 0.90 / 0.88 | **Answer-level utilization fraction:** RMSE, lower is better. GPT-3.5 is RAGBench's stored score; RAGBench has no other utilization baseline. | Test set | This model | GPT-3.5 | Constant | |---|---|---|---| | CUAD contracts (415 answers with a baseline score) | **0.039** | 0.059 | 0.063 | | FinQA reports (2,294) | **0.033** | 0.134 | 0.067 | Both differences against GPT-3.5 and the constant are significant (context-clustered bootstrap). On RAGBench's own CUAD labels the model ties GPT-3.5 (0.108 vs 0.105). ## Limitations - **CUAD recall:** at the default threshold it misses about 40% of used sentences (recall 0.60). When it flags a sentence, it is usually right (precision 0.78). - **Label noise:** in a blind Claude Sonnet audit, 6 of 10 sentences the review added and 4 of 10 model misses were confirmed as used. - **Not reported:** evidence completeness (relevant sentences the answer used). Combined with cbx-rag-context-relevance it did not beat a constant. - **Coverage:** two domains (contracts, financial reports), English, LLM-made labels.