File size: 7,298 Bytes
ec76fb8
 
041950a
 
 
 
 
 
 
 
ec76fb8
041950a
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
---
license: apache-2.0
base_model: jhu-clsp/ettin-encoder-68m
pipeline_tag: text-classification
language:
  - en
tags:
  - rag
  - rag-evaluation
  - context-utilization
---

# cbx-rag-context-utilization-68M

Judges which **context sentences the answer actually used**. Give it the question, the passages and the answer. It
returns:
- a score and flag for every context sentence;
- the **utilization fraction**: the share of context sentences the answer used, with the RAGBench convention.

It is one of four CBX RAG evaluator models.

## How to use

```bash
pip install -U huggingface_hub
hf download CobuildX-AI/cbx-rag-context-utilization-68M --revision v0.1 --local-dir cbx-rag-context-utilization-68M
cd cbx-rag-context-utilization-68M
pip install -r requirements.txt
python example.py
```

```python
from cbx_rag_eval import ContextUtilization

judge = ContextUtilization(".")
out = judge(question, passages, answer)
out["utilization"]
[s for s in out["sentences"] if s["utilized"]]
```

**How the fraction is computed:** each sentence probability is calibrated (isotonic, fitted on validation), and the
fraction is their average.

**Prefilter (`contract` profile):** only the 64 sentences closest to the question and answer (BM25), plus their
neighbours, are scored; the rest count as unused. On validation this kept 98.8% of used sentences while scoring
about 39% of all sentences, with the same RMSE. Pass `prefilter=False` to score everything. The test numbers
below are without the prefilter.

**Speed** (seconds per call, model loaded; GPU NVIDIA TITAN RTX fp16, CPU 8 threads fp32):

| Input | GPU | CPU |
|---|---|---|
| one contract (about 414 sentences) | 0.61 s | 23.43 s |
| several short passages (about 29 sentences) | 0.12 s | 3.04 s |

## The four CBX RAG evaluator models

| Model | Judges | Output |
|---|---|---|
| [cbx-rag-faithfulness-68M](https://huggingface.co/CobuildX-AI/cbx-rag-faithfulness-68M) | each answer sentence against the passages | support score per sentence, answer score, hallucination flag |
| [cbx-rag-response-relevance-68M](https://huggingface.co/CobuildX-AI/cbx-rag-response-relevance-68M) | the answer against the question | direct / partial / irrelevant |
| [cbx-rag-context-relevance-68M](https://huggingface.co/CobuildX-AI/cbx-rag-context-relevance-68M) | each context sentence against the question | relevance per sentence, relevance fraction, context precision |
| [cbx-rag-context-utilization-68M](https://huggingface.co/CobuildX-AI/cbx-rag-context-utilization-68M) | each context sentence against the question and answer | use per sentence, utilization fraction |

All four share the wrapper `cbx_rag_eval.py`. Every model is fine-tuned from
[jhu-clsp/ettin-encoder-68m](https://huggingface.co/jhu-clsp/ettin-encoder-68m) (68M parameters). Current version:
**v0.1** (use `--revision v0.1` to pin it).

## Sentences and profiles

- **Sentences:** the models judge sentences, and they were trained on RAGBench's sentence splits. The wrapper
  splits raw text by profile (`split_sentences`; chosen on validation):
  - `contract`: line breaks, then NLTK punkt;
  - `financial`: blank lines, then pysbd.

  On validation, the wrapper's splitting instead of RAGBench's own sentence splits changed sentence AUROC 0.990 → 0.981 (CUAD); sentence AUROC 0.996 → 0.995 (FinQA); answer RMSE 0.040 → 0.043 (CUAD); answer RMSE 0.036 → 0.040 (FinQA). For the best accuracy, pass your own sentences (a list of strings per passage, and `answer_sentences=`
  for the answer); the splitter is then skipped.
- **Profiles:** the settings were fixed on two kinds of validation data, so every call takes a `profile`.
  - `"contract"`: one document, such as a contract (CUAD-like).
  - `"financial"`: several short passages with numbers and tables (FinQA-like). Faithfulness adds a numeric check
    here.
  - `"auto"` (default): `"contract"` for a single passage, `"financial"` for two or more.

  For other kinds of text, choose the profile whose shape is closest and check the results on your own data.

## Training data and labels

- **Source:** the CUAD (contracts) and FinQA (financial reports) subsets of
  [RAGBench](https://huggingface.co/datasets/galileo-ai/ragbench) (CC BY 4.0). Its answers were written by
  claude-3-haiku and gpt-3.5-turbo; its labels came from gpt-4-turbo.
  - Train: 1,530 CUAD and 12,502 FinQA answers.
  - Test: 510 CUAD (102 contracts) and 2,294 FinQA answers.
- **Labels were reviewed by LLMs, not humans:**
  - **Faithfulness:** majority vote of RAGBench and two LLM reviewers: Claude Haiku and gpt-5.4-mini, or two Haiku
    passes for CUAD train.
  - **CUAD context relevance / utilization:** majority vote of RAGBench and two Claude Haiku passes.
  - **FinQA context labels:** kept as RAGBench made them.
  - **Audits:** samples were checked blind by Claude Opus and Claude Sonnet.
- **Response relevance:** uses its own set (rr-v0.1). Claude Haiku wrote varied answers to the RAGBench questions
  and labelled them.
- **Test sets:** the RAGBench test splits, with the same label views.
- **Baselines:** RAGAS, GPT-3.5 and TruLens scores are the ones stored in RAGBench, not re-runs.
- **Rule:** every threshold and setting was fixed on validation.

## Licence

Apache-2.0 for the model weights and code.

**Data attribution:**
- RAGBench (Friel et al., 2024), CC BY 4.0;
- CUAD (The Atticus Project), CC BY 4.0;
- FinQA (Chen et al., 2021), MIT.

Labels were produced with Claude (Anthropic) and gpt-5.4-mini (OpenAI) models.

**Disclaimer:** the scores are statistical estimates for evaluating RAG systems. They are not a guarantee that an
answer is correct, complete or safe. Do not use them as the sole basis for legal, financial or other consequential
decisions.

## Results (test sets)

- **Labels:** CUAD uses reviewed labels (majority of RAGBench and two Claude Haiku passes); FinQA uses RAGBench
  labels.
- **Constant:** predicts the validation average.

**Sentence level:**

| Test set | AUROC [95% CI] | PR-AUC [95% CI] | Precision / recall at the threshold |
|---|---|---|---|
| CUAD contracts | 0.981 [0.973, 0.987] | 0.737 [0.697, 0.774] | 0.78 / 0.60 |
| FinQA reports | 0.995 [0.994, 0.996] | 0.945 [0.933, 0.956] | 0.90 / 0.88 |

**Answer-level utilization fraction:** RMSE, lower is better. GPT-3.5 is RAGBench's stored score; RAGBench has no
other utilization baseline.

| Test set | This model | GPT-3.5 | Constant |
|---|---|---|---|
| CUAD contracts (415 answers with a baseline score) | **0.039** | 0.059 | 0.063 |
| FinQA reports (2,294) | **0.033** | 0.134 | 0.067 |

Both differences against GPT-3.5 and the constant are significant (context-clustered bootstrap). On RAGBench's own
CUAD labels the model ties GPT-3.5 (0.108 vs 0.105).

## Limitations

- **CUAD recall:** at the default threshold it misses about 40% of used sentences (recall 0.60). When it flags a
  sentence, it is usually right (precision 0.78).
- **Label noise:** in a blind Claude Sonnet audit, 6 of 10 sentences the review added and 4 of 10 model misses were
  confirmed as used.
- **Not reported:** evidence completeness (relevant sentences the answer used). Combined with
  cbx-rag-context-relevance it did not beat a constant.
- **Coverage:** two domains (contracts, financial reports), English, LLM-made labels.