File size: 6,908 Bytes
4f8feb6
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
# Evaluation and testing

The test suite, the CI gates, claim-level answer diagnostics, and how Lumen compares with the large evaluation libraries. Back to the [README](../README.md).

## 🧪 Testing

```bash
pytest -q                                                   # 138 pass offline, 13 skip without optional services
ruff check .                                                # lint
python scripts/check_eval.py data/eval.jsonl --k 3          # eval regression guard (v1 smoke set)
python scripts/validate_eval_set.py                         # eval set v2 consistency
python scripts/eval_report.py run --root . --out head.json  # v2 metrics, same as the PR bot
```

CI runs on Python 3.10 & 3.12: lint → pytest → `check_eval.py` (enforces recall@k ≥ 0.80, hit\_rate ≥ 0.80, mrr ≥ 0.70). The build fails if scores drop.

### PR eval report

Every pull request that touches `lumen_rag/`, `data/eval_v2/` or the report script gets a
sticky comment with the retrieval metrics (recall@5, MRR, nDCG@5, plus MRR per question
category) for vector, BM25 and hybrid search on the [v2 eval set](https://github.com/WickTech/lumen-rag/blob/master/data/eval_v2/README.md),
measured on the PR head **and** on its base commit, with the deltas. The
[`Eval report`](https://github.com/WickTech/lumen-rag/blob/master/github/workflows/eval-report.yml) job fails if any tracked metric drops
by more than 0.01. Retrieval is deterministic offline (hashing embedder, no reranker), so a
drop is a real change, not noise. If a PR changes the eval set itself the numbers are shown
but marked not comparable and never block the merge. Fork PRs cannot receive the comment
(read-only token); the report is in the job summary instead.

## 🔬 Claim-level diagnostics — where do wrong answers come from?

Retrieval metrics say whether the right *documents* came back; answer judges say whether an
answer *reads* well. Neither says why an answer is wrong. `lumen eval-answers --claims`
splits every answer into claims, aligns each claim with the retrieved chunks, and attributes
failures to the retriever or the generator. It needs no API key and no judge model.

```bash
lumen ingest data/eval_v2/docs
lumen eval-answers data/eval_v2/eval.jsonl --claims --mode bm25 --out per_case.json
```

Per **claim**: `supported` (grounded in a retrieved chunk from a gold document), `distractor`
(grounded in retrieved *non-gold* text: faithful, but the wrong evidence) or `unsupported`
(not grounded in anything retrieved). Per **answerable question**: `ok`, `retrieval-miss`
(answer wrong and a gold document was not retrieved) or `generation-error` (answer wrong
although every gold document was retrieved). Per **unanswerable question**: `abstained` or
`hallucinated`.

Results on the [v2 set](https://github.com/WickTech/lumen-rag/blob/master/data/eval_v2/README.md), offline extractive answerer, k=5:

| Embedder | Retrieval mode | answer ok | retrieval-miss | generation-error | claim precision | faithfulness |
|---|---|---|---|---|---|---|
| hashing | vector-only | 0.73 | 0.23 | 0.05 | 0.27 | 1.00 |
| hashing | BM25 only | 0.88 | 0.09 | 0.03 | 0.36 | 1.00 |
| hashing | hybrid (BM25 + RRF) | 0.78 | 0.16 | 0.06 | 0.30 | 1.00 |
| bge-small | vector-only | **0.94** | 0.01 | 0.05 | 0.39 | 1.00 |
| bge-small | hybrid (BM25 + RRF) | 0.92 | 0.02 | 0.06 | 0.39 | 1.00 |

Reading it:

- **Most wrong answers were retrieval failures, and a real embedder removes them.** With the hashing
  embedder, 23% of questions fail because the gold document was not retrieved, against 5% where it
  was retrieved but the answer still missed it. bge-small cuts retrieval misses to 1% and lifts
  answer accuracy from 0.73 to 0.94; what is left is generation error.
- **A `generation-error` here means the answerer ignored evidence it had.** The offline answerer
  pastes only the top 2 chunks, so a gold document retrieved at rank 3 to 5 still produces a wrong
  answer.
- **`unsupported` is 0 and faithfulness is 1.00 by construction**: an extractive answerer only
  repeats retrieved text. These claim rates become informative with a generative answerer
  (`OPENAI_API_KEY`), where fabricated claims can appear.
- **Claim precision is mostly answer noise**: the share of answer sentences that come from the gold
  document rather than from the other retrieved chunks it was stitched together with.
- **The answerer itself never declines**, so without abstention 0 of 63 unanswerable questions are
  `abstained`. With `--abstain-threshold 0.664` (bge-small, vector) 17 are; see
  [Abstention](abstention.md).

Alignment is a deterministic lexical overlap (share of a claim's content words found in a chunk,
default threshold `--tau 0.6`). It cannot see paraphrase or negation, so with a generative
answerer treat the rates as an approximation. `diagnose_claims(..., aligner=fn)` accepts any
`(claim, chunk_text) -> score` function, so an NLI model or an LLM judge can replace it.

## 🧭 Where Lumen fits

Lumen is a small RAG *engine* that ships with its own evaluation. It is not a replacement for the large
evaluation libraries, and `lumen eval-answers` actually uses one of them (DeepEval) as its LLM judge.

| | Lumen RAG | [Ragas](https://github.com/explodinggradients/ragas) | [DeepEval](https://github.com/confident-ai/deepeval) |
|---|---|---|---|
| What it is | RAG engine + built-in evaluation | RAG evaluation metrics library | LLM / RAG evaluation framework, pytest-style |
| Brings its own retrieval pipeline | Yes (swappable store, retriever, answerer) | No, you supply the pipeline | No, you supply the pipeline |
| Metric breadth | Small: recall / MRR / nDCG, claim-level attribution, optional DeepEval judging | RAG metrics (faithfulness, answer relevance, context precision / recall), synthetic test data | 50+ metrics across RAG, agents, multi-turn, safety |
| Needs an LLM judge | Not for retrieval metrics or `--claims`; yes for `eval-answers` | LLM-based metrics need a judge model | Most metrics are LLM-judged |
| Works with no API key at all | Yes, by default (hashing embedder, extractive answerer) | Depends on the metric and the judge model | Depends on the metric and the judge model |
| Regression gate in CI | Threshold script + a PR comment with metric deltas, in this repo | Not a built-in feature I found | pytest integration |
| Hosted dashboard | No; exports OpenTelemetry traces instead | No | Optional (Confident AI) |

How to read it: pick Ragas or DeepEval when you already have a pipeline and want many LLM-judged metrics.
Pick Lumen when you want a pipeline you can read end to end, deterministic offline evaluation that gates a
pull request, and a way to tell *retrieval* failures from *generation* failures (see claim-level diagnostics
below). The other two columns summarise their public documentation as reported by secondary sources in
October 2026; check their docs for current details.