File size: 6,908 Bytes
4f8feb6 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 | # Evaluation and testing
The test suite, the CI gates, claim-level answer diagnostics, and how Lumen compares with the large evaluation libraries. Back to the [README](../README.md).
## 🧪 Testing
```bash
pytest -q # 138 pass offline, 13 skip without optional services
ruff check . # lint
python scripts/check_eval.py data/eval.jsonl --k 3 # eval regression guard (v1 smoke set)
python scripts/validate_eval_set.py # eval set v2 consistency
python scripts/eval_report.py run --root . --out head.json # v2 metrics, same as the PR bot
```
CI runs on Python 3.10 & 3.12: lint → pytest → `check_eval.py` (enforces recall@k ≥ 0.80, hit\_rate ≥ 0.80, mrr ≥ 0.70). The build fails if scores drop.
### PR eval report
Every pull request that touches `lumen_rag/`, `data/eval_v2/` or the report script gets a
sticky comment with the retrieval metrics (recall@5, MRR, nDCG@5, plus MRR per question
category) for vector, BM25 and hybrid search on the [v2 eval set](https://github.com/WickTech/lumen-rag/blob/master/data/eval_v2/README.md),
measured on the PR head **and** on its base commit, with the deltas. The
[`Eval report`](https://github.com/WickTech/lumen-rag/blob/master/github/workflows/eval-report.yml) job fails if any tracked metric drops
by more than 0.01. Retrieval is deterministic offline (hashing embedder, no reranker), so a
drop is a real change, not noise. If a PR changes the eval set itself the numbers are shown
but marked not comparable and never block the merge. Fork PRs cannot receive the comment
(read-only token); the report is in the job summary instead.
## 🔬 Claim-level diagnostics — where do wrong answers come from?
Retrieval metrics say whether the right *documents* came back; answer judges say whether an
answer *reads* well. Neither says why an answer is wrong. `lumen eval-answers --claims`
splits every answer into claims, aligns each claim with the retrieved chunks, and attributes
failures to the retriever or the generator. It needs no API key and no judge model.
```bash
lumen ingest data/eval_v2/docs
lumen eval-answers data/eval_v2/eval.jsonl --claims --mode bm25 --out per_case.json
```
Per **claim**: `supported` (grounded in a retrieved chunk from a gold document), `distractor`
(grounded in retrieved *non-gold* text: faithful, but the wrong evidence) or `unsupported`
(not grounded in anything retrieved). Per **answerable question**: `ok`, `retrieval-miss`
(answer wrong and a gold document was not retrieved) or `generation-error` (answer wrong
although every gold document was retrieved). Per **unanswerable question**: `abstained` or
`hallucinated`.
Results on the [v2 set](https://github.com/WickTech/lumen-rag/blob/master/data/eval_v2/README.md), offline extractive answerer, k=5:
| Embedder | Retrieval mode | answer ok | retrieval-miss | generation-error | claim precision | faithfulness |
|---|---|---|---|---|---|---|
| hashing | vector-only | 0.73 | 0.23 | 0.05 | 0.27 | 1.00 |
| hashing | BM25 only | 0.88 | 0.09 | 0.03 | 0.36 | 1.00 |
| hashing | hybrid (BM25 + RRF) | 0.78 | 0.16 | 0.06 | 0.30 | 1.00 |
| bge-small | vector-only | **0.94** | 0.01 | 0.05 | 0.39 | 1.00 |
| bge-small | hybrid (BM25 + RRF) | 0.92 | 0.02 | 0.06 | 0.39 | 1.00 |
Reading it:
- **Most wrong answers were retrieval failures, and a real embedder removes them.** With the hashing
embedder, 23% of questions fail because the gold document was not retrieved, against 5% where it
was retrieved but the answer still missed it. bge-small cuts retrieval misses to 1% and lifts
answer accuracy from 0.73 to 0.94; what is left is generation error.
- **A `generation-error` here means the answerer ignored evidence it had.** The offline answerer
pastes only the top 2 chunks, so a gold document retrieved at rank 3 to 5 still produces a wrong
answer.
- **`unsupported` is 0 and faithfulness is 1.00 by construction**: an extractive answerer only
repeats retrieved text. These claim rates become informative with a generative answerer
(`OPENAI_API_KEY`), where fabricated claims can appear.
- **Claim precision is mostly answer noise**: the share of answer sentences that come from the gold
document rather than from the other retrieved chunks it was stitched together with.
- **The answerer itself never declines**, so without abstention 0 of 63 unanswerable questions are
`abstained`. With `--abstain-threshold 0.664` (bge-small, vector) 17 are; see
[Abstention](abstention.md).
Alignment is a deterministic lexical overlap (share of a claim's content words found in a chunk,
default threshold `--tau 0.6`). It cannot see paraphrase or negation, so with a generative
answerer treat the rates as an approximation. `diagnose_claims(..., aligner=fn)` accepts any
`(claim, chunk_text) -> score` function, so an NLI model or an LLM judge can replace it.
## 🧭 Where Lumen fits
Lumen is a small RAG *engine* that ships with its own evaluation. It is not a replacement for the large
evaluation libraries, and `lumen eval-answers` actually uses one of them (DeepEval) as its LLM judge.
| | Lumen RAG | [Ragas](https://github.com/explodinggradients/ragas) | [DeepEval](https://github.com/confident-ai/deepeval) |
|---|---|---|---|
| What it is | RAG engine + built-in evaluation | RAG evaluation metrics library | LLM / RAG evaluation framework, pytest-style |
| Brings its own retrieval pipeline | Yes (swappable store, retriever, answerer) | No, you supply the pipeline | No, you supply the pipeline |
| Metric breadth | Small: recall / MRR / nDCG, claim-level attribution, optional DeepEval judging | RAG metrics (faithfulness, answer relevance, context precision / recall), synthetic test data | 50+ metrics across RAG, agents, multi-turn, safety |
| Needs an LLM judge | Not for retrieval metrics or `--claims`; yes for `eval-answers` | LLM-based metrics need a judge model | Most metrics are LLM-judged |
| Works with no API key at all | Yes, by default (hashing embedder, extractive answerer) | Depends on the metric and the judge model | Depends on the metric and the judge model |
| Regression gate in CI | Threshold script + a PR comment with metric deltas, in this repo | Not a built-in feature I found | pytest integration |
| Hosted dashboard | No; exports OpenTelemetry traces instead | No | Optional (Confident AI) |
How to read it: pick Ragas or DeepEval when you already have a pipeline and want many LLM-judged metrics.
Pick Lumen when you want a pipeline you can read end to end, deterministic offline evaluation that gates a
pull request, and a way to tell *retrieval* failures from *generation* failures (see claim-level diagnostics
below). The other two columns summarise their public documentation as reported by secondary sources in
October 2026; check their docs for current details.
|