Download docs/evaluation.md from WickTech/lumen-rag: direct link, hf CLI and curl.
- Browser
- Download file 6.91 kB
-
https://huggingface.co/spaces/WickTech/lumen-rag/resolve/main/docs/evaluation.md
- Command line
-
hf download hf://spaces/WickTech/lumen-rag/docs/evaluation.md
-
curl -L -o evaluation.md https://huggingface.co/spaces/WickTech/lumen-rag/resolve/main/docs/evaluation.md
A newer version of the Gradio SDK is available: 6.32.0
Evaluation and testing
The test suite, the CI gates, claim-level answer diagnostics, and how Lumen compares with the large evaluation libraries. Back to the README.
🧪 Testing
pytest -q # 138 pass offline, 13 skip without optional services
ruff check . # lint
python scripts/check_eval.py data/eval.jsonl --k 3 # eval regression guard (v1 smoke set)
python scripts/validate_eval_set.py # eval set v2 consistency
python scripts/eval_report.py run --root . --out head.json # v2 metrics, same as the PR bot
CI runs on Python 3.10 & 3.12: lint → pytest → check_eval.py (enforces recall@k ≥ 0.80, hit_rate ≥ 0.80, mrr ≥ 0.70). The build fails if scores drop.
PR eval report
Every pull request that touches lumen_rag/, data/eval_v2/ or the report script gets a
sticky comment with the retrieval metrics (recall@5, MRR, nDCG@5, plus MRR per question
category) for vector, BM25 and hybrid search on the v2 eval set,
measured on the PR head and on its base commit, with the deltas. The
Eval report job fails if any tracked metric drops
by more than 0.01. Retrieval is deterministic offline (hashing embedder, no reranker), so a
drop is a real change, not noise. If a PR changes the eval set itself the numbers are shown
but marked not comparable and never block the merge. Fork PRs cannot receive the comment
(read-only token); the report is in the job summary instead.
🔬 Claim-level diagnostics — where do wrong answers come from?
Retrieval metrics say whether the right documents came back; answer judges say whether an
answer reads well. Neither says why an answer is wrong. lumen eval-answers --claims
splits every answer into claims, aligns each claim with the retrieved chunks, and attributes
failures to the retriever or the generator. It needs no API key and no judge model.
lumen ingest data/eval_v2/docs
lumen eval-answers data/eval_v2/eval.jsonl --claims --mode bm25 --out per_case.json
Per claim: supported (grounded in a retrieved chunk from a gold document), distractor
(grounded in retrieved non-gold text: faithful, but the wrong evidence) or unsupported
(not grounded in anything retrieved). Per answerable question: ok, retrieval-miss
(answer wrong and a gold document was not retrieved) or generation-error (answer wrong
although every gold document was retrieved). Per unanswerable question: abstained or
hallucinated.
Results on the v2 set, offline extractive answerer, k=5:
| Embedder | Retrieval mode | answer ok | retrieval-miss | generation-error | claim precision | faithfulness |
|---|---|---|---|---|---|---|
| hashing | vector-only | 0.73 | 0.23 | 0.05 | 0.27 | 1.00 |
| hashing | BM25 only | 0.88 | 0.09 | 0.03 | 0.36 | 1.00 |
| hashing | hybrid (BM25 + RRF) | 0.78 | 0.16 | 0.06 | 0.30 | 1.00 |
| bge-small | vector-only | 0.94 | 0.01 | 0.05 | 0.39 | 1.00 |
| bge-small | hybrid (BM25 + RRF) | 0.92 | 0.02 | 0.06 | 0.39 | 1.00 |
Reading it:
- Most wrong answers were retrieval failures, and a real embedder removes them. With the hashing embedder, 23% of questions fail because the gold document was not retrieved, against 5% where it was retrieved but the answer still missed it. bge-small cuts retrieval misses to 1% and lifts answer accuracy from 0.73 to 0.94; what is left is generation error.
- A
generation-errorhere means the answerer ignored evidence it had. The offline answerer pastes only the top 2 chunks, so a gold document retrieved at rank 3 to 5 still produces a wrong answer. unsupportedis 0 and faithfulness is 1.00 by construction: an extractive answerer only repeats retrieved text. These claim rates become informative with a generative answerer (OPENAI_API_KEY), where fabricated claims can appear.- Claim precision is mostly answer noise: the share of answer sentences that come from the gold document rather than from the other retrieved chunks it was stitched together with.
- The answerer itself never declines, so without abstention 0 of 63 unanswerable questions are
abstained. With--abstain-threshold 0.664(bge-small, vector) 17 are; see Abstention.
Alignment is a deterministic lexical overlap (share of a claim's content words found in a chunk,
default threshold --tau 0.6). It cannot see paraphrase or negation, so with a generative
answerer treat the rates as an approximation. diagnose_claims(..., aligner=fn) accepts any
(claim, chunk_text) -> score function, so an NLI model or an LLM judge can replace it.
🧭 Where Lumen fits
Lumen is a small RAG engine that ships with its own evaluation. It is not a replacement for the large
evaluation libraries, and lumen eval-answers actually uses one of them (DeepEval) as its LLM judge.
| Lumen RAG | Ragas | DeepEval | |
|---|---|---|---|
| What it is | RAG engine + built-in evaluation | RAG evaluation metrics library | LLM / RAG evaluation framework, pytest-style |
| Brings its own retrieval pipeline | Yes (swappable store, retriever, answerer) | No, you supply the pipeline | No, you supply the pipeline |
| Metric breadth | Small: recall / MRR / nDCG, claim-level attribution, optional DeepEval judging | RAG metrics (faithfulness, answer relevance, context precision / recall), synthetic test data | 50+ metrics across RAG, agents, multi-turn, safety |
| Needs an LLM judge | Not for retrieval metrics or --claims; yes for eval-answers |
LLM-based metrics need a judge model | Most metrics are LLM-judged |
| Works with no API key at all | Yes, by default (hashing embedder, extractive answerer) | Depends on the metric and the judge model | Depends on the metric and the judge model |
| Regression gate in CI | Threshold script + a PR comment with metric deltas, in this repo | Not a built-in feature I found | pytest integration |
| Hosted dashboard | No; exports OpenTelemetry traces instead | No | Optional (Confident AI) |
How to read it: pick Ragas or DeepEval when you already have a pipeline and want many LLM-judged metrics. Pick Lumen when you want a pipeline you can read end to end, deterministic offline evaluation that gates a pull request, and a way to tell retrieval failures from generation failures (see claim-level diagnostics below). The other two columns summarise their public documentation as reported by secondary sources in October 2026; check their docs for current details.