File size: 4,021 Bytes
f8a5189 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 | # Eval set v2
A harder retrieval benchmark for Lumen RAG. It exists because the original set
(`data/eval.jsonl`, 7 documents, 19 questions) is saturated: with `k=5` the
retriever returns 5 of the 7 documents, so recall is about 1.0 for any
configuration and cannot show an improvement.
## Contents
- `docs/` — 120 short documents (60 to 110 words each), one markdown file per
document. The file name (without `.md`) is the document id.
- `eval.jsonl` — 216 labelled questions.
## Provenance and licence
All documents and questions were written for this repository about a
**fictional** company, "Halden Robotics" (warehouse picking robots). No real
company, person or customer data is used and nothing is copied from another
source, so there are no third-party licence terms. The data is released under
the repository's MIT licence.
## Question categories
| Category | Count | What it tests |
|---|---|---|
| `direct` | 42 | Wording overlaps the document; the easy baseline. |
| `paraphrase` | 51 | Same fact, different words ("holiday days" vs "paid leave"). |
| `disambiguation` | 31 | Several near-identical documents (per region, plan, model or team); only one is right. |
| `multihop` | 29 | The answer needs two documents; both must be retrieved. |
| `unanswerable` | 63 | The corpus does not contain the answer. No relevant docs; not scored for retrieval, used to calibrate abstention. The last 40 are *near misses*: on-topic for a document that exists, but the fact asked for is absent ("Which brand of hardware security key does the company issue?"). |
The corpus is built from 24 topic families of about five documents each (time
off, equipment, expenses, on-call, runbooks, release process, pricing, support
terms, security, data retention, hiring, reviews, robot hardware, robot safety,
fleet API, deployment, procurement, finance, legal, marketing, facilities,
internal tools, engineering practice, customer success). Near-duplicate
documents within a family are deliberate distractors.
## Row format
```json
{"question": "...", "relevant_doc_ids": ["vacation-eu"], "expected_answer": "30 working days.",
"category": "direct", "answerable": true,
"evidence": [{"doc_id": "vacation-eu", "span": "receive 30 working days of annual paid leave"}]}
```
`evidence[].span` is a verbatim excerpt of the cited document, so every answerable
label can be checked mechanically. Unanswerable rows have `relevant_doc_ids: []`,
`answerable: false`, no `evidence` and no `expected_answer`.
## Using it
```bash
python scripts/validate_eval_set.py # consistency checks (ids, spans, duplicates, categories)
python scripts/benchmark.py --set v2 # configurations x categories table
lumen ingest data/eval_v2/docs
lumen eval data/eval_v2/eval.jsonl --k 5 # one-off scorecard with per-category breakdown
lumen eval-answers data/eval_v2/eval.jsonl --claims # why answers go wrong: retrieval vs generation
```
## Baseline (offline hashing embedder, no reranker, k=5)
| Configuration | recall@5 | MRR | nDCG@5 |
|---|---|---|---|
| vector-only | 0.74 | 0.69 | 0.69 |
| BM25 only | 0.90 | 0.88 | 0.87 |
| hybrid (BM25 + RRF) | 0.82 | 0.73 | 0.74 |
The spread between best and worst configuration is 0.19 MRR (the M0 target was at
least 0.10). `tests/test_eval_set_v2.py` guards this.
## Known limits
- The documents are short, so chunking has no effect on this set.
- Paraphrases still share some vocabulary with the source documents, which keeps
BM25 strong. A set written against a real semantic embedder would be harder.
- The cross-encoder reranker was not measured (it needs a model download).
- Written by one author with LLM assistance; questions and answers were checked
mechanically (spans, ids) but not by independent annotators.
## Extending it
Add a document under `docs/`, add rows to `eval.jsonl` with a verbatim `evidence`
span, then run `python scripts/validate_eval_set.py`. Keep one fact per question
and add distractor documents when you add a new topic.
|