File size: 8,636 Bytes
4f8feb6
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
# Benchmarks

What embeddings, retrieval mode, reranking, contextual headers and query decomposition actually buy you, measured on the bundled eval sets. Back to the [README](../README.md).

## 📊 Benchmarks — what embeddings, retrieval mode and reranking actually buy you

Measured with `scripts/benchmark.py`, k=5. Two eval sets ship with the repo:

- **v1** — `data/docs` (7 files) + `data/eval.jsonl` (19 questions). A fast smoke
  test, but **saturated**: k=5 returns 5 of the 7 documents, so recall is ~1.0
  for any configuration and tells you nothing.
- **v2** — `data/eval_v2` (120 short docs, 216 questions in five categories:
  direct, paraphrase, disambiguation, multihop, unanswerable). Built so that
  configurations can be told apart; see [`data/eval_v2/README.md`](https://github.com/WickTech/lumen-rag/blob/master/data/eval_v2/README.md).

Reproduce:

```bash
python scripts/benchmark.py --set v2 --abstain                                # hashing embedder (CI default)
pip install -e ".[embed,rerank]"
python scripts/benchmark.py --set v2 --embedder fastembed --rerank --abstain  # bge-small + MiniLM reranker
```

### v2 with a real embedder (`BAAI/bge-small-en-v1.5`, local, CPU)

| Configuration | recall@5 | precision@5 | MRR | nDCG@5 | hit rate | ms/query |
|---|---|---|---|---|---|---|
| vector-only | 0.97 | 0.23 | 0.95 | 0.94 | 0.99 | 19 |
| BM25 only | 0.90 | 0.21 | 0.88 | 0.87 | 0.94 | 1 |
| hybrid (BM25 + RRF) | 0.97 | 0.23 | 0.91 | 0.91 | 0.99 | 15 |
| hybrid + rerank | 0.97 | 0.23 | 0.96 | 0.95 | 0.98 | 488 |
| vector-only + rerank | **0.98** | 0.23 | **0.97** | **0.96** | 0.99 | 489 |

MRR by question category (153 answerable questions; the 63 unanswerable ones are used for abstention below):

| Configuration | direct | disambiguation | multihop | paraphrase |
|---|---|---|---|---|
| vector-only | 0.98 | 0.96 | 0.93 | 0.93 |
| BM25 only | 0.98 | 0.89 | 0.86 | 0.80 |
| hybrid (BM25 + RRF) | 0.99 | 0.87 | 0.95 | 0.86 |
| hybrid + rerank | 1.00 | 0.97 | 0.96 | 0.92 |
| vector-only + rerank | 1.00 | 0.98 | 0.96 | 0.94 |

### v2 with the offline hashing embedder (what CI runs)

| Configuration | recall@5 | precision@5 | MRR | nDCG@5 | hit rate |
|---|---|---|---|---|---|
| vector-only | 0.74 | 0.17 | 0.69 | 0.69 | 0.78 |
| BM25 only | **0.90** | 0.21 | **0.88** | **0.87** | **0.94** |
| hybrid (BM25 + RRF) | 0.82 | 0.19 | 0.73 | 0.74 | 0.86 |

| Configuration | direct | disambiguation | multihop | paraphrase |
|---|---|---|---|---|
| vector-only | 0.84 | 0.77 | 0.67 | 0.54 |
| BM25 only | 0.98 | 0.89 | 0.86 | 0.80 |
| hybrid (BM25 + RRF) | 0.84 | 0.78 | 0.76 | 0.59 |

What the two tables say:

- **The embedder was the bottleneck.** Swapping the hashing embedder for bge-small lifts
  vector-only MRR from 0.69 to 0.95, and paraphrase MRR from 0.54 to 0.93: the questions that
  share no words with their document are exactly what a semantic model fixes.
- **The ranking of strategies flips.** With the weak embedder, BM25 alone wins and fusion drags it
  down. With a real one, dense search wins and *BM25 now drags the fusion down* (MRR 0.95 → 0.91,
  disambiguation 0.96 → 0.87): equal-weight RRF lets the weaker list promote near-duplicate
  distractors. "Hybrid is always better" is not true on this corpus; measure it on yours.
- **Reranking buys the last few points at 25× the latency.** A 22M-parameter cross-encoder
  (`Xenova/ms-marco-MiniLM-L-6-v2`, CPU) takes MRR from 0.95 to 0.97 and costs ~470 ms a query.
  Worth it when precision at rank 1 matters; not for an autocomplete box.
- **Contextual chunk headers help keyword search, barely move dense search.** With
  `LUMEN_CONTEXTUAL=1` each chunk is embedded and BM25-indexed with a header naming its document
  and section, e.g. `Document: Artificial Intelligence Risk Management Framework (AI RMF 1.0).
  Section: 3. AI Risks and Trustworthiness > 3.4 Accountable and Transparent.`, built offline from
  the title and the markdown, numbered or appendix headings (tables of contents and list items are
  skipped). The header is used for search only; answers and citations show the plain chunk, and
  citations name the section. See [Contextual chunk headers](#contextual-chunk-headers) below.
- **Query decomposition did not help (a negative result, kept opt-in).** Splitting compound
  questions at the second question word ("…, and how soon must I submit the claim?") and adding
  each sub-question's best new document recovers the missing hop on some multi-hop questions and
  loses it on as many: multi-hop recall@5 stays 0.90 with bge-small (0.95 with the reranker) and
  drops from 0.88 to 0.83 in hybrid mode. Equal-weight RRF of the sub-query lists did worse
  (0.88). The remaining misses are *implicit* hops ("How much bonus do I earn if checkout is
  down?" needs the severity table) that no lexical split can see; an LLM rewrite is the next thing
  to try. Reproduce with `--decompose`; enable with `LUMEN_DECOMPOSE=1`.
- **Not measured yet:** OpenAI `text-embedding-3-small`. The run is wired up
  (`--embedder openai`) but the key available when these numbers were taken had no API credit.

### Contextual chunk headers

[Anthropic's Contextual Retrieval](https://www.anthropic.com/news/contextual-retrieval) prepends an
LLM-written context to each chunk and reports 35% fewer failed retrievals (49% with contextual BM25).
Lumen's headers are the offline, deterministic version: document title plus heading path, no model.
v2 documents fit in one 120-word chunk, so they only show the effect with 40-word chunks, where a
chunk loses its title; the NIST AI RMF set (22 questions graded by evidence spans,
[`data/nist_rmf`](../data/nist_rmf/README.md)) tests a real 48-page PDF with its section structure.

| Set | Embedder, mode | without headers | with headers |
|---|---|---|---|
| v2, 40-word chunks (MRR) | bge-small, vector | 0.93 | 0.92 |
| v2, 40-word chunks (MRR) | bge-small, hybrid | 0.90 | **0.92** (disambiguation 0.87 → 0.92) |
| v2, 40-word chunks (MRR) | BM25 | 0.81 | **0.85** |
| NIST PDF (hit@5) | bge-small, vector | 0.86 | **0.91** |
| NIST PDF (hit@5) | bge-small, hybrid | 0.91 | 0.91 |
| NIST PDF (hit@5) | BM25 | 0.82 | 0.77 |
| NIST PDF (hit@5) | hashing, hybrid | 0.55 | **0.73** |

- **Keyword search gains most on multi-document corpora**: the title gives every chunk the words
  that tell look-alike documents apart (Pro vs Starter plan). Inside one document the title is the
  same for every chunk, only the section path discriminates, and BM25 loses a question to the longer
  chunks.
- **Dense search with a good embedder barely moves** (one NIST question gained, v2 flat). The
  gains are a fraction of Anthropic's because a title and a heading path say much less than an
  LLM-written summary of where the chunk sits; that version needs an API key and is the obvious next
  step.
- **Headers move the abstention scores, so recalibrate when you turn them on.** On v2 the
  similarity threshold barely changes (0.666 vs 0.664), but on the NIST PDF, whose header is a
  long title plus section path, the end-to-end check with both gates drops from 10/10 to 8/10:
  "Which function is designed to be cross-cutting…" falls from 0.70 to 0.60 similarity and is
  wrongly refused, and one near miss rises above the reader threshold. That is why headers are off
  by default and off on the HF Space, whose thresholds were calibrated without them.
- NIST differences are one question (0.045) at a time, so read them as direction, not size.

Reproduce: `python scripts/benchmark.py --set v2 --embedder fastembed --contextual`,
`python scripts/real_pdf_check.py --retrieval`, and `LUMEN_CONTEXTUAL=1 python scripts/real_pdf_check.py`.
Enable with `LUMEN_CONTEXTUAL=1` and re-ingest.

### v1 (smoke test, hashing embedder)

| Configuration | recall@5 | precision@5 | MRR | nDCG@5 | hit rate |
|---|---|---|---|---|---|
| naive (1 chunk/doc, vector-only) | 1.00 | 0.21 | 0.96 | 0.98 | 1.00 |
| + sentence chunking (vector-only) | 0.97 | 0.20 | **1.00** | 0.98 | 1.00 |
| BM25 only (sentence chunking) | 0.97 | 0.20 | 0.96 | 0.96 | 1.00 |
| hybrid (BM25 + RRF, sentence chunking) | 0.97 | 0.20 | 0.97 | 0.97 | 1.00 |

> **Correction (2026-10-18):** earlier versions of this table (and
> [`docs/case-study.md`](https://github.com/WickTech/lumen-rag/blob/master/docs/case-study.md)) reported naive MRR 0.93 and "+ hybrid"
> equal to chunked-vector. `evaluate()` did not apply the retrieval mode, so every
> row had in fact been measured in hybrid mode and the "vector-only" labels were
> wrong. `evaluate()` now takes `mode=`, and the numbers above are the real ones.