docs/case-study.md

# Case study: what chunking and hybrid retrieval actually buy you

Lumen ships an eval harness (`lumen_rag/eval/`) specifically so claims like
"chunking helps" or "hybrid retrieval helps" don't have to stay vibes. This
is the measurement behind the numbers in the README.

## Setup

- **Corpus**: `data/docs/` — 7 markdown documents (~2,250 words total), each
  covering multiple unrelated handbook topics per file (e.g. `deploys-ops.md`
  bundles deployments, on-call, incident response, disaster recovery, plus
  three unrelated filler sections). This mirrors a real internal wiki page:
  long, multi-topic, with the answer to any given question living in one
  paragraph out of several.
- **Eval set**: `data/eval.jsonl` — 19 labelled questions, each with the
  correct source document id(s).
- **Embedder**: the offline deterministic `HashingEmbedder` (bag-of-words
  feature hashing, L2-normalised) — no API key, fully reproducible, what CI
  and the hosted demo run by default.
- **Metric**: `lumen_rag.eval.harness.evaluate`, doc-level (chunk hits are
  collapsed to unique parent documents before scoring), k=5.
- **Reproduce**: `python scripts/benchmark.py`

## Results

| Configuration | recall@5 | precision@5 | MRR | nDCG@5 | hit rate |
|---|---|---|---|---|---|
| naive — 1 chunk per doc, vector-only | 0.97 | 0.20 | 0.93 | 0.94 | 1.00 |
| + sentence-aware chunking, vector-only | 0.97 | 0.20 | **0.97** | **0.97** | 1.00 |
| + hybrid (BM25 + Reciprocal Rank Fusion) | 0.97 | 0.20 | **0.97** | **0.97** | 1.00 |

## Reading the numbers honestly

**Recall and hit-rate are already saturated** at 0.97/1.00 in the naive
config — with only 7 candidate documents and distinct enough vocabulary
per topic, the correct document almost always lands somewhere in the top 5
regardless of technique. Publishing only recall would (falsely) suggest
chunking doesn't matter here. It does — just not on that metric.

**MRR and nDCG@5 are where the effect shows up.** Both measure *where* the
correct document ranks, not just whether it's present. Moving from naive to
chunked lifts MRR from 0.93 to 0.97 — a small absolute jump that maps to a
concrete failure mode disappearing: when a whole 300–450 word multi-topic
document is embedded as a single vector, the sections irrelevant to the
query dilute the average, and the correct document occasionally ranks 2nd
or 3rd behind a partial-vocabulary-match decoy instead of 1st. Sentence-aware
chunking (120-word windows, 20-word overlap) embeds each section on its own,
so the answer-bearing chunk competes on its own signal instead of being
outvoted by the rest of the document.

**Hybrid ties chunked-vector, not because RRF doesn't work, but because the
offline `HashingEmbedder` is itself a bag-of-words signal** — term-frequency
counts, L2-normalised. That's structurally close to what BM25 computes, so
fusing the two rankers mostly agrees with itself. Hybrid's actual value
proposition — catching exact rare-term or numeric matches that a *semantic*
embedding model under-weights in favor of topical similarity — needs a real
semantic embedder (`OPENAI_API_KEY` set, `text-embedding-3-small`) and a
larger, noisier corpus to demonstrate honestly. That's flagged as a follow-up
rather than asserted with numbers we didn't measure.

## Takeaways for anyone building on Lumen

1. **Don't just report recall/hit-rate.** They saturate fast on small
   corpora and hide ranking-quality regressions. MRR/nDCG catch what recall
   misses.
2. **Chunking's benefit scales with document length and topic density**, not
   corpus size. A corpus of short, single-topic documents won't show this
   effect — you need documents where the answer is a minority of the content.
3. **Hybrid retrieval's payoff is embedder-dependent.** With a lexical/hash
   embedder it's close to redundant with vector search; with a real semantic
   embedder it complements it. Re-run `scripts/benchmark.py` with
   `OPENAI_API_KEY` set to see the difference on your own corpus.