data/writeup0.md

Retrieval ablation (golden set v1)

Golden set v1 (eval/golden/, drafted by Claude Code; a random sample of 30 reviewed by Nihar Damera (28 approved; 2 flagged, both resolved without change)). Answerable questions only: dev n = 50, test n = 70. 95% percentile-bootstrap intervals (10,000 resamples). A retrieved chunk counts when it overlaps a gold span; hit@10 = every part of the answer touched in the top 10 (recall@10 at question level); span recall@10 = share of the gold text the top 10 cover. Raw results: retrieval_dev.json, retrieval_test.json; paired differences: paired_differences.txt (computed from the per-question scores in those files).

Test (headline)

Configuration	hit@10	span recall@10	MRR	nDCG@10	words@10
section / dense (granite)	87.1 [78.6, 94.3]	89.3 [82.1, 95.7]	60.6 [51.3, 69.7]	67.2 [59.3, 74.9]	1851
section / dense (bge)	81.4 [71.4, 90.0]	85.0 [77.1, 92.1]	55.7 [47.3, 64.4]	63.2 [55.2, 71.2]	1933
section / BM25	84.3 [75.7, 92.9]	87.9 [80.7, 94.3]	64.0 [54.6, 73.1]	69.6 [61.3, 77.7]	2101
section / hybrid	92.9 [85.7, 98.6]	94.3 [88.6, 98.6]	68.3 [60.1, 75.9]	74.6 [67.7, 81.2]	1954
section / hybrid + rerank top 20	91.4 [84.3, 97.1]	95.0 [90.7, 98.6]	68.9 [60.7, 76.8]	74.9 [68.4, 81.2]	1961
fixed / hybrid	90.0 [82.9, 95.7]	91.3 [84.3, 97.0]	64.9 [55.8, 73.9]	71.3 [63.5, 78.7]	2117
recursive / hybrid	88.6 [80.0, 95.7]	88.4 [80.6, 95.0]	65.8 [56.8, 74.4]	72.4 [64.2, 80.1]	1700
fixed / dense (granite)	81.4 [71.4, 90.0]	82.9 [73.6, 91.4]	54.3 [45.1, 63.7]	61.0 [52.7, 69.5]	2083

All 18 configurations, both splits: retrieval_test.md, retrieval_dev.md.

Paired differences (same questions), test and dev

Comparison	test hit@10	test MRR	dev hit@10	dev MRR
hybrid − dense (section)	+5.7 [+1.4, +11.4]	+7.7 [−0.7, +16.2]	+8.0 [+0.0, +18.0]	+14.0 [+7.6, +20.9]
hybrid − BM25 (section)	+8.6 [+1.4, +15.7]	+4.2 [−2.6, +11.0]	−2.0 [−8.0, +4.0]	+4.8 [−2.4, +12.6]
rerank top 20 − hybrid (section)	−1.4 [−7.1, +4.3]	+0.6 [−7.1, +8.3]	+6.0 [+0.0, +14.0]	+4.5 [−2.4, +12.1]
rerank top 50 − top 20 (section)	−1.4 [−4.3, +0.0]	−0.7 [−1.4, −0.1]	+0.0	−0.2 [−0.5, +0.0]
section − fixed (hybrid)	+2.9 [−2.9, +8.6]	+3.4 [−5.0, +11.7]	−4.0 [−14.0, +6.0]	+8.5 [+1.3, +15.8]
section − recursive (hybrid)	+4.3 [−2.9, +11.4]	+2.5 [−5.6, +10.7]	+2.0 [−6.0, +10.0]	+3.1 [−3.1, +9.6]
bge − granite (section, dense)	−5.7 [−12.9, +0.0]	−4.9 [−14.3, +4.7]	+0.0 [−8.0, +8.0]	−0.1 [−10.8, +10.6]

What it says

1. Hybrid beats dense-only on recall, on both splits (test hit@10 +5.7 [+1.4, +11.4]). On ranking it is clear on dev (MRR +14.0 [+7.6, +20.9]) and on test nDCG@10 (+7.5 [+0.7, +14.5]), but not on test MRR (+7.7 [−0.7, +16.2]). On test, hybrid also beats BM25 alone on recall.
2. The cross-encoder reranker does not reliably help. On dev it looked like a small gain (hit@10 +6.0, interval touching zero); on test it is −1.4 on hit@10 and +0.6 on MRR. It costs 1.6-5 s a query on CPU (eval/retrieval_latency.json) against 9 ms for hybrid. Reranking 50 candidates instead of 20 adds nothing on either split. The rerankers are MS MARCO models trained on web queries; domain shift is a plausible reason, not a tested one.
3. Section-aware chunking is best on test but its lead over fixed and recursive chunks is within the intervals (it is clearer on dev MRR). Chunk sizes are matched (words@10 1,700-2,100), so this is about where cuts fall.
4. bge is no better than granite.

A correction: hybrid fusion was not repeatable

The first version of this table fused hybrid results inside Qdrant, whose RRF broke ties differently on every call (9 of 40 queries changed their top 5 on an immediate repeat). Fusion now happens in code with the same formula and ties broken by chunk ID (0 of 150 queries differ on repeat), and every hybrid row was re-run. Recall did not move; section / hybrid test MRR fell from 71.1 to 68.3, and the test MRR gain over dense went from +10.5 [+2.1, +19.1], which excluded zero, to +7.7 [−0.7, +16.2], which does not. Part of the first number was how ties happened to fall.

The default, and how it was chosen

Default: section-aware chunks, hybrid (granite + BM25, RRF), no reranker. Chosen on dev alone, the reranker (top 20) would have been kept: it led on dev hit@10 and nDCG. It is left off because its dev gain was at the edge of its interval, it adds seconds per query, and test did not reproduce it. That last reason used the test split, so the test numbers for the default are not a clean held-out estimate of a dev-chosen system; the dev-chosen alternative (section / hybrid + rerank 20) scores 91.4 hit@10 and 68.9 MRR on test, against 92.9 and 68.3 for the default.