metadata
license: mit
tags:
- rag
- retrieval
- evaluation
- statistics
- regression-testing
01 — Regression gate statistics
How often does a fixed-threshold PASS/FAIL gate on a small RAG golden set make the wrong call, and does a
paired non-inferiority gate do better? Full design: spec.html. Pre-registered hypotheses: prereg.md.
Published: results dataset · Space · code.
Run
Requires uv. CPU only.
uv sync # Python 3.12 env from uv.lock
uv run python scripts/run_all.py # data, runs, per-query metrics, simulation -> results/
uv run python scripts/latency.py # per-query latency on 2 threads (run on an idle machine)
uv run python scripts/report.py # hypotheses.json, summary CSVs, space/data/summary.json
uv run python scripts/figures.py # article/figures/*.png from results/
uv run python scripts/export_hf.py # hf_dataset/ ready for upload
uv run pytest -q # stats coverage, ranx cross-check, determinism
Quick pilot: uv run python scripts/run_all.py --datasets nfcorpus --R 200 --out pilot.
Embeddings and runs are cached in cache/, so re-running the simulation is fast.
Layout
| Path | What |
|---|---|
src/gatestats/stats.py |
Shared statistics toolkit (bootstrap, sign-flip, Wilson, MDE, Holm); copied by projects 02–20 |
src/gatestats/gates.py |
Gates G1, G2, G3, G3u, G3′, GL and a CI command (python -m gatestats.gates) |
src/gatestats/simulate.py |
Scenarios and golden-set subsampling simulation |
src/gatestats/{data,retrieve,metrics}.py |
BEIR loading, BM25 / exact / HNSW retrieval, per-query metrics |
scripts/ |
Pipeline entry points |
ci/rag-gate.example.yml |
Example GitHub Actions job using the paired gate |
space/ |
Static Hugging Face Space over the results |
article/draft.md |
Article draft |
article/article.html |
Article as a standalone page, built from the draft (command below) |
Rebuild the article page after editing the draft (requires pandoc):
pandoc article/draft.md -f gfm -s --template article/template.html --shift-heading-level-by=-1 --toc --toc-depth=2 --wrap=none \
| sed 's#<table>#<div class="tbl"><table>#; s#</table>#</table></div>#' > article/article.html
Use the gate in your own CI
uv run python -m gatestats.gates baseline.csv candidate.csv --metric recall@5 --margin 0.02
Both CSVs need qid and the metric column for the same queries. Exit code 1 means FAIL.