bunyamineymen/rag-gate-stats-code

01 — Regression gate statistics

How often does a fixed-threshold PASS/FAIL gate on a small RAG golden set make the wrong call, and does a paired non-inferiority gate do better? Full design: spec.html. Pre-registered hypotheses: prereg.md.

Published: results dataset · Space · code.

Run

Requires uv. CPU only.

uv sync                                   # Python 3.12 env from uv.lock
uv run python scripts/run_all.py          # data, runs, per-query metrics, simulation -> results/
uv run python scripts/latency.py          # per-query latency on 2 threads (run on an idle machine)
uv run python scripts/report.py           # hypotheses.json, summary CSVs, space/data/summary.json
uv run python scripts/figures.py          # article/figures/*.png from results/
uv run python scripts/export_hf.py        # hf_dataset/ ready for upload
uv run pytest -q                          # stats coverage, ranx cross-check, determinism

Quick pilot: uv run python scripts/run_all.py --datasets nfcorpus --R 200 --out pilot. Embeddings and runs are cached in cache/, so re-running the simulation is fast.

Layout

Path	What
src/gatestats/stats.py	Shared statistics toolkit (bootstrap, sign-flip, Wilson, MDE, Holm); copied by projects 02–20
src/gatestats/gates.py	Gates G1, G2, G3, G3u, G3′, GL and a CI command (python -m gatestats.gates)
src/gatestats/simulate.py	Scenarios and golden-set subsampling simulation
src/gatestats/{data,retrieve,metrics}.py	BEIR loading, BM25 / exact / HNSW retrieval, per-query metrics
scripts/	Pipeline entry points
ci/rag-gate.example.yml	Example GitHub Actions job using the paired gate
space/	Static Hugging Face Space over the results
article/draft.md	Article draft
article/article.html	Article as a standalone page, built from the draft (command below)

Rebuild the article page after editing the draft (requires pandoc):

pandoc article/draft.md -f gfm -s --template article/template.html --shift-heading-level-by=-1 --toc --toc-depth=2 --wrap=none \
  | sed 's#<table>#<div class="tbl"><table>#; s#</table>#</table></div>#' > article/article.html

Use the gate in your own CI

uv run python -m gatestats.gates baseline.csv candidate.csv --metric recall@5 --margin 0.02

Both CSVs need qid and the metric column for the same queries. Exit code 1 means FAIL.