| --- |
| license: mit |
| tags: |
| - rag |
| - retrieval |
| - evaluation |
| - statistics |
| - regression-testing |
| --- |
| |
| # 01 — Regression gate statistics |
|
|
| How often does a fixed-threshold PASS/FAIL gate on a small RAG golden set make the wrong call, and does a |
| paired non-inferiority gate do better? Full design: `spec.html`. Pre-registered hypotheses: `prereg.md`. |
|
|
| Published: [results dataset](https://huggingface.co/datasets/bunyamineymen/rag-gate-stats-results) · [Space](https://huggingface.co/spaces/bunyamineymen/rag-gate-stats) · [code](https://huggingface.co/bunyamineymen/rag-gate-stats-code). |
|
|
| ## Run |
|
|
| Requires [uv](https://docs.astral.sh/uv/). CPU only. |
|
|
| ```bash |
| uv sync # Python 3.12 env from uv.lock |
| uv run python scripts/run_all.py # data, runs, per-query metrics, simulation -> results/ |
| uv run python scripts/latency.py # per-query latency on 2 threads (run on an idle machine) |
| uv run python scripts/report.py # hypotheses.json, summary CSVs, space/data/summary.json |
| uv run python scripts/figures.py # article/figures/*.png from results/ |
| uv run python scripts/export_hf.py # hf_dataset/ ready for upload |
| uv run pytest -q # stats coverage, ranx cross-check, determinism |
| ``` |
|
|
| Quick pilot: `uv run python scripts/run_all.py --datasets nfcorpus --R 200 --out pilot`. |
| Embeddings and runs are cached in `cache/`, so re-running the simulation is fast. |
|
|
| ## Layout |
|
|
| | Path | What | |
| |---|---| |
| | `src/gatestats/stats.py` | Shared statistics toolkit (bootstrap, sign-flip, Wilson, MDE, Holm); copied by projects 02–20 | |
| | `src/gatestats/gates.py` | Gates G1, G2, G3, G3u, G3′, GL and a CI command (`python -m gatestats.gates`) | |
| | `src/gatestats/simulate.py` | Scenarios and golden-set subsampling simulation | |
| | `src/gatestats/{data,retrieve,metrics}.py` | BEIR loading, BM25 / exact / HNSW retrieval, per-query metrics | |
| | `scripts/` | Pipeline entry points | |
| | `ci/rag-gate.example.yml` | Example GitHub Actions job using the paired gate | |
| | `space/` | Static Hugging Face Space over the results | |
| | `article/draft.md` | Article draft | |
| | `article/article.html` | Article as a standalone page, built from the draft (command below) | |
|
|
| Rebuild the article page after editing the draft (requires [pandoc](https://pandoc.org/)): |
|
|
| ```bash |
| pandoc article/draft.md -f gfm -s --template article/template.html --shift-heading-level-by=-1 --toc --toc-depth=2 --wrap=none \ |
| | sed 's#<table>#<div class="tbl"><table>#; s#</table>#</table></div>#' > article/article.html |
| ``` |
|
|
| ## Use the gate in your own CI |
|
|
| ```bash |
| uv run python -m gatestats.gates baseline.csv candidate.csv --metric recall@5 --margin 0.02 |
| ``` |
|
|
| Both CSVs need `qid` and the metric column for the same queries. Exit code 1 means FAIL. |
|
|
