Hugging Face's logo

bunyamineymen
/
rag-gate-stats-code

rag
retrieval
evaluation
statistics
regression-testing
Model card Files
xet
 Community
rag-gate-stats-code
File size: 2,782 Bytes
668f59a
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
---
license: mit
tags:
- rag
- retrieval
- evaluation
- statistics
- regression-testing
---

# 01 — Regression gate statistics

How often does a fixed-threshold PASS/FAIL gate on a small RAG golden set make the wrong call, and does a
paired non-inferiority gate do better? Full design: `spec.html`. Pre-registered hypotheses: `prereg.md`.

Published: [results dataset](https://huggingface.co/datasets/bunyamineymen/rag-gate-stats-results) · [Space](https://huggingface.co/spaces/bunyamineymen/rag-gate-stats) · [code](https://huggingface.co/bunyamineymen/rag-gate-stats-code).

## Run

Requires [uv](https://docs.astral.sh/uv/). CPU only.

```bash
uv sync                                   # Python 3.12 env from uv.lock
uv run python scripts/run_all.py          # data, runs, per-query metrics, simulation -> results/
uv run python scripts/latency.py          # per-query latency on 2 threads (run on an idle machine)
uv run python scripts/report.py           # hypotheses.json, summary CSVs, space/data/summary.json
uv run python scripts/figures.py          # article/figures/*.png from results/
uv run python scripts/export_hf.py        # hf_dataset/ ready for upload
uv run pytest -q                          # stats coverage, ranx cross-check, determinism
```

Quick pilot: `uv run python scripts/run_all.py --datasets nfcorpus --R 200 --out pilot`.
Embeddings and runs are cached in `cache/`, so re-running the simulation is fast.

## Layout

| Path | What |
|---|---|
| `src/gatestats/stats.py` | Shared statistics toolkit (bootstrap, sign-flip, Wilson, MDE, Holm); copied by projects 02–20 |
| `src/gatestats/gates.py` | Gates G1, G2, G3, G3u, G3′, GL and a CI command (`python -m gatestats.gates`) |
| `src/gatestats/simulate.py` | Scenarios and golden-set subsampling simulation |
| `src/gatestats/{data,retrieve,metrics}.py` | BEIR loading, BM25 / exact / HNSW retrieval, per-query metrics |
| `scripts/` | Pipeline entry points |
| `ci/rag-gate.example.yml` | Example GitHub Actions job using the paired gate |
| `space/` | Static Hugging Face Space over the results |
| `article/draft.md` | Article draft |
| `article/article.html` | Article as a standalone page, built from the draft (command below) |

Rebuild the article page after editing the draft (requires [pandoc](https://pandoc.org/)):

```bash
pandoc article/draft.md -f gfm -s --template article/template.html --shift-heading-level-by=-1 --toc --toc-depth=2 --wrap=none \
  | sed 's#<table>#<div class="tbl"><table>#; s#</table>#</table></div>#' > article/article.html
```

## Use the gate in your own CI

```bash
uv run python -m gatestats.gates baseline.csv candidate.csv --metric recall@5 --margin 0.02
```

Both CSVs need `qid` and the metric column for the same queries. Exit code 1 means FAIL.