docs/REPRODUCIBILITY.md

# Reproducibility

**Status tags:** `IMPLEMENTED` · `VERIFIED` · `MEASURED` · `NOT RUN` · `BLOCKED`.

This document states exactly what a third party can reproduce, with what, and what they cannot.

---

## 1. The reproducibility contract

| Guarantee | How it is enforced |
|---|---|
| **Frozen configuration** | All tunables live in `configs/base.yaml`; no magic numbers in Python. The loader validates invariants and computes a hash. |
| **Frozen config hash** | `78f1e3700da15aa1`. Every artifact records the hash it was produced against. |
| **Pinned backbones** | Every backbone is pinned by revision in `configs/base.yaml`; the Hub resolves the exact commit. |
| **Seed** | `project.seed: 42`. |
| **Immutable public test** | `evaluation.immutable_public_test: true`; `hidden_data_access: false`. |
| **Byte-verified artifacts** | Every released artifact ships with a sha256 in `models/checksums.sha256`. |
| **Verified metrics** | Every quoted number is checked against its artifact by `tools/verify_readme_metrics.py`. |

### 1.1 Invariants the loader enforces

Editing `configs/base.yaml` moves the hash and **invalidates every artifact keyed to it**. Two
invariants exist because their violation is a *silent* error:

```
fusion.input_dim            == 3 * croma.encoder_dim + croma.optical_channels + croma.sar_channels   # 2318
grounding_head.feature_dim  == 4 * grounding.encoder_projected_dim                                  # 2048
```

The second is especially load-bearing: a mismatch is a silent shape error that torch only raises at
the similarity step, after patch features are already cached. It is therefore rejected at load time.

## 2. Reproduce the metric verification (cheap, no GPU)

```bash
python release/tools/verify_readme_metrics.py
```

- **Reads** the artifacts under `artifacts/`.
- **Compares** each of the 20 quoted metrics at the precision printed in the README.
- **Also asserts** statuses (VLM headline contains `ACCEPTANCE-REJECTED`; router `corpus_limited` /
  `n_val`; calibration temperature and `ece_improvement`).
- **Exits 0** and prints `ALL CLAIMS VERIFIED` only when everything matches.

Committed output: `release/tools/readme_metrics_report.txt`.

## 3. Reproduce the environment (local run)

```bash
git clone https://github.com/Anish-lab-blip/SatQuery-AI
cd SatQuery-AI
python -m venv .venv
source .venv/Scripts/activate      # Windows git-bash; .venv/bin/activate on Linux/macOS
pip install -r requirements.txt
```

Python 3.11+ and a CPU are sufficient. Device is selected via `SATQUERY_DEVICE`; all placement is
`.to(device)`, never `.cuda()`. Backbones are fetched from the Hugging Face Hub on first use, pinned
by revision.

## 4. Reproduce the tests

| Suite | Command | Expected |
|---|---|---|
| Frontend live-wiring | `pytest tests/unit/test_frontend_live_wiring.py` | **106 passed** |
| Doc/frontend suite | `pytest` on the 5 doc/frontend files | **183 passed** |
| Full unit suite | `pytest tests/unit` | 5–6 **environmental** failures (sandbox delete guard × 4, 1 ordering flake, 1 stale adapter test) — see [`EVALUATION.md`](EVALUATION.md) §4 |

> **Environment note.** In the authoring sandbox the full suite trips a **bulk-delete guard** and
> pytest is only installed in the repository virtualenv. Neither affects a normal user environment.

## 5. Reproduce a live run

The deployed stack is reachable:

```bash
curl --noproxy '*' https://satquery-backend-m4yv.onrender.com/api/health
curl --noproxy '*' https://satquery-backend-m4yv.onrender.com/api/capabilities
```

`/api/capabilities` returns six tasks, all `available: true`. A live run requires the tunnel agent to
be connected (`agent_connected:true`); if the Codespace is stopped, the request parks until the
tunnel timeout. See [`DEPLOYMENT.md`](DEPLOYMENT.md) §5–6.

## 6. Reproduce the trained artifacts

Backbones are public and pinned; the **trained modules** are released on the Hugging Face Hub under
`thundercode/SatQuery`, each with its backbone dependency and checksum. To reproduce training:

| Artifact | Where it trains | Guide |
|---|---|---|
| router adapter | local CPU | `configs/base.yaml` §router.training |
| grounding head | local | `configs/base.yaml` §grounding_training |
| change head | local | `configs/base.yaml` §change |
| optical_sar fusion head | local, seed sweep | `docs/PHASE14_OPTICAL_SAR_DECISIONS.md` |
| change_vqa head | **external GPU (Kaggle)** | `docs/R02_KAGGLE_TRAINING_GUIDE.md` |
| vlm LoRA adapter | **external GPU** | `configs/base.yaml` §training |

### 6.1 What "reproduce" means for externally-trained artifacts

The change-VQA and VLM artifacts were trained **outside this repository**. Reproducing them requires
the external GPU environment described in their guides. The repository reproduces:
- the **promotion gate** (byte-identity, sha256, zero non-finite tensors);
- the **evaluation**;
- the **serving wiring**.

It does **not** ship a one-command retrain for those two artifacts. This is stated rather than
implied.

## 7. What is NOT reproducible from this release

| Item | Reason |
|---|---|
| The private deployment repos | They are private; the deployed sources are not in this release. |
| System-level end-to-end benchmark | **No such benchmark exists.** |
| Router test-split number | **Not run.** |
| CDVQA / SECOND imagery | Public but large; the release documents the acquisition + name-verification procedure, not the data. |
| BigEarthNet full corpus | **Not downloaded** (only a 28k S2 subset was used). |
| The historical ZeroGPU/Gradio deploy target | Frozen paperwork only; no runtime exists in code. |

## 8. Environment traps recorded for reproducibility

- **Dead proxy in the authoring sandbox** — outbound calls need `--noproxy '*'` (curl) or
  `ProxyHandler({})` (Python).
- **pytest only in the repo virtualenv** (`.venv/Scripts/python.exe`).
- **Full-suite pytest trips a bulk-delete guard** in the sandbox.
- **Cloudflare 308-redirects `X.html` → `/X`.**
- **Chrome drops synthetic CDP key events when the window lacks OS focus** — relevant to any
  browser-driven reproduction of the live validation.