Reproducibility

Status tags: IMPLEMENTED · VERIFIED · MEASURED · NOT RUN · BLOCKED.

This document states exactly what a third party can reproduce, with what, and what they cannot.

1. The reproducibility contract

Guarantee	How it is enforced
Frozen configuration	All tunables live in `configs/base.yaml`; no magic numbers in Python. The loader validates invariants and computes a hash.
Frozen config hash	`78f1e3700da15aa1`. Every artifact records the hash it was produced against.
Pinned backbones	Every backbone is pinned by revision in `configs/base.yaml`; the Hub resolves the exact commit.
Seed	`project.seed: 42`.
Immutable public test	`evaluation.immutable_public_test: true`; `hidden_data_access: false`.
Byte-verified artifacts	Every released artifact ships with a sha256 in `models/checksums.sha256`.
Verified metrics	Every quoted number is checked against its artifact by `tools/verify_readme_metrics.py`.

1.1 Invariants the loader enforces

Editing configs/base.yaml moves the hash and invalidates every artifact keyed to it. Two invariants exist because their violation is a silent error:

fusion.input_dim            == 3 * croma.encoder_dim + croma.optical_channels + croma.sar_channels   # 2318
grounding_head.feature_dim  == 4 * grounding.encoder_projected_dim                                  # 2048

The second is especially load-bearing: a mismatch is a silent shape error that torch only raises at the similarity step, after patch features are already cached. It is therefore rejected at load time.

2. Reproduce the metric verification (cheap, no GPU)

python release/tools/verify_readme_metrics.py

Reads the artifacts under artifacts/.
Compares each of the 20 quoted metrics at the precision printed in the README.
Also asserts statuses (VLM headline contains ACCEPTANCE-REJECTED; router corpus_limited / n_val; calibration temperature and ece_improvement).
Exits 0 and prints ALL CLAIMS VERIFIED only when everything matches.

Committed output: release/tools/readme_metrics_report.txt.

3. Reproduce the environment (local run)

git clone https://github.com/Anish-lab-blip/SatQuery-AI
cd SatQuery-AI
python -m venv .venv
source .venv/Scripts/activate      # Windows git-bash; .venv/bin/activate on Linux/macOS
pip install -r requirements.txt

Python 3.11+ and a CPU are sufficient. Device is selected via SATQUERY_DEVICE; all placement is .to(device), never .cuda(). Backbones are fetched from the Hugging Face Hub on first use, pinned by revision.

4. Reproduce the tests

Suite	Command	Expected
Frontend live-wiring	`pytest tests/unit/test_frontend_live_wiring.py`	106 passed
Doc/frontend suite	`pytest` on the 5 doc/frontend files	183 passed
Full unit suite	`pytest tests/unit`	5–6 environmental failures (sandbox delete guard × 4, 1 ordering flake, 1 stale adapter test) — see `EVALUATION.md` §4

Environment note. In the authoring sandbox the full suite trips a bulk-delete guard and pytest is only installed in the repository virtualenv. Neither affects a normal user environment.

5. Reproduce a live run

The deployed stack is reachable:

curl --noproxy '*' https://satquery-backend-m4yv.onrender.com/api/health
curl --noproxy '*' https://satquery-backend-m4yv.onrender.com/api/capabilities

/api/capabilities returns six tasks, all available: true. A live run requires the tunnel agent to be connected (agent_connected:true); if the Codespace is stopped, the request parks until the tunnel timeout. See DEPLOYMENT.md §5–6.

6. Reproduce the trained artifacts

Backbones are public and pinned; the trained modules are released on the Hugging Face Hub under thundercode/SatQuery, each with its backbone dependency and checksum. To reproduce training:

Artifact	Where it trains	Guide
router adapter	local CPU	`configs/base.yaml` §router.training
grounding head	local	`configs/base.yaml` §grounding_training
change head	local	`configs/base.yaml` §change
optical_sar fusion head	local, seed sweep	`docs/PHASE14_OPTICAL_SAR_DECISIONS.md`
change_vqa head	external GPU (Kaggle)	`docs/R02_KAGGLE_TRAINING_GUIDE.md`
vlm LoRA adapter	external GPU	`configs/base.yaml` §training

6.1 What "reproduce" means for externally-trained artifacts

The change-VQA and VLM artifacts were trained outside this repository. Reproducing them requires the external GPU environment described in their guides. The repository reproduces:

the promotion gate (byte-identity, sha256, zero non-finite tensors);
the evaluation;
the serving wiring.

It does not ship a one-command retrain for those two artifacts. This is stated rather than implied.

7. What is NOT reproducible from this release

Item	Reason
The private deployment repos	They are private; the deployed sources are not in this release.
System-level end-to-end benchmark	No such benchmark exists.
Router test-split number	Not run.
CDVQA / SECOND imagery	Public but large; the release documents the acquisition + name-verification procedure, not the data.
BigEarthNet full corpus	Not downloaded (only a 28k S2 subset was used).
The historical ZeroGPU/Gradio deploy target	Frozen paperwork only; no runtime exists in code.

8. Environment traps recorded for reproducibility

Dead proxy in the authoring sandbox — outbound calls need --noproxy '*' (curl) or ProxyHandler({}) (Python).
pytest only in the repo virtualenv (.venv/Scripts/python.exe).
Full-suite pytest trips a bulk-delete guard in the sandbox.
Cloudflare 308-redirects X.html → /X.
Chrome drops synthetic CDP key events when the window lacks OS focus — relevant to any browser-driven reproduction of the live validation.