| # Reproducibility |
|
|
| **Status tags:** `IMPLEMENTED` · `VERIFIED` · `MEASURED` · `NOT RUN` · `BLOCKED`. |
|
|
| This document states exactly what a third party can reproduce, with what, and what they cannot. |
|
|
| --- |
|
|
| ## 1. The reproducibility contract |
|
|
| | Guarantee | How it is enforced | |
| |---|---| |
| | **Frozen configuration** | All tunables live in `configs/base.yaml`; no magic numbers in Python. The loader validates invariants and computes a hash. | |
| | **Frozen config hash** | `78f1e3700da15aa1`. Every artifact records the hash it was produced against. | |
| | **Pinned backbones** | Every backbone is pinned by revision in `configs/base.yaml`; the Hub resolves the exact commit. | |
| | **Seed** | `project.seed: 42`. | |
| | **Immutable public test** | `evaluation.immutable_public_test: true`; `hidden_data_access: false`. | |
| | **Byte-verified artifacts** | Every released artifact ships with a sha256 in `models/checksums.sha256`. | |
| | **Verified metrics** | Every quoted number is checked against its artifact by `tools/verify_readme_metrics.py`. | |
|
|
| ### 1.1 Invariants the loader enforces |
|
|
| Editing `configs/base.yaml` moves the hash and **invalidates every artifact keyed to it**. Two |
| invariants exist because their violation is a *silent* error: |
|
|
| ``` |
| fusion.input_dim == 3 * croma.encoder_dim + croma.optical_channels + croma.sar_channels # 2318 |
| grounding_head.feature_dim == 4 * grounding.encoder_projected_dim # 2048 |
| ``` |
|
|
| The second is especially load-bearing: a mismatch is a silent shape error that torch only raises at |
| the similarity step, after patch features are already cached. It is therefore rejected at load time. |
|
|
| ## 2. Reproduce the metric verification (cheap, no GPU) |
|
|
| ```bash |
| python release/tools/verify_readme_metrics.py |
| ``` |
|
|
| - **Reads** the artifacts under `artifacts/`. |
| - **Compares** each of the 20 quoted metrics at the precision printed in the README. |
| - **Also asserts** statuses (VLM headline contains `ACCEPTANCE-REJECTED`; router `corpus_limited` / |
| `n_val`; calibration temperature and `ece_improvement`). |
| - **Exits 0** and prints `ALL CLAIMS VERIFIED` only when everything matches. |
|
|
| Committed output: `release/tools/readme_metrics_report.txt`. |
|
|
| ## 3. Reproduce the environment (local run) |
|
|
| ```bash |
| git clone https://github.com/Anish-lab-blip/SatQuery-AI |
| cd SatQuery-AI |
| python -m venv .venv |
| source .venv/Scripts/activate # Windows git-bash; .venv/bin/activate on Linux/macOS |
| pip install -r requirements.txt |
| ``` |
|
|
| Python 3.11+ and a CPU are sufficient. Device is selected via `SATQUERY_DEVICE`; all placement is |
| `.to(device)`, never `.cuda()`. Backbones are fetched from the Hugging Face Hub on first use, pinned |
| by revision. |
|
|
| ## 4. Reproduce the tests |
|
|
| | Suite | Command | Expected | |
| |---|---|---| |
| | Frontend live-wiring | `pytest tests/unit/test_frontend_live_wiring.py` | **106 passed** | |
| | Doc/frontend suite | `pytest` on the 5 doc/frontend files | **183 passed** | |
| | Full unit suite | `pytest tests/unit` | 5–6 **environmental** failures (sandbox delete guard × 4, 1 ordering flake, 1 stale adapter test) — see [`EVALUATION.md`](EVALUATION.md) §4 | |
|
|
| > **Environment note.** In the authoring sandbox the full suite trips a **bulk-delete guard** and |
| > pytest is only installed in the repository virtualenv. Neither affects a normal user environment. |
|
|
| ## 5. Reproduce a live run |
|
|
| The deployed stack is reachable: |
|
|
| ```bash |
| curl --noproxy '*' https://satquery-backend-m4yv.onrender.com/api/health |
| curl --noproxy '*' https://satquery-backend-m4yv.onrender.com/api/capabilities |
| ``` |
|
|
| `/api/capabilities` returns six tasks, all `available: true`. A live run requires the tunnel agent to |
| be connected (`agent_connected:true`); if the Codespace is stopped, the request parks until the |
| tunnel timeout. See [`DEPLOYMENT.md`](DEPLOYMENT.md) §5–6. |
|
|
| ## 6. Reproduce the trained artifacts |
|
|
| Backbones are public and pinned; the **trained modules** are released on the Hugging Face Hub under |
| `thundercode/SatQuery`, each with its backbone dependency and checksum. To reproduce training: |
|
|
| | Artifact | Where it trains | Guide | |
| |---|---|---| |
| | router adapter | local CPU | `configs/base.yaml` §router.training | |
| | grounding head | local | `configs/base.yaml` §grounding_training | |
| | change head | local | `configs/base.yaml` §change | |
| | optical_sar fusion head | local, seed sweep | `docs/PHASE14_OPTICAL_SAR_DECISIONS.md` | |
| | change_vqa head | **external GPU (Kaggle)** | `docs/R02_KAGGLE_TRAINING_GUIDE.md` | |
| | vlm LoRA adapter | **external GPU** | `configs/base.yaml` §training | |
|
|
| ### 6.1 What "reproduce" means for externally-trained artifacts |
|
|
| The change-VQA and VLM artifacts were trained **outside this repository**. Reproducing them requires |
| the external GPU environment described in their guides. The repository reproduces: |
| - the **promotion gate** (byte-identity, sha256, zero non-finite tensors); |
| - the **evaluation**; |
| - the **serving wiring**. |
|
|
| It does **not** ship a one-command retrain for those two artifacts. This is stated rather than |
| implied. |
|
|
| ## 7. What is NOT reproducible from this release |
|
|
| | Item | Reason | |
| |---|---| |
| | The private deployment repos | They are private; the deployed sources are not in this release. | |
| | System-level end-to-end benchmark | **No such benchmark exists.** | |
| | Router test-split number | **Not run.** | |
| | CDVQA / SECOND imagery | Public but large; the release documents the acquisition + name-verification procedure, not the data. | |
| | BigEarthNet full corpus | **Not downloaded** (only a 28k S2 subset was used). | |
| | The historical ZeroGPU/Gradio deploy target | Frozen paperwork only; no runtime exists in code. | |
|
|
| ## 8. Environment traps recorded for reproducibility |
|
|
| - **Dead proxy in the authoring sandbox** — outbound calls need `--noproxy '*'` (curl) or |
| `ProxyHandler({})` (Python). |
| - **pytest only in the repo virtualenv** (`.venv/Scripts/python.exe`). |
| - **Full-suite pytest trips a bulk-delete guard** in the sandbox. |
| - **Cloudflare 308-redirects `X.html` → `/X`.** |
| - **Chrome drops synthetic CDP key events when the window lacks OS focus** — relevant to any |
| browser-driven reproduction of the live validation. |
|
|
