release: add docs/LIMITATIONS.md

docs/LIMITATIONS.md

@@ -1,7 +1,8 @@
 # Limitations
 
-An honest catalogue of everything this system does **not** do, does poorly, or does not know. Negative
-results and open items are listed here rather than omitted.
+An honest, exhaustive catalogue of everything SatQuery AI does **not** do, does poorly, or does not
+know. Negative results and open items are listed here rather than omitted, because a limitation that
+is not written down is a limitation that will be discovered by someone else at the worst moment.
 
 **Status tags:** `OPEN` · `NOT RUN` · `REJECTED` · `DEFERRED` · `BY DESIGN`.
 
@@ -11,72 +12,97 @@ results and open items are listed here rather than omitted.
 
 | # | Limitation | Detail |
 |---|---|---|
-| 1 | **Grounding IoU is low in absolute terms** | mean best IoU 0.2838 (canonical) / 0.2566 (matched6). The trained head clearly beats the zero-shot baseline (0.0972), but 0.28 is not "solved". |
-| 2 | **Grounding depends on the box convention** | Two protocols and two decode variants produce very different numbers (0.2838 vs 0.1215 argmax). Absolute values are protocol-sensitive. |
-| 3 | **Optical-SAR accuracy is carried by common classes** | accuracy 0.931 but macro-F1 0.434161. 5 of 19 classes are absent in the scored split and contribute 0.0 to macro-F1 by construction. |
-| 4 | **Change-VQA is weak on rare classes** | accuracy 0.697626 / macro-F1 0.378373 (and 0.651469 / 0.372309 on the second test set). |
+| 1 | **Grounding IoU is low in absolute terms** | mean best IoU **0.2838** (canonical) / **0.2566** (matched6). The trained head clearly beats the zero-shot baseline (0.0972), but 0.28 is not "solved". |
+| 2 | **Grounding is protocol-sensitive** | two protocols × two decode variants give very different numbers: head_threshold 0.2838, head_argmax **0.1215**, zero-shot 0.0972. An absolute value is meaningless without its protocol. |
+| 3 | **Optical-SAR accuracy is carried by common classes** | accuracy **0.931** but macro-F1 **0.434161**. 5 of 19 classes are absent in the scored split and contribute 0.0 to macro-F1 by construction. Never quote accuracy alone. |
+| 4 | **Change-VQA is weak on rare classes** | accuracy 0.697626 / macro-F1 0.378373 (test) and 0.651469 / 0.372309 (test2). The wide accuracy–macroF1 gap is the signature of class imbalance. |
 | 5 | **VQA is weak-but-related** | the live VQA path answers broadly related content (e.g. "Grassland") rather than a crisp class. |
-| 6 | **Optical-SAR returns a bare class index** | the live service returns `class_18`, not a human-readable label. |
-| 7 | **Calibration made things worse** | ECE 0.013755 → 0.014929. Retained only because it is part of the frozen config. |
-| 8 | **The VLM adapter is not accepted** | metrics usable (exact_match 0.963), but the artifact is acceptance-rejected; the deployed path uses the unadapted model. |
-| 9 | **The router number is not a test result** | 0.965116 is validation, **ungated**, n = 86. The test split was **NOT RUN**. |
+| 6 | **Optical-SAR returns a bare class index** | the live service returns `class_18`, not a human-readable CLC label. |
+| 7 | **Calibration made things worse** | ECE **0.013755 → 0.014929** (`ece_improvement` −0.001174). Retained only because it is part of the frozen config — **not** because it helped. |
+| 8 | **The VLM adapter is not accepted** | metrics usable (exact_match 0.963, F1 0.96432, +49.5 pp) but the artifact is **ACCEPTANCE-REJECTED**; the deployed caption/VQA path uses the **unadapted** model. |
+| 9 | **The router number is not a test result** | 0.965116 is **validation**, **ungated**, **n = 86**, corpus-limited. The test split was **NOT RUN**. |
+| 10 | **The router's routing is not perfect** | known residuals below (§2). |
 
 ## 2. Router residuals (known misroutes)
 
 | # | Query | Behaviour | Note |
 |---|---|---|---|
-| 10 | *"What is the new runway?"* | reads `change`, not `vqa` | residual ambiguity — "new" triggers a change reading |
-| 11 | *"How much built-up area was added?"* | reads `vqa` (under-trigger) | a change-style quantifier not caught by the router |
-| 12 | *"What changed between the earlier and later image?"* with one asset | console **reads** `change`, **dispatches** `change_vqa` | intentional (asset-count-aware dispatch) but visually surprising — documented in [`ARCHITECTURE.md`](ARCHITECTURE.md) §4 |
+| 11 | *"What is the new runway?"* | reads `change`, not `vqa` | the word "new" triggers a change reading |
+| 12 | *"How much built-up area was added?"* | reads `vqa` (under-trigger) | a change-style quantifier the router does not catch |
+| 13 | *"What changed between the earlier and later image?"* with **one** asset | console **reads** `change`, **dispatches** `change_vqa` | **intentional** — reading is asset-count-blind, dispatch is asset-count-aware — but visually surprising. Documented in [`architecture/04-router.md`](architecture/04-router.md). |
 
 ## 3. Evaluation gaps
 
 | # | Gap | State |
 |---|---|---|
-| 13 | **No system-level end-to-end benchmark** | **NOT RUN — none exists.** No end-to-end accuracy is claimed. |
-| 14 | **Router test split** | **NOT RUN** |
-| 15 | **Benchmark adapters** | **NOT RUN** |
-| 16 | **End-to-end latency benchmark** | **NOT RUN** (per-specialist latency only, incidentally) |
-| 17 | **Cross-dataset generalisation** | **NOT RUN** — each specialist is evaluated on its own training-family split only |
-| 18 | **Human evaluation** | **NOT RUN** |
-| 19 | **Robustness / adversarial evaluation** | **NOT RUN** |
-| 20 | **BigEarthNet label semantics** | the local subset is 100 % single-label vs the official 1–11 multi-label scheme, so metrics are **not comparable** to published numbers |
-| 21 | **Reliability diagram** | the pre-scaling diagram is labelled as such; the calibrated curve is not plotted |
+| 14 | **No system-level end-to-end benchmark** | **NOT RUN — none exists.** No end-to-end accuracy is claimed anywhere. |
+| 15 | **Router test split** | **NOT RUN** |
+| 16 | **Benchmark adapters** | **NOT RUN** |
+| 17 | **End-to-end latency benchmark** | **NOT RUN** (per-specialist latency is recorded only incidentally) |
+| 18 | **Cross-dataset generalisation** | **NOT RUN** — each specialist is evaluated only on its own training-family split |
+| 19 | **Human evaluation** | **NOT RUN** |
+| 20 | **Robustness / adversarial evaluation** | **NOT RUN** |
+| 21 | **BigEarthNet label semantics** | the local subset is **100 % single-label** vs the official 1–11 multi-label scheme, so its metrics are **not comparable** to published numbers |
+| 22 | **Reliability diagram** | the shipped diagram is the **pre-scaling** curve (labelled as such); the calibrated curve is not plotted |
+| 23 | **Statistical significance for most metrics** | only the grounding resolution decision (448 vs 224) has a paired test with a confidence interval. Other per-task numbers are point estimates. |
 
 ## 4. Operational limitations
 
 | # | Limitation | State |
 |---|---|---|
-| 22 | **B-07: transient tunnel gaps** | a request can hang or return 504. Patch prepared, **NOT deployed**. **OPEN** |
-| 23 | **B-07 root shape** | in `auto` mode a tunnel timeout falls through to the forward path, burning `wake_timeout_s` (120 s) on a `302`; worst case ≈ 249 s. Measured. |
-| 24 | **Cold start is tens of seconds** | Render free tier sleeps; the Codespace may be stopped. Documented, not hidden. |
-| 25 | **B-02: `codespace_name` trailing newline** | cosmetic; the wake path strips it. **OPEN (cosmetic)** |
-| 26 | **No database, auth, or queue** | stateless gateway by design — no persistence of runs or users. |
-| 27 | **Deployment repos are private** | their links 404 for an outside audience. **BY DESIGN** |
-| 28 | **`deploy/` in the monorepo is stale/untracked** | not the deployed source. Trap. |
+| 24 | **B-07: transient tunnel gaps** | a request can hang or return 504. Patch prepared, **NOT deployed**. **OPEN** |
+| 25 | **B-07 root shape** | in `auto` mode a tunnel timeout falls through to the forward path, burning `wake_timeout_s` (120 s) on a `302`; worst case ≈ **249 s** (150 + 120). Measured. |
+| 26 | **Cold start is tens of seconds** | Render free tier sleeps; the Codespace may be stopped. Documented, not hidden. |
+| 27 | **B-02: `codespace_name` trailing newline** | cosmetic; the wake path strips it. **OPEN (cosmetic)** |
+| 28 | **No database, auth, or queue** | stateless gateway **by design** — no persistence of runs or users. |
+| 29 | **Deployment repos are private** | their links 404 for an outside audience. **BY DESIGN** |
+| 30 | **`deploy/` in the monorepo is stale/untracked** | not the deployed source. Trap. |
+| 31 | **No APM, distributed tracing, or cost accounting** | observability is limited to the health payload and per-run traces. |
+| 32 | **Single-region, no HA** | one Render service, one Codespace. |
 
 ## 5. Packaging and licensing
 
 | # | Limitation | State |
 |---|---|---|
-| 29 | **No `LICENSE` file** | none exists in the source repository. **OPEN** — a licence must be selected before public release of the code. |
-| 30 | **Backbones are not redistributed** | fetched from the Hub at run time; their licences are their own. |
-| 31 | **Artifact bloat** | `artifacts/` is ~3.7 GB, mostly reproducible caches, duplicate ZIPs and superseded checkpoints rather than released weights. |
+| 33 | **No `LICENSE` file** | none exists in the source repository. **OPEN** — a licence must be selected before public release of the code. |
+| 34 | **Backbones are not redistributed** | fetched from the Hub at run time; their licences are their own. |
+| 35 | **Artifact bloat** | `artifacts/` is ~3.7 GB, mostly reproducible caches, a duplicate 774 MB ZIP, a duplicate 241 MB probe, two 231 MB feature caches, a superseded 276 MB head, and ~235 MB of selection manifests — not released weights. |
+| 36 | **The released weights require their backbones** | the six artifacts are small modules; a consumer must also fetch the pinned backbones. |
 
 ## 6. Documentation caveats
 
 | # | Caveat | State |
 |---|---|---|
-| 32 | **`docs/FINAL_DELIVERY_REPORT.md` §6 is stale** | it still lists the bundled EO change pair as DEGRADED and B-01 as BLOCKED; both were resolved on 2026-09-25. |
-| 33 | **The monorepo `README.md` was materially stale** | it described a hermetic frontend, an in-progress Render/Codespace, and a `/v1/*` contract. Superseded by this release's README. |
-| 34 | **`hf/` docs were stale** | they asserted the project owns no weights and has no HF credentials. Both were false at release time. |
-| 35 | **Anatomy plate image variant** | points at the 720×720 variant of an image the recorded run analysed at 730×730. Cosmetic. |
+| 37 | **`docs/FINAL_DELIVERY_REPORT.md` §6 is stale** | it still lists the bundled EO change pair as DEGRADED and B-01 as BLOCKED; both were resolved on 2026-09-25. |
+| 38 | **The monorepo `README.md` was materially stale** | it described a hermetic frontend, an in-progress Render/Codespace, and a `/v1/*` contract. Superseded by this release's README. |
+| 39 | **`hf/` docs were stale** | they asserted the project owns no weights and has no HF credentials. Both were false at release time. |
+| 40 | **Anatomy plate image variant** | points at the 720×720 variant of an image the recorded run analysed at 730×730. Cosmetic. |
+| 41 | **The original master plan describes a superseded deployment** | it specifies a Gradio GUI + HF Space + ZeroGPU + Railway. The shipped system is a static frontend + Render + Codespace tunnel, serving JSON. |
 
 ## 7. Explicit non-claims
 
+These are things a reader might reasonably assume, which this project does **not** claim:
+
 - **No claim of state-of-the-art performance** on any benchmark.
-- **No claim of production readiness** for the model quality — the deployment runs, but the models
-  carry the limitations above.
+- **No claim of production readiness for model quality.** The deployment runs; the models carry the
+  limitations above.
 - **No claim that the trained heads generalise** beyond their training-family test splits.
-- **No claim that calibration improves confidence.**
+- **No claim that calibration improves confidence** — it made ECE worse.
 - **No claim that the VLM adapter is accepted** for production use.
+- **No claim of an end-to-end accuracy number** — none exists.
+- **No claim that the router is correct on all phrasings** — residuals exist.
+- **No claim of robustness** to adversarial, corrupted, or out-of-distribution inputs.
+- **No claim of geolocation accuracy** — grounding boxes are image-relative, not geodetic.
+- **No claim that the system is a safety-, legal-, or life-critical tool.**
+
+## 8. Where the evidence lives
+
+| Topic | Evidence |
+|---|---|
+| All measured metrics | `artifacts/**/*.json`, verified by `release/tools/verify_readme_metrics.py` |
+| Metric honesty rules | [`BENCHMARKS.md`](BENCHMARKS.md), [`EVALUATION.md`](EVALUATION.md) |
+| The grounding resolution rejection | `docs/PHASE7_RESOLUTION_DECISION.md` |
+| The VLM rejection | `artifacts/vlm/phase6_closure.json` (`why_acceptance_rejected`) |
+| B-07 / B-02 | [`DEPLOYMENT.md`](DEPLOYMENT.md) §8.1, §5 |
+| Router residuals | [`architecture/04-router.md`](architecture/04-router.md) |
+| Environment traps | [`REPRODUCIBILITY.md`](REPRODUCIBILITY.md) |