| # Limitations |
|
|
| An honest catalogue of everything this system does **not** do, does poorly, or does not know. Negative |
| results and open items are listed here rather than omitted. |
|
|
| **Status tags:** `OPEN` · `NOT RUN` · `REJECTED` · `DEFERRED` · `BY DESIGN`. |
|
|
| --- |
|
|
| ## 1. Model quality |
|
|
| | # | Limitation | Detail | |
| |---|---|---| |
| | 1 | **Grounding IoU is low in absolute terms** | mean best IoU 0.2838 (canonical) / 0.2566 (matched6). The trained head clearly beats the zero-shot baseline (0.0972), but 0.28 is not "solved". | |
| | 2 | **Grounding depends on the box convention** | Two protocols and two decode variants produce very different numbers (0.2838 vs 0.1215 argmax). Absolute values are protocol-sensitive. | |
| | 3 | **Optical-SAR accuracy is carried by common classes** | accuracy 0.931 but macro-F1 0.434161. 5 of 19 classes are absent in the scored split and contribute 0.0 to macro-F1 by construction. | |
| | 4 | **Change-VQA is weak on rare classes** | accuracy 0.697626 / macro-F1 0.378373 (and 0.651469 / 0.372309 on the second test set). | |
| | 5 | **VQA is weak-but-related** | the live VQA path answers broadly related content (e.g. "Grassland") rather than a crisp class. | |
| | 6 | **Optical-SAR returns a bare class index** | the live service returns `class_18`, not a human-readable label. | |
| | 7 | **Calibration made things worse** | ECE 0.013755 → 0.014929. Retained only because it is part of the frozen config. | |
| | 8 | **The VLM adapter is not accepted** | metrics usable (exact_match 0.963), but the artifact is acceptance-rejected; the deployed path uses the unadapted model. | |
| | 9 | **The router number is not a test result** | 0.965116 is validation, **ungated**, n = 86. The test split was **NOT RUN**. | |
| |
| ## 2. Router residuals (known misroutes) |
| |
| | # | Query | Behaviour | Note | |
| |---|---|---|---| |
| | 10 | *"What is the new runway?"* | reads `change`, not `vqa` | residual ambiguity — "new" triggers a change reading | |
| | 11 | *"How much built-up area was added?"* | reads `vqa` (under-trigger) | a change-style quantifier not caught by the router | |
| | 12 | *"What changed between the earlier and later image?"* with one asset | console **reads** `change`, **dispatches** `change_vqa` | intentional (asset-count-aware dispatch) but visually surprising — documented in [`ARCHITECTURE.md`](ARCHITECTURE.md) §4 | |
|
|
| ## 3. Evaluation gaps |
|
|
| | # | Gap | State | |
| |---|---|---| |
| | 13 | **No system-level end-to-end benchmark** | **NOT RUN — none exists.** No end-to-end accuracy is claimed. | |
| | 14 | **Router test split** | **NOT RUN** | |
| | 15 | **Benchmark adapters** | **NOT RUN** | |
| | 16 | **End-to-end latency benchmark** | **NOT RUN** (per-specialist latency only, incidentally) | |
| | 17 | **Cross-dataset generalisation** | **NOT RUN** — each specialist is evaluated on its own training-family split only | |
| | 18 | **Human evaluation** | **NOT RUN** | |
| | 19 | **Robustness / adversarial evaluation** | **NOT RUN** | |
| | 20 | **BigEarthNet label semantics** | the local subset is 100 % single-label vs the official 1–11 multi-label scheme, so metrics are **not comparable** to published numbers | |
| | 21 | **Reliability diagram** | the pre-scaling diagram is labelled as such; the calibrated curve is not plotted | |
|
|
| ## 4. Operational limitations |
|
|
| | # | Limitation | State | |
| |---|---|---| |
| | 22 | **B-07: transient tunnel gaps** | a request can hang or return 504. Patch prepared, **NOT deployed**. **OPEN** | |
| | 23 | **B-07 root shape** | in `auto` mode a tunnel timeout falls through to the forward path, burning `wake_timeout_s` (120 s) on a `302`; worst case ≈ 249 s. Measured. | |
| | 24 | **Cold start is tens of seconds** | Render free tier sleeps; the Codespace may be stopped. Documented, not hidden. | |
| | 25 | **B-02: `codespace_name` trailing newline** | cosmetic; the wake path strips it. **OPEN (cosmetic)** | |
| | 26 | **No database, auth, or queue** | stateless gateway by design — no persistence of runs or users. | |
| | 27 | **Deployment repos are private** | their links 404 for an outside audience. **BY DESIGN** | |
| | 28 | **`deploy/` in the monorepo is stale/untracked** | not the deployed source. Trap. | |
| |
| ## 5. Packaging and licensing |
| |
| | # | Limitation | State | |
| |---|---|---| |
| | 29 | **No `LICENSE` file** | none exists in the source repository. **OPEN** — a licence must be selected before public release of the code. | |
| | 30 | **Backbones are not redistributed** | fetched from the Hub at run time; their licences are their own. | |
| | 31 | **Artifact bloat** | `artifacts/` is ~3.7 GB, mostly reproducible caches, duplicate ZIPs and superseded checkpoints rather than released weights. | |
| |
| ## 6. Documentation caveats |
| |
| | # | Caveat | State | |
| |---|---|---| |
| | 32 | **`docs/FINAL_DELIVERY_REPORT.md` §6 is stale** | it still lists the bundled EO change pair as DEGRADED and B-01 as BLOCKED; both were resolved on 2026-09-25. | |
| | 33 | **The monorepo `README.md` was materially stale** | it described a hermetic frontend, an in-progress Render/Codespace, and a `/v1/*` contract. Superseded by this release's README. | |
| | 34 | **`hf/` docs were stale** | they asserted the project owns no weights and has no HF credentials. Both were false at release time. | |
| | 35 | **Anatomy plate image variant** | points at the 720×720 variant of an image the recorded run analysed at 730×730. Cosmetic. | |
| |
| ## 7. Explicit non-claims |
| |
| - **No claim of state-of-the-art performance** on any benchmark. |
| - **No claim of production readiness** for the model quality — the deployment runs, but the models |
| carry the limitations above. |
| - **No claim that the trained heads generalise** beyond their training-family test splits. |
| - **No claim that calibration improves confidence.** |
| - **No claim that the VLM adapter is accepted** for production use. |
| |
