| # Benchmarks |
|
|
| **Every number here is artifact-backed.** Where a metric exists, its source file is named. Where a |
| benchmark does **not** exist, that is stated explicitly and tagged `NOT RUN` — an absent number is |
| never silently omitted or replaced with an estimate. |
|
|
| **Status tags:** `VERIFIED` · `MEASURED` · `NOT RUN` · `OPEN` · `REJECTED`. |
|
|
| --- |
|
|
| ## 1. Headline table |
|
|
| | Capability | Metric | Value | Split / protocol | Status | |
| |---|---|---|---|---| |
| | Change detection | pooled IoU | **0.8122** | LEVIR-CD-256 test, n = 2048, thr 0.50 | **VERIFIED** | |
| | Change detection | macro IoU | **0.8457** | same | **VERIFIED** | |
| | Change detection | pooled F1 | **0.8964** | same | **VERIFIED** | |
| | Grounding | mean best IoU | **0.2838** / **0.2566** | VRSBench n = 16159, canonical / matched6 | MEASURED (2 protocols) | |
| | Grounding | recall@0.5 | **0.2198** / **0.1938** | same | MEASURED (2 protocols) | |
| | Grounding | head-argmax IoU | **0.1215** | canonical | MEASURED | |
| | Grounding | zero-shot baseline IoU | **0.0972** | canonical | MEASURED (baseline) | |
| | Optical-SAR fusion | accuracy | **0.931** | held-out test n = 4000, 19 classes | MEASURED, ruling **OPEN** | |
| | Optical-SAR fusion | macro F1 | **0.434161** | same | MEASURED, ruling **OPEN** | |
| | Change-VQA | accuracy | **0.697626** | test n = 39686 | MEASURED, ruling **OPEN** | |
| | Change-VQA | macro F1 | **0.378373** | same | MEASURED, ruling **OPEN** | |
| | Change-VQA | accuracy (2nd set) | **0.651469** | test2 | MEASURED, ruling **OPEN** | |
| | Change-VQA | macro F1 (2nd set) | **0.372309** | test2 | MEASURED, ruling **OPEN** | |
| | VLM (adapted) | exact_match | **0.963** | frozen 1000-question subset | MEASURED, **ACCEPTANCE-REJECTED** | |
| | VLM (adapted) | F1 | **0.96432** | same | MEASURED, **ACCEPTANCE-REJECTED** | |
| | Router | overall **ungated** accuracy | **0.965116** | val n = 86, corpus-limited | MEASURED — **TEST NOT RUN** | |
| | Calibration | ECE before / after | **0.013755 → 0.014929** | val n = 16441, T = 0.9773 | MEASURED — **worse** | |
| |
| ### Source artifacts |
| |
| | Metric family | Artifact | |
| |---|---| |
| | change | `artifacts/change/eval_test/eval_result.json` | |
| | grounding | `artifacts/grounding/remoteclip_grounding_v001/eval_result_canonical.json`, `…_matched6.json` | |
| | optical-SAR | `artifacts/optical_sar/fusion_head_production_v001/pre_registered_115_metric.json` | |
| | change-VQA | `artifacts/change_vqa/run/PROMOTION.json` | |
| | router | `artifacts/router/threshold_sweep_val.json` | |
| | calibration | `artifacts/calibration_v001.json` | |
| | VLM | `artifacts/vlm/phase6_closure.json` | |
|
|
| > All 20 values in the table above are checked against these files by |
| > [`../tools/verify_readme_metrics.py`](../tools/verify_readme_metrics.py). Its output |
| > (`ALL CLAIMS VERIFIED`) is committed as `../tools/readme_metrics_report.txt`. |
|
|
| ## 2. Rules this table follows |
|
|
| 1. **Two protocols are never collapsed.** Grounding is reported under *both* the canonical and |
| matched6 protocols. Quoting 0.2838 alone would be selective. |
| 2. **Two test sets are never collapsed.** Change-VQA is reported on `test` **and** `test2`. |
| 3. **accuracy never travels without macro-F1.** For imbalanced multi-class heads (optical-SAR, |
| change-VQA) the macro-F1 is reported alongside accuracy, always. |
| 4. **Validation is not test.** The router number is labelled "overall **ungated** accuracy", val, |
| n = 86. It is not a test result. |
| 5. **A negative result stays negative.** Calibration ECE worsened and is shown worsening. |
| 6. **USABLE ≠ ACCEPTED.** The VLM metrics are real; the artifact is nevertheless acceptance-rejected. |
|
|
| ## 3. What is NOT benchmarked |
|
|
| | Benchmark | Status | Note | |
| |---|---|---| |
| | **System-level end-to-end accuracy** | **NOT RUN — none exists** | There is no measured end-to-end benchmark of the full router→specialist→envelope pipeline. No such number is claimed anywhere. | |
| | **Router test split** | **NOT RUN** | Only the validation split (n = 86) was scored. | |
| | **Benchmark adapters** | **NOT RUN** | Adapter-based benchmark runs were not executed. | |
| | **Efficiency / latency benchmark** | not systematically measured | Per-specialist latency is recorded incidentally in artifacts (e.g. grounding `latency_ms_per_image` 2.205 ms for the head), but there is no end-to-end latency benchmark. | |
| | **Cross-dataset generalisation** | **NOT RUN** | Each specialist is evaluated only on its own training-family test split. | |
|
|
| ## 4. Live validation (behavioural, not accuracy) |
|
|
| Accuracy is separate from **behavioural** validation. The deployed stack was driven end-to-end in a |
| headed browser, one upload per case, with per-case screenshots and recorded run ids: |
|
|
| | Property | Result | |
| |---|---| |
| | Independent full passes | **3** | |
| | Cases per pass | 8 (6 regression + 2 router-defect) | |
| | Passes at 8/8 | **3 of 3** | |
| | Live runs executed | **24** | |
| | Correct dispatches | **24** | |
| | Mock-node contamination | **0** on every live run | |
| | Trace fill | **94.4444 %** on every live run | |
| | Frontend regression suite | **106 passed** (`tests/unit/test_frontend_live_wiring.py`) | |
|
|
| Every pass produced **fresh run identifiers** — no run id is shared between passes. This is |
| behavioural evidence that the pipeline runs and routes correctly; it is **not** an accuracy claim. |
|
|
| ## 5. How to reproduce |
|
|
| ```bash |
| # verify every README/benchmark number against its artifact |
| python release/tools/verify_readme_metrics.py |
| ``` |
|
|
| The script resolves nested artifact keys (including keys that themselves contain dots, such as the |
| `recall` dict keyed `"0.10"/"0.25"/"0.50"`) and compares each value at the precision printed in the |
| README. It exits non-zero if any claim fails. See [`REPRODUCIBILITY.md`](REPRODUCIBILITY.md) for the |
| full reproduction guide. |
|
|
