# Benchmarks

**Every number here is artifact-backed.** Where a metric exists, its source file is named. Where a
benchmark does **not** exist, that is stated explicitly and tagged `NOT RUN` — an absent number is
never silently omitted or replaced with an estimate.

**Status tags:** `VERIFIED` · `MEASURED` · `NOT RUN` · `OPEN` · `REJECTED`.

---

## 1. Headline table

| Capability | Metric | Value | Split / protocol | Status |
|---|---|---|---|---|
| Change detection | pooled IoU | **0.8122** | LEVIR-CD-256 test, n = 2048, thr 0.50 | **VERIFIED** |
| Change detection | macro IoU | **0.8457** | same | **VERIFIED** |
| Change detection | pooled F1 | **0.8964** | same | **VERIFIED** |
| Grounding | mean best IoU | **0.2838** / **0.2566** | VRSBench n = 16159, canonical / matched6 | MEASURED (2 protocols) |
| Grounding | recall@0.5 | **0.2198** / **0.1938** | same | MEASURED (2 protocols) |
| Grounding | head-argmax IoU | **0.1215** | canonical | MEASURED |
| Grounding | zero-shot baseline IoU | **0.0972** | canonical | MEASURED (baseline) |
| Optical-SAR fusion | accuracy | **0.931** | held-out test n = 4000, 19 classes | MEASURED, ruling **OPEN** |
| Optical-SAR fusion | macro F1 | **0.434161** | same | MEASURED, ruling **OPEN** |
| Change-VQA | accuracy | **0.697626** | test n = 39686 | MEASURED, ruling **OPEN** |
| Change-VQA | macro F1 | **0.378373** | same | MEASURED, ruling **OPEN** |
| Change-VQA | accuracy (2nd set) | **0.651469** | test2 | MEASURED, ruling **OPEN** |
| Change-VQA | macro F1 (2nd set) | **0.372309** | test2 | MEASURED, ruling **OPEN** |
| VLM (adapted) | exact_match | **0.963** | frozen 1000-question subset | MEASURED, **ACCEPTANCE-REJECTED** |
| VLM (adapted) | F1 | **0.96432** | same | MEASURED, **ACCEPTANCE-REJECTED** |
| Router | overall **ungated** accuracy | **0.965116** | val n = 86, corpus-limited | MEASURED — **TEST NOT RUN** |
| Calibration | ECE before / after | **0.013755 → 0.014929** | val n = 16441, T = 0.9773 | MEASURED — **worse** |

### Source artifacts

| Metric family | Artifact |
|---|---|
| change | `artifacts/change/eval_test/eval_result.json` |
| grounding | `artifacts/grounding/remoteclip_grounding_v001/eval_result_canonical.json`, `…_matched6.json` |
| optical-SAR | `artifacts/optical_sar/fusion_head_production_v001/pre_registered_115_metric.json` |
| change-VQA | `artifacts/change_vqa/run/PROMOTION.json` |
| router | `artifacts/router/threshold_sweep_val.json` |
| calibration | `artifacts/calibration_v001.json` |
| VLM | `artifacts/vlm/phase6_closure.json` |

> All 20 values in the table above are checked against these files by
> [`../tools/verify_readme_metrics.py`](../tools/verify_readme_metrics.py). Its output
> (`ALL CLAIMS VERIFIED`) is committed as `../tools/readme_metrics_report.txt`.

## 2. Rules this table follows

1. **Two protocols are never collapsed.** Grounding is reported under *both* the canonical and
   matched6 protocols. Quoting 0.2838 alone would be selective.
2. **Two test sets are never collapsed.** Change-VQA is reported on `test` **and** `test2`.
3. **accuracy never travels without macro-F1.** For imbalanced multi-class heads (optical-SAR,
   change-VQA) the macro-F1 is reported alongside accuracy, always.
4. **Validation is not test.** The router number is labelled "overall **ungated** accuracy", val,
   n = 86. It is not a test result.
5. **A negative result stays negative.** Calibration ECE worsened and is shown worsening.
6. **USABLE ≠ ACCEPTED.** The VLM metrics are real; the artifact is nevertheless acceptance-rejected.

## 3. What is NOT benchmarked

| Benchmark | Status | Note |
|---|---|---|
| **System-level end-to-end accuracy** | **NOT RUN — none exists** | There is no measured end-to-end benchmark of the full router→specialist→envelope pipeline. No such number is claimed anywhere. |
| **Router test split** | **NOT RUN** | Only the validation split (n = 86) was scored. |
| **Benchmark adapters** | **NOT RUN** | Adapter-based benchmark runs were not executed. |
| **Efficiency / latency benchmark** | not systematically measured | Per-specialist latency is recorded incidentally in artifacts (e.g. grounding `latency_ms_per_image` 2.205 ms for the head), but there is no end-to-end latency benchmark. |
| **Cross-dataset generalisation** | **NOT RUN** | Each specialist is evaluated only on its own training-family test split. |

## 4. Live validation (behavioural, not accuracy)

Accuracy is separate from **behavioural** validation. The deployed stack was driven end-to-end in a
headed browser, one upload per case, with per-case screenshots and recorded run ids:

| Property | Result |
|---|---|
| Independent full passes | **3** |
| Cases per pass | 8 (6 regression + 2 router-defect) |
| Passes at 8/8 | **3 of 3** |
| Live runs executed | **24** |
| Correct dispatches | **24** |
| Mock-node contamination | **0** on every live run |
| Trace fill | **94.4444 %** on every live run |
| Frontend regression suite | **106 passed** (`tests/unit/test_frontend_live_wiring.py`) |

Every pass produced **fresh run identifiers** — no run id is shared between passes. This is
behavioural evidence that the pipeline runs and routes correctly; it is **not** an accuracy claim.

## 5. How to reproduce

```bash
# verify every README/benchmark number against its artifact
python release/tools/verify_readme_metrics.py
```

The script resolves nested artifact keys (including keys that themselves contain dots, such as the
`recall` dict keyed `"0.10"/"0.25"/"0.50"`) and compares each value at the precision printed in the
README. It exits non-zero if any claim fails. See [`REPRODUCIBILITY.md`](REPRODUCIBILITY.md) for the
full reproduction guide.