Hugging Face's logo

thundercode
/
SatQuery

Image-to-Text
PyTorch
Safetensors
PEFT
English
remote-sensing
satellite-imagery
earth-observation
change-detection
visual-grounding
image-captioning
visual-question-answering
optical-sar-fusion
sar
multimodal
lora
Model card Files
xet
 Community
SatQuery / docs /BENCHMARKS.md
thundercode's picture
thundercode
release: add docs/BENCHMARKS.md
ad9cc6a verified
preview code
|
Raw History Blame
5.72 kB

Benchmarks

Every number here is artifact-backed. Where a metric exists, its source file is named. Where a benchmark does not exist, that is stated explicitly and tagged NOT RUN — an absent number is never silently omitted or replaced with an estimate.

Status tags: VERIFIED · MEASURED · NOT RUN · OPEN · REJECTED.

1. Headline table

Capability Metric Value Split / protocol Status
Change detection pooled IoU 0.8122 LEVIR-CD-256 test, n = 2048, thr 0.50 VERIFIED
Change detection macro IoU 0.8457 same VERIFIED
Change detection pooled F1 0.8964 same VERIFIED
Grounding mean best IoU 0.2838 / 0.2566 VRSBench n = 16159, canonical / matched6 MEASURED (2 protocols)
Grounding recall@0.5 0.2198 / 0.1938 same MEASURED (2 protocols)
Grounding head-argmax IoU 0.1215 canonical MEASURED
Grounding zero-shot baseline IoU 0.0972 canonical MEASURED (baseline)
Optical-SAR fusion accuracy 0.931 held-out test n = 4000, 19 classes MEASURED, ruling OPEN
Optical-SAR fusion macro F1 0.434161 same MEASURED, ruling OPEN
Change-VQA accuracy 0.697626 test n = 39686 MEASURED, ruling OPEN
Change-VQA macro F1 0.378373 same MEASURED, ruling OPEN
Change-VQA accuracy (2nd set) 0.651469 test2 MEASURED, ruling OPEN
Change-VQA macro F1 (2nd set) 0.372309 test2 MEASURED, ruling OPEN
VLM (adapted) exact_match 0.963 frozen 1000-question subset MEASURED, ACCEPTANCE-REJECTED
VLM (adapted) F1 0.96432 same MEASURED, ACCEPTANCE-REJECTED
Router overall ungated accuracy 0.965116 val n = 86, corpus-limited MEASURED — TEST NOT RUN
Calibration ECE before / after 0.013755 → 0.014929 val n = 16441, T = 0.9773 MEASURED — worse

Source artifacts

Metric family Artifact
change artifacts/change/eval_test/eval_result.json
grounding artifacts/grounding/remoteclip_grounding_v001/eval_result_canonical.json, …_matched6.json
optical-SAR artifacts/optical_sar/fusion_head_production_v001/pre_registered_115_metric.json
change-VQA artifacts/change_vqa/run/PROMOTION.json
router artifacts/router/threshold_sweep_val.json
calibration artifacts/calibration_v001.json
VLM artifacts/vlm/phase6_closure.json

All 20 values in the table above are checked against these files by ../tools/verify_readme_metrics.py. Its output (ALL CLAIMS VERIFIED) is committed as ../tools/readme_metrics_report.txt.

2. Rules this table follows

  1. Two protocols are never collapsed. Grounding is reported under both the canonical and matched6 protocols. Quoting 0.2838 alone would be selective.
  2. Two test sets are never collapsed. Change-VQA is reported on test and test2.
  3. accuracy never travels without macro-F1. For imbalanced multi-class heads (optical-SAR, change-VQA) the macro-F1 is reported alongside accuracy, always.
  4. Validation is not test. The router number is labelled "overall ungated accuracy", val, n = 86. It is not a test result.
  5. A negative result stays negative. Calibration ECE worsened and is shown worsening.
  6. USABLE ≠ ACCEPTED. The VLM metrics are real; the artifact is nevertheless acceptance-rejected.

3. What is NOT benchmarked

Benchmark Status Note
System-level end-to-end accuracy NOT RUN — none exists There is no measured end-to-end benchmark of the full router→specialist→envelope pipeline. No such number is claimed anywhere.
Router test split NOT RUN Only the validation split (n = 86) was scored.
Benchmark adapters NOT RUN Adapter-based benchmark runs were not executed.
Efficiency / latency benchmark not systematically measured Per-specialist latency is recorded incidentally in artifacts (e.g. grounding latency_ms_per_image 2.205 ms for the head), but there is no end-to-end latency benchmark.
Cross-dataset generalisation NOT RUN Each specialist is evaluated only on its own training-family test split.

4. Live validation (behavioural, not accuracy)

Accuracy is separate from behavioural validation. The deployed stack was driven end-to-end in a headed browser, one upload per case, with per-case screenshots and recorded run ids:

Property Result
Independent full passes 3
Cases per pass 8 (6 regression + 2 router-defect)
Passes at 8/8 3 of 3
Live runs executed 24
Correct dispatches 24
Mock-node contamination 0 on every live run
Trace fill 94.4444 % on every live run
Frontend regression suite 106 passed (tests/unit/test_frontend_live_wiring.py)

Every pass produced fresh run identifiers — no run id is shared between passes. This is behavioural evidence that the pipeline runs and routes correctly; it is not an accuracy claim.

5. How to reproduce 

# verify every README/benchmark number against its artifact
python release/tools/verify_readme_metrics.py

The script resolves nested artifact keys (including keys that themselves contain dots, such as the recall dict keyed "0.10"/"0.25"/"0.50") and compares each value at the precision printed in the README. It exits non-zero if any claim fails. See REPRODUCIBILITY.md for the full reproduction guide.