release: add MODEL_CARD.md

MODEL_CARD.md

@@ -0,0 +1,164 @@
+---
+language: en
+license: other
+library_name: pytorch
+tags:
+  - remote-sensing
+  - satellite-imagery
+  - earth-observation
+  - change-detection
+  - visual-grounding
+  - image-captioning
+  - visual-question-answering
+  - optical-sar-fusion
+  - lora
+  - peft
+pipeline_tag: image-to-text
+config_hash: 78f1e3700da15aa1
+---
+
+# Model Card — SatQuery AI
+
+SatQuery AI answers natural-language questions about satellite imagery using a **router + specialists**
+design. This card covers the **six trained artifacts** released by the project. It is deliberately
+explicit about what is measured, what is not, and what was rejected.
+
+> **The six trained artifacts are small modules on top of frozen, publicly-pinned backbones. No
+> backbone weights are redistributed by this release** — they are fetched from the Hugging Face Hub at
+> run time, pinned by revision.
+
+Machine-readable identities (byte counts and sha256) are in
+[`models/manifest.json`](models/manifest.json) and [`models/checksums.sha256`](models/checksums.sha256),
+**generated by reading the files**. Where this card and the generated manifest disagree, the manifest
+wins.
+
+---
+
+## 1. Artifacts in this release
+
+| # | Task | Kind | File | Bytes | sha256 (first 16) |
+|---|---|---|---|---|---|
+| 1 | `change` | trained head | `head.pt` | 63,231,009 | `c5ef31277b67aa01` |
+| 2 | `change_vqa` | trained head | `head.pt` | 5,822,809 | `cfae5e43b97ca930` |
+| 3 | `optical_sar` | trained head | `head.pt` | 14,427,457 | `785815729a3a39fc` |
+| 4 | `grounding` | trained head | `head.pt` | 12,639,041 | `93432f7034be91a8` |
+| 5 | `router` | trained adapter | `adapter.pt` | 211,961 | `8527c3ed28a293e1` |
+| 6 | `vlm` | LoRA adapter | `adapter_model.safetensors` | 34,798,048 | `07c76a75fa046248` |
+
+Two of these hashes (`change_vqa`, `vlm`) **agree exactly** with hashes recorded independently at
+promotion time — an external cross-check, not a self-consistency claim.
+
+## 2. Backbone dependencies (frozen, pinned by revision)
+
+| Role | Repository | Revision |
+|---|---|---|
+| Router encoder | `sentence-transformers/all-MiniLM-L6-v2` | `1110a243fdf4` |
+| VLM | `HuggingFaceTB/SmolVLM-500M-Instruct` | `a7da5b986cb5` |
+| Grounding | `chendelong/RemoteCLIP` (`RemoteCLIP-ViT-B-32.pt`) | `bf1d8a3ccf2d` |
+| Optical-SAR | `antofuller/CROMA` (`CROMA_base.pt`) | `0dd28e3d633b` |
+| Change | STANet-style (ResNet-18 + PAM) | trained in-project |
+
+## 3. Intended use
+
+- **Research and demonstration** of a modular, CPU-first remote-sensing QA system.
+- **Routing and dispatch** of natural-language queries to the appropriate specialist.
+- **Reproducible evaluation** of each specialist on its own documented split.
+
+## 4. Out-of-scope use
+
+- **Safety-, legal- or life-critical decisions.** No accuracy, calibration or robustness guarantee is
+  offered for any high-stakes use.
+- **Operational geospatial production** without independent validation.
+- **Any use of the VLM adapter as a production model** — it is acceptance-rejected (§6).
+- **Treating per-specialist metrics as system-level accuracy.** No end-to-end benchmark exists (§7).
+
+## 5. Measured performance
+
+| Task | Metric | Value | Split / protocol |
+|---|---|---|---|
+| change | pooled IoU / macro IoU / pooled F1 | **0.8122 / 0.8457 / 0.8964** | LEVIR-CD-256 test, n = 2048 |
+| grounding | mean best IoU / recall@0.5 | **0.2838 / 0.2198** canonical; **0.2566 / 0.1938** matched6 | VRSBench, n = 16159 |
+| grounding | head-argmax / zero-shot baseline IoU | **0.1215 / 0.0972** | canonical |
+| optical_sar | accuracy / macro F1 | **0.931 / 0.434161** | held-out test, n = 4000, 19 classes |
+| change_vqa | accuracy / macro F1 | **0.697626 / 0.378373** (test); **0.651469 / 0.372309** (test2) | two test sets |
+| vlm | exact_match / F1 | **0.963 / 0.96432** | frozen 1000-question subset |
+| router | overall **ungated** accuracy | **0.965116** | val, n = 86 — **TEST NOT RUN** |
+
+**Every value is checked against its artifact** by `tools/verify_readme_metrics.py`.
+
+### 5.1 Calibration — reported as a negative result
+
+Temperature scaling is enabled (T = 0.9772732, fit on val n = 16,441). **ECE worsened**:
+0.013755 → **0.014929**. It is retained because it is part of the frozen configuration, not because it
+helped.
+
+## 6. Acceptance status
+
+| Artifact | Metrics | Acceptance |
+|---|---|---|
+| change | VERIFIED | accepted (shipped) |
+| grounding | measured (2 protocols) | shipped |
+| optical_sar | measured | **ruling OPEN** |
+| change_vqa | measured (2 test sets) | **ruling OPEN** |
+| router | measured (val only) | shipped; test NOT RUN |
+| **vlm** | usable (exact_match 0.963) | **ACCEPTANCE-REJECTED** |
+
+**`USABLE_VERIFIED` ≠ `ACCEPTANCE-ACCEPTED`.** The VLM adapter works and is not promoted; the deployed
+caption/VQA path uses the **unadapted** model.
+
+## 7. Evaluation gaps (stated, not hidden)
+
+- **No system-level end-to-end benchmark exists.** None is claimed.
+- **Router test split: NOT RUN.**
+- **Benchmark adapters: NOT RUN.**
+- **Cross-dataset generalisation: NOT RUN.**
+- **Human and robustness evaluation: NOT RUN.**
+
+## 8. Limitations
+
+- Grounding absolute IoU is low (0.28) and protocol-sensitive.
+- Optical-SAR accuracy is carried by common classes (macro-F1 0.434161).
+- The BigEarthNet local subset is **100 % single-label** vs the official 1–11 multi-label scheme, so
+  its metrics are **not comparable** to published numbers.
+- The optical-SAR service returns a bare class index, not a label.
+- Known router residuals exist (e.g. *"What is the new runway?"* reads `change`).
+- No `LICENSE` file exists in the source repository.
+
+See [`docs/LIMITATIONS.md`](docs/LIMITATIONS.md) for the full catalogue.
+
+## 9. Training summary
+
+Small modules on frozen backbones; seed 42; every artifact records the frozen config hash
+`78f1e3700da15aa1`. Router and change/grounding/fusion heads train on CPU; the VLM LoRA adapter and
+the change-VQA head were trained on **external GPUs** (the latter via a documented Kaggle run). Full
+detail in [`docs/TRAINING.md`](docs/TRAINING.md).
+
+## 10. Provenance and verification
+
+| Item | Location |
+|---|---|
+| Byte-exact manifest | `models/manifest.json` |
+| Checksums | `models/checksums.sha256` |
+| Metric verification tool | `tools/verify_readme_metrics.py` |
+| Metric verification output | `tools/readme_metrics_report.txt` |
+| Full documentation | `docs/` |
+| Release manifest | `RELEASE_MANIFEST.md` |
+
+## 11. Licence
+
+The project ships **no licence file**; a licence must be selected by the owner before public release
+of the *code*. **Model weights carry the terms of their backbone licences** — consult each backbone's
+Hugging Face page. Backbones are not redistributed here.
+
+## 12. Citation
+
+If you use this work, cite the project repository:
+
+```bibtex
+@misc{satquery_ai_2026,
+  title  = {SatQuery AI: A Modular Router-and-Specialists System for Satellite Imagery Question Answering},
+  author = {SatQuery AI},
+  year   = {2026},
+  note   = {Public release: https://github.com/Anish-lab-blip/SatQuery-AI}
+}
+```