| --- |
| language: en |
| license: other |
| library_name: pytorch |
| tags: |
| - remote-sensing |
| - satellite-imagery |
| - earth-observation |
| - change-detection |
| - visual-grounding |
| - image-captioning |
| - visual-question-answering |
| - optical-sar-fusion |
| - lora |
| - peft |
| pipeline_tag: image-to-text |
| config_hash: 78f1e3700da15aa1 |
| --- |
| |
| # Model Card — SatQuery AI |
|
|
| SatQuery AI answers natural-language questions about satellite imagery using a **router + specialists** |
| design. This card covers the **six trained artifacts** released by the project. It is deliberately |
| explicit about what is measured, what is not, and what was rejected. |
|
|
| > **The six trained artifacts are small modules on top of frozen, publicly-pinned backbones. No |
| > backbone weights are redistributed by this release** — they are fetched from the Hugging Face Hub at |
| > run time, pinned by revision. |
|
|
| Machine-readable identities (byte counts and sha256) are in |
| [`models/manifest.json`](models/manifest.json) and [`models/checksums.sha256`](models/checksums.sha256), |
| **generated by reading the files**. Where this card and the generated manifest disagree, the manifest |
| wins. |
|
|
| --- |
|
|
| ## 1. Artifacts in this release |
|
|
| | # | Task | Kind | File | Bytes | sha256 (first 16) | |
| |---|---|---|---|---|---| |
| | 1 | `change` | trained head | `head.pt` | 63,231,009 | `c5ef31277b67aa01` | |
| | 2 | `change_vqa` | trained head | `head.pt` | 5,822,809 | `cfae5e43b97ca930` | |
| | 3 | `optical_sar` | trained head | `head.pt` | 14,427,457 | `785815729a3a39fc` | |
| | 4 | `grounding` | trained head | `head.pt` | 12,639,041 | `93432f7034be91a8` | |
| | 5 | `router` | trained adapter | `adapter.pt` | 211,961 | `8527c3ed28a293e1` | |
| | 6 | `vlm` | LoRA adapter | `adapter_model.safetensors` | 34,798,048 | `07c76a75fa046248` | |
|
|
| Two of these hashes (`change_vqa`, `vlm`) **agree exactly** with hashes recorded independently at |
| promotion time — an external cross-check, not a self-consistency claim. |
|
|
| ## 2. Backbone dependencies (frozen, pinned by revision) |
|
|
| | Role | Repository | Revision | |
| |---|---|---| |
| | Router encoder | `sentence-transformers/all-MiniLM-L6-v2` | `1110a243fdf4` | |
| | VLM | `HuggingFaceTB/SmolVLM-500M-Instruct` | `a7da5b986cb5` | |
| | Grounding | `chendelong/RemoteCLIP` (`RemoteCLIP-ViT-B-32.pt`) | `bf1d8a3ccf2d` | |
| | Optical-SAR | `antofuller/CROMA` (`CROMA_base.pt`) | `0dd28e3d633b` | |
| | Change | STANet-style (ResNet-18 + PAM) | trained in-project | |
|
|
| ## 3. Intended use |
|
|
| - **Research and demonstration** of a modular, CPU-first remote-sensing QA system. |
| - **Routing and dispatch** of natural-language queries to the appropriate specialist. |
| - **Reproducible evaluation** of each specialist on its own documented split. |
|
|
| ## 4. Out-of-scope use |
|
|
| - **Safety-, legal- or life-critical decisions.** No accuracy, calibration or robustness guarantee is |
| offered for any high-stakes use. |
| - **Operational geospatial production** without independent validation. |
| - **Any use of the VLM adapter as a production model** — it is acceptance-rejected (§6). |
| - **Treating per-specialist metrics as system-level accuracy.** No end-to-end benchmark exists (§7). |
|
|
| ## 5. Measured performance |
|
|
| | Task | Metric | Value | Split / protocol | |
| |---|---|---|---| |
| | change | pooled IoU / macro IoU / pooled F1 | **0.8122 / 0.8457 / 0.8964** | LEVIR-CD-256 test, n = 2048 | |
| | grounding | mean best IoU / recall@0.5 | **0.2838 / 0.2198** canonical; **0.2566 / 0.1938** matched6 | VRSBench, n = 16159 | |
| | grounding | head-argmax / zero-shot baseline IoU | **0.1215 / 0.0972** | canonical | |
| | optical_sar | accuracy / macro F1 | **0.931 / 0.434161** | held-out test, n = 4000, 19 classes | |
| | change_vqa | accuracy / macro F1 | **0.697626 / 0.378373** (test); **0.651469 / 0.372309** (test2) | two test sets | |
| | vlm | exact_match / F1 | **0.963 / 0.96432** | frozen 1000-question subset | |
| | router | overall **ungated** accuracy | **0.965116** | val, n = 86 — **TEST NOT RUN** | |
| |
| **Every value is checked against its artifact** by `tools/verify_readme_metrics.py`. |
| |
| ### 5.1 Calibration — reported as a negative result |
| |
| Temperature scaling is enabled (T = 0.9772732, fit on val n = 16,441). **ECE worsened**: |
| 0.013755 → **0.014929**. It is retained because it is part of the frozen configuration, not because it |
| helped. |
| |
| ## 6. Acceptance status |
| |
| | Artifact | Metrics | Acceptance | |
| |---|---|---| |
| | change | VERIFIED | accepted (shipped) | |
| | grounding | measured (2 protocols) | shipped | |
| | optical_sar | measured | **ruling OPEN** | |
| | change_vqa | measured (2 test sets) | **ruling OPEN** | |
| | router | measured (val only) | shipped; test NOT RUN | |
| | **vlm** | usable (exact_match 0.963) | **ACCEPTANCE-REJECTED** | |
|
|
| **`USABLE_VERIFIED` ≠ `ACCEPTANCE-ACCEPTED`.** The VLM adapter works and is not promoted; the deployed |
| caption/VQA path uses the **unadapted** model. |
| |
| ## 7. Evaluation gaps (stated, not hidden) |
| |
| - **No system-level end-to-end benchmark exists.** None is claimed. |
| - **Router test split: NOT RUN.** |
| - **Benchmark adapters: NOT RUN.** |
| - **Cross-dataset generalisation: NOT RUN.** |
| - **Human and robustness evaluation: NOT RUN.** |
| |
| ## 8. Limitations |
| |
| - Grounding absolute IoU is low (0.28) and protocol-sensitive. |
| - Optical-SAR accuracy is carried by common classes (macro-F1 0.434161). |
| - The BigEarthNet local subset is **100 % single-label** vs the official 1–11 multi-label scheme, so |
| its metrics are **not comparable** to published numbers. |
| - The optical-SAR service returns a bare class index, not a label. |
| - Known router residuals exist (e.g. *"What is the new runway?"* reads `change`). |
| - No `LICENSE` file exists in the source repository. |
| |
| See [`docs/LIMITATIONS.md`](docs/LIMITATIONS.md) for the full catalogue. |
| |
| ## 9. Training summary |
| |
| Small modules on frozen backbones; seed 42; every artifact records the frozen config hash |
| `78f1e3700da15aa1`. Router and change/grounding/fusion heads train on CPU; the VLM LoRA adapter and |
| the change-VQA head were trained on **external GPUs** (the latter via a documented Kaggle run). Full |
| detail in [`docs/TRAINING.md`](docs/TRAINING.md). |
| |
| ## 10. Provenance and verification |
| |
| | Item | Location | |
| |---|---| |
| | Byte-exact manifest | `models/manifest.json` | |
| | Checksums | `models/checksums.sha256` | |
| | Metric verification tool | `tools/verify_readme_metrics.py` | |
| | Metric verification output | `tools/readme_metrics_report.txt` | |
| | Full documentation | `docs/` | |
| | Release manifest | `RELEASE_MANIFEST.md` | |
| |
| ## 11. Licence |
| |
| The project ships **no licence file**; a licence must be selected by the owner before public release |
| of the *code*. **Model weights carry the terms of their backbone licences** — consult each backbone's |
| Hugging Face page. Backbones are not redistributed here. |
|
|
| ## 12. Citation |
|
|
| If you use this work, cite the project repository: |
|
|
| ```bibtex |
| @misc{satquery_ai_2026, |
| title = {SatQuery AI: A Modular Router-and-Specialists System for Satellite Imagery Question Answering}, |
| author = {SatQuery AI}, |
| year = {2026}, |
| note = {Public release: https://github.com/Anish-lab-blip/SatQuery-AI} |
| } |
| ``` |
|
|
