release: add docs/DATASETS.md

docs/DATASETS.md

@@ -0,0 +1,162 @@
+# Datasets
+
+Every figure below was **measured from the data on disk**, not copied from a paper or a dataset
+schema. Where a corpus was only partially acquired, or where the local subset differs from the
+official release, that is stated — not smoothed over.
+
+**Status tags:** `ACQUIRED` · `MEASURED` · `PARTIAL` · `NOT DOWNLOADED` · `REJECTED`.
+
+---
+
+## 1. Summary
+
+| Task | Dataset | Role | Local state |
+|---|---|---|---|
+| `change` | **LEVIR-CD-256** | train / val / test | ACQUIRED — split 7120 / 1024 / 2048 |
+| `change_vqa` | **CDVQA** (+ **SECOND** imagery) | train / val / test | annotations + imagery ACQUIRED; name-verified 2,968/2,968 |
+| `grounding` | **VRSBench** | eval (16,159 records) | ACQUIRED — boxes normalised 0–100 |
+| `optical_sar` | **BigEarthNet** (CLC-19) | train / held-out test | PARTIAL — 28,000-patch local subset |
+| `vlm` | **BigEarthNet** (instruction pairs) | LoRA adaptation | PARTIAL — same 28k subset |
+| — | SECOND (SCD) | CDVQA imagery source | ACQUIRED, extracted to `data/cdvqa/{im1,im2,label1,label2}/` |
+
+## 2. LEVIR-CD-256 — change detection
+
+The standard change-detection benchmark, 256×256 tiles. Split used (from `configs/base.yaml`):
+
+| Split | Count |
+|---|---|
+| train | 7,120 |
+| val | 1,024 |
+| test | **2,048** |
+
+**Measured test-split detail** (`artifacts/change/eval_test/eval_result.json`):
+
+| Property | Value |
+|---|---|
+| n | 2,048 |
+| images containing change | **935** |
+| mean change fraction | **0.0509** (≈ 5 % of pixels) |
+| threshold | 0.50 |
+| tile size | 256 |
+| device (eval) | cuda |
+| total pixels scored | 134,217,728 |
+
+The 5 % change fraction is why **pooled** and **macro** metrics are both reported: with a strong
+class imbalance, pooled IoU (0.8122) and macro IoU (0.8457) answer different questions. The
+per-pixel confusion counts (tp 5,978,997 / fp 523,658 / fn 858,407 / tn 126,856,666) are stored so
+any metric can be recomputed rather than trusted.
+
+## 3. CDVQA (+ SECOND) — change-VQA
+
+**The CDVQA repository publishes annotations only — no imagery.** The imagery is publicly available
+as **SECOND (SCD)**. This project acquired SECOND, **name-verified it (2,968 / 2,968 MATCH)**,
+extracted it to `data/cdvqa/{im1,im2,label1,label2}/`, and verified a CDVQA example loads against it
+end-to-end.
+
+**Measured corpus figures:**
+
+| Property | Value |
+|---|---|
+| Val images | 400 |
+| Val questions | **16,441** |
+| Test questions | **39,686** |
+| Test2 questions | second held-out set (accuracy 0.651469) |
+
+**Temporal and label semantics — established from evidence, with honest uncertainty:**
+
+| Mapping | State |
+|---|---|
+| `label1` = pre, `label2` = post | **established** |
+| `im1` = pre, `im2` = post | **supported, not proven** |
+
+**Corpus root:** `data/cdvqa` — the directory that *contains* `annotations/`. Passing
+`data/cdvqa/annotations` is **rejected** by `_resolve_root` (verified by execution: it raises
+`no 'annotations' directory under data\cdvqa\annotations`). The correct call is
+`load_cdvqa('data/cdvqa')`.
+
+> **Not established: trainability of the raw loader.** The adapter is pair-aware and
+> `require_images=True` succeeds, but the raw corpus has no training loop of its own — the shipped
+> `change_vqa` head trains on **cached features**, not on the raw loader. See
+> [`TRAINING.md`](TRAINING.md).
+
+## 4. VRSBench — grounding evaluation
+
+The grounding head is evaluated on VRSBench. **Measured: 16,159 eval records.**
+
+VRSBench stores boxes **normalised to 0–100**; this project stores boxes **normalised to 0–1**. The
+conversion is declared explicitly as `grounding.benchmark_box_scale: 100.0` so it cannot be applied
+twice or forgotten.
+
+Because the box convention is a common source of silent error, grounding is reported under **two
+protocols** (canonical and matched6) and **two decode variants** — see
+[`BENCHMARKS.md`](BENCHMARKS.md) §1.
+
+## 5. BigEarthNet — optical-SAR fusion and VLM adaptation
+
+BigEarthNet is used in two places: as the **label space and evaluation benchmark** for the
+optical-SAR fusion head, and as the **instruction-pair source** for the SmolVLM LoRA adaptation.
+
+**Measured local subset:**
+
+| Property | Value |
+|---|---|
+| S2 patches (local subset) | **28,000** |
+| tiles | 98 |
+| bands per patch | 12 |
+| full official corpus | 480,038 patches |
+| label space | **19 CLC classes** |
+
+### 5.1 Label-semantics caveat (important)
+
+> **The local 28k subset is 100 % single-label, against the official 1–11 multi-label scheme.**
+> Metrics computed on this subset are therefore **not comparable** to published multi-label
+> BigEarthNet numbers. Any statement of the form "BigEarthNet mAP = X" is **false** for this subset.
+
+### 5.2 Fusion evaluation detail
+
+**Measured** (`artifacts/optical_sar/fusion_head_production_v001/pre_registered_115_metric.json`):
+
+| Property | Value |
+|---|---|
+| split | test |
+| n scored | **4,000** |
+| classes in label space | 19 |
+| classes present in the scored split | 14 |
+| classes absent | 5 |
+| macro-F1 denominator | **all 19 classes (absent classes contribute 0.0)** |
+| accuracy | 0.931 |
+| macro F1 | 0.434161 |
+| deciding statistic | **False** |
+
+The macro-F1 denominator is recorded explicitly so a reader can see that absent classes drag the
+macro score down by construction. This is why accuracy (0.931) and macro-F1 (0.434161) must be read
+together.
+
+### 5.3 The BigEarthNet data-format contradiction
+
+> **PARTIALLY — reported, not silently resolved.** The BigEarthNet data format contradicts the
+> original plan. This was reported rather than quietly patched, because silently changing the
+> preprocessing would move the frozen config hash. See
+> [`RESEARCH_NOTES.md`](RESEARCH_NOTES.md) for the full record.
+
+The BigEarthNet documentation — its uses, mentions, or endorsements — does **not** specify a
+percentile stretch. This project nevertheless applies percentile normalisation for optical inputs
+(2/98) to match the CROMA contract. That is a deliberate, documented choice, not an upstream fact.
+
+## 6. Data hygiene and leakage controls
+
+- **Splits are by group, never by example**, for the router: template / hard-negative families are
+  kept whole, and hard-negative families are placed in the **test** split so their accuracy measures
+  generalisation rather than memorisation.
+- **Leakage split key: `scene_id`** (declared in `configs/base.yaml`).
+- **Immutable public test: true.** `hidden_data_access: false`. The evaluation config forbids
+  touching hidden data.
+
+## 7. What is NOT available
+
+| Corpus | State |
+|---|---|
+| BigEarthNet S1+S2 full corpus | **NOT DOWNLOADED** (only the 28k S2 subset is local) |
+| BigEarthNet multi-label (reBEN) results | **not produced** — the local subset is single-label |
+| Cross-dataset generalisation sets | **not used** |
+| Any private / hidden evaluation data | **not accessed** (`hidden_data_access: false`) |