# Models — deep reference **Status tags used on every substantive claim:** `IMPLEMENTED` · `VERIFIED` · `MEASURED` · `ATTEMPTED` · `NOT RUN` · `BLOCKED` · `DEFERRED` · `REJECTED` · `OPEN` · `RESOLVED` · `CLOSED`. SatQuery AI trains **six** artifacts. Four are task heads, one is a router adapter, one is a LoRA adapter. Every backbone is **frozen** and publicly pinned by revision in `configs/base.yaml` — the project trains small modules on top of frozen encoders, not end-to-end networks. No backbone is fine-tuned; no backbone weight is redistributed. > **This file is the human-readable companion to the machine-generated > [`../models/manifest.json`](../models/manifest.json) and > [`../models/checksums.sha256`](../models/checksums.sha256).** > Where the two disagree, **the generated manifest wins** — it is computed by reading the files > (`release/tools/generate_model_manifest.py`), this document is written by hand. **The single most important rule in this document: do not fabricate.** Every number below is copied from a file that was read, and every number names the file and (where the source is a JSON artifact) the exact key path. Where a fact is not established, this document says so explicitly rather than estimating. --- ## Table of contents 1. [The six trained artifacts](#1-the-six-trained-artifacts) 2. [Frozen backbones — pinned, never retrained](#2-frozen-backbones--pinned-never-retrained) 3. [Per-artifact deep reference](#3-per-artifact-deep-reference) - 3.0 [Enforced configuration invariants, with arithmetic](#30-enforced-configuration-invariants-with-arithmetic) - 3.1 [`change` — STANet-style Siamese change detector](#31-change--stanet-style-siamese-change-detector) - 3.2 [`change_vqa` — change question answering head](#32-change_vqa--change-question-answering-head) - 3.3 [`optical_sar` — CROMA-base fusion head](#33-optical_sar--croma-base-fusion-head) - 3.4 [`grounding` — RemoteCLIP grounding head](#34-grounding--remoteclip-grounding-head) - 3.5 [`router` — intent adapter over frozen MiniLM](#35-router--intent-adapter-over-frozen-minilm) - 3.6 [`vlm` — SmolVLM LoRA adapter (USABLE_VERIFIED, ACCEPTANCE-REJECTED)](#36-vlm--smolvlm-lora-adapter-usable_verified-acceptance-rejected) 4. [Rejected, deferred and open model decisions](#4-rejected-deferred-and-open-model-decisions) 5. [Calibration — a measured negative result](#5-calibration--a-measured-negative-result) 6. [Distribution and licensing](#6-distribution-and-licensing) 7. [Status summary — what is NOT established](#7-status-summary--what-is-not-established) 8. [Evidence index](#8-evidence-index) --- ## 1. The six trained artifacts All six are published on the Hugging Face Hub under `thundercode/SatQuery`, one directory per task. The table below is reproduced from [`../models/manifest.json`](../models/manifest.json) (`artifacts[*]`), cross-checked against [`../models/checksums.sha256`](../models/checksums.sha256). | # | `id` | Task | Artifact path | Bytes | sha256 (full) | Kind | Backbone (frozen) | |---|---|---|---|---|---|---|---| | 1 | `change_head` | `change` | `artifacts/change/levir_change_v001/head.pt` | 63,231,009 | `c5ef31277b67aa01a593aec0eac503eeaccc6d674349fda20ca44c9cc6f8e9fa` | trained head | none external — ResNet-18 (torchvision ImageNet) + PAM, trained in-project | | 2 | `change_vqa_head` | `change_vqa` | `artifacts/change_vqa/run/head.pt` | 5,822,809 | `cfae5e43b97ca930f568dc5b8ae4f36b24e9ff717af226159802206ffd63a82a` | trained head | STANet change detector (frozen, backing the head's change features) | | 3 | `optical_sar_fusion_head` | `optical_sar` | `artifacts/optical_sar/fusion_head_production_v001/head.pt` | 14,427,457 | `785815729a3a39fc34dc41894efaf00d8739365d970a3f830a326e68ae888dab` | trained head (production) | `antofuller/CROMA` (`CROMA_base.pt`, rev `0dd28e3d633b`) | | 4 | `grounding_head` | `grounding` | `artifacts/grounding/remoteclip_grounding_v001/head.pt` | 12,639,041 | `93432f7034be91a8ffd9c1a84e3eeec00bed7832c043fe7f83d2be230284c6bb` | trained head | `chendelong/RemoteCLIP` (`RemoteCLIP-ViT-B-32.pt`, rev `bf1d8a3ccf2d`) | | 5 | `router_adapter` | `router` | `artifacts/router/router_adapter_v001/adapter.pt` | 211,961 | `8527c3ed28a293e13293d48601d48e3ceafa137b9acabddaf5de31a58a509b5c` | trained adapter | `sentence-transformers/all-MiniLM-L6-v2` (rev `1110a243fdf4`) | | 6 | `vlm_lora_adapter` | `vlm` | `.scratch/phase6_real_adapter/phase6_adapter/adapter_model.safetensors` | 34,798,048 | `07c76a75fa04624880ed7730590f5fdd7b145a8232e3c0af411c3c545a5adf5e` | **LoRA adapter** (PEFT) | `HuggingFaceTB/SmolVLM-500M-Instruct` (rev `a7da5b986cb5`) | Every artifact in the manifest carries `status: "PRESENT"` and `config_hash: "78f1e3700da15aa1"`. ### 1.1 What each `kind` means | `kind` | Meaning | How it is loaded | |---|---|---| | `trained_head` | A module trained in-project on top of a frozen encoder. The encoder is fetched separately. | `torch.load(...)` of a `state_dict`, or the module's own `load_*` function. | | `trained_adapter` | A small classifier attached to a frozen sentence encoder whose embeddings are cached. | `IntentAdapter.from_config_dict` + `load_state_dict`. | | `lora_adapter` | A PEFT LoRA delta attached to a frozen VLM at load time. | `peft.PeftModel.from_pretrained(model, dir)` (`specialists/vqa/model.py::_attach_adapter`). | ### 1.2 Training checkpoints are NOT released artifacts Training intermediates exist on the machines that trained — e.g. the grounding head's `checkpoint_last.pt` and the VLM adapter's `checkpoint-1500/`, `checkpoint-2000/`. These are **archived as provenance, not released as model weights** (`docs/TRAINING.md` §8). For the VLM adapter this matters concretely: the promoted adapter is the end-of-training **top-level** save, and it is **not** `checkpoint-2000` (three distinct digests; see §3.6). ### 1.3 Two cross-checks where the computed hash agrees with an independently-recorded value The manifest is generated by hashing the files on disk. For two artifacts, those computed digests can be compared against values that were recorded *independently*, at a different time, by a different process — which is a genuine external cross-check rather than a self-consistency claim. **Cross-check A — `change_vqa` = `cfae5e43…`.** The manifest's `sha256` for `change_vqa/head.pt` equals, exactly: - `artifacts/change_vqa/run/PROMOTION.json` → `artifact.sha256`; - the same file → `source.checkpoint_sha256_in_run_record` (the digest recorded in the Kaggle run record, *before* promotion); - `artifacts/calibration_v001.json` → `provenance.checkpoint_sha256` (recorded when the temperature was fitted, a separate step). `PROMOTION.json` → `source.hash_agrees_across` additionally records that the digest agrees across `model_metadata.json`, `run_record.json` and `hashes.json (run.checkpoint_sha256)`. The bytes were copied byte-identically (`source.byte_identical_to_source: true`) and `artifact.weights_modified: false`. **Cross-check B — `vlm` = `07c76a75…`.** The manifest's `sha256` for `vlm/adapter_model.safetensors` equals, exactly: - `artifacts/vlm/phase6_closure.json` → `production_adapter`… (via `artifacts/vlm/run1_test_recovery/adapter_verification.json`) → `weights_file_sha256` = `07c76a75fa04624880ed7730590f5fdd7b145a8232e3c0af411c3c545a5adf5e`; - the same file → `promoted_adapter.sha256`; - the adapter's own `ARTIFACT_SHA256SUMS.json`, against which the directory was verified (`manifest_check.clean: true`, `n_files_in_manifest: 14`, `n_files_on_disk: 14`, `mismatched_files: []`). Both cross-checks are recorded in [`../MODEL_CARD.md`](../MODEL_CARD.md) §1 as well. ### 1.4 Why `parameters` is `null` for most artifacts The manifest records `parameters: null` for four of the six artifacts. That is deliberate: the generator does not open the checkpoints (opening a `.pt` to count parameters would require the model code and torch, and would make the manifest's generation depend on the environment). Parameter counts that *were* measured elsewhere are recorded in this document with their source. Where a parameter count is not established, this document writes `UNKNOWN — not established from the available evidence` rather than deriving one from file bytes. --- ## 2. Frozen backbones — pinned, never retrained Backbones are resolved from the Hugging Face Hub on first use, pinned by revision. The revision pins are the load-bearing part: a moving `main` would make every benchmark number unreproducible. | Role | Repository | Revision | Size | Measured identity | Notes | |---|---|---|---|---|---| | Router encoder | `sentence-transformers/all-MiniLM-L6-v2` | `1110a243fdf4` | 90.9 MB | 22,713,216 params, 384-dim embeddings | tokenizer ceiling **256**; truncation set to **128** | | VLM | `HuggingFaceTB/SmolVLM-500M-Instruct` | `a7da5b986cb5` | ~1015 MB safetensors | 516,165,824 params (base) | processor `longest_edge` **must** be pinned (F5-2) | | Grounding | `chendelong/RemoteCLIP` (`RemoteCLIP-ViT-B-32.pt`) | `bf1d8a3ccf2d` | 605.2 MB | 151,277,313 params; width **768**, **projected** dim **512** | patch size 32; 7×7 tokens at 224 | | Optical-SAR | `antofuller/CROMA` (`CROMA_base.pt`) | `0dd28e3d633b` | 777.6 MB (777,563,846 bytes) | 194,365,440 params; `encoder_dim` 768 | `image_resolution` 120; asymmetric `s1_depth=6`, `s2_depth=12` | | Change encoder | — (torchvision) | — | — | ResNet-18, `IMAGENET1K_V1` | `pretrained_used: true` in the trained artifact | ### 2.1 Router encoder — `all-MiniLM-L6-v2` - Declared in `configs/base.yaml` under `router:` — `model`, `revision: 1110a243fdf4`, `max_length: 128`, `embedding_dim: 384`. - **Finding F4-1.** The MiniLM tokenizer's own ceiling is **256** (verified by probe). `128` is a deliberate truncation well inside that ceiling, not the model limit. The encoder **asserts the configured value is ≤ 256**, because truncating above the ceiling is a silent no-op — a control that appears to work and does nothing. - **Finding F4-2.** The encoder is frozen, so embeddings are cached and the adapter trains on cached vectors. Measured on CPU: **20 epochs over 4,096 × 384 in 0.28 s**. No GPU required. - The revision is asserted by `router/encoder.py` (guards on revision and `max_length`). ### 2.2 VLM — `SmolVLM-500M-Instruct` - Declared under `vlm:` — `checkpoint`, `revision: a7da5b986cb5`, `max_new_tokens: 128`, `do_sample: false`, `temperature: 0.0`, `processor_longest_edge: 512`, `do_image_splitting: true`, `max_images_per_call: 1`. - **Finding F5-1.** `AutoModelForVision2Seq` does **not exist** in transformers 5.17.0 (it is not merely deprecated — referencing it raises `AttributeError`). The loader class is resolved by feature detection over `("AutoModelForImageTextToText", "AutoModelForVision2Seq", "AutoModelForMultimodalLM")`, never hardcoded (`specialists/vqa/model.py::resolve_loader_class`). - **Finding F5-2 (cost).** The processor's default `longest_edge` is 2048, which upscales a 512-px tile 4× and then splits it (`do_image_splitting=True`) into **17 sub-images** and 1142 prompt tokens. MEASURED: default → `pixel_values (1, 17, 3, 512, 512)`; pinned → `pixel_values (1, 1, 3, 512, 512)`. The plan estimated a 4× cost overrun; the real figure is **~17×**. The pin **must** be set explicitly on the processor at construction time, and `generate` raises loudly if more than one image is produced for a single input. - **Finding F5-3.** SmolVLM requires one `` token per image in the prompt; hand-written prompt strings raise `ValueError`. All prompts are built through `processor.apply_chat_template()`. - **Finding F5-4.** The dtype kwarg is not discoverable by signature (`from_pretrained` is `**kwargs`-only). Resolved by a call-time fallback over `("dtype", "torch_dtype")`. ### 2.3 Grounding — `RemoteCLIP ViT-B/32` - Declared under `grounding:` — `checkpoint_repo`, `checkpoint_file: RemoteCLIP-ViT-B-32.pt`, `checkpoint_revision: bf1d8a3ccf2d`, `model_name: ViT-B-32`, `image_size: 224`, `resolution_frozen: true`, `nms_iou: 0.50`, `max_candidates: 20`, `confidence_threshold: 0.40`, `benchmark_box_scale: 100.0`, `coordinate_system: normalized_0_1`, `encoder_projected_dim: 512`. - **Measured, not assumed** (`specialists/grounding/remoteclip.py`): `VERIFIED_PATCH_SIZE = 32`, `VERIFIED_TRANSFORMER_WIDTH = 768`, `VERIFIED_PROJECTED_DIM = 512`, `VERIFIED_PARAMETERS = 151_277_313`. `_verify_contract()` asserts all four against the loaded model at load time. - **The 768-vs-512 distinction is the one that bites.** `visual.positional_embedding` is 768 wide and `visual.proj` is `(768, 512)`; the embeddings the text tower can be compared against are the **projected** ones (512). Using 768 anywhere here is a shape error torch would surface only at the similarity step — after the patch features have already been computed and cached. - Tokens: 7×7 = 49 (+1 CLS) at 224; 14×14 = 196 (+1 CLS) at 448. `SUPPORTED_RESOLUTIONS = (224, 448)`; which one to *use* was an empirical question answered by the resolution experiment (§4.1), and neither is marked "preferred" in the encoder. ### 2.4 Optical-SAR — `CROMA-base` - Declared under `croma:` — `checkpoint_repo`, `checkpoint_file: CROMA_base.pt`, `checkpoint_revision: 0dd28e3d633b`, `variant: base`, `image_resolution: 120`, `encoder_dim: 768`, `optical_channels: 12`, `sar_channels: 2`, `modalities: [optical, sar, joint]`, `modalities_used: [optical, sar, joint]`. - **Finding C-7.** `image_resolution % 8 == 0`; native 120 → 15×15 = **225 patches**. Enforced in `core/config.py` and re-checked in `CROMAEncoder.__init__`. - **Finding C-1 — CROMA is never given a mask.** The forward pass takes exactly two arguments, `model(SAR_images=..., optical_images=...)`. The availability mask is consumed by the **fusion head**, not by CROMA (see §3.3). The rationale: CROMA is a masked autoencoder, and handing it an availability mask invites it to reconstruct missing channels — precisely the fabrication the sensor adapter exists to prevent. - First real forward pass (measured): CPU, 0.89 s for a batch of 2 at 120 px, 194,365,440 parameters, `n_patches = 225`, `(B, 768)` GAPs and `(B, 225, 768)` tokens confirmed. Two facts no document had recorded surfaced: CROMA-base is **asymmetric** (`s1_depth=6`, `s2_depth=12`) and the joint cross-attention is **directional** (SAR queries optical). - **DEV-1 (constructor).** CROMA is distributed as a GitHub repository, not a pip package; the constructor is `use_croma.PretrainedCROMA`, and `use_croma.py` **must be vendored**. Ruled **ACCEPTED as an implementation detail** — no `ARCHITECTURE CHANGE` entry (`docs/PHASE14_OPTICAL_SAR_DECISIONS.md` §2). The vendored file is recorded at `specialists/optical_sar/vendor/use_croma.py`, 14,556 bytes, sha256 `a38567beed29eb08108a47cdc97fe98aec50fd4be0bd98a5266bcd18aafb7c5b`. - **DEV-2 (input normalisation).** Resolved 2026-09-18: option (a) — match upstream's per-channel `mean ± 2·std` stretch immediately before the forward pass with `use_8_bit` enabled — adopted as the required implementation; option (c) registered as a **gated** experiment with (a) as its control; option (b) rejected as a standalone path. Implemented in `specialists/optical_sar/radiometry.py` and wired through `croma.py`. ### 2.5 Change encoder — ResNet-18, trained in-project The change detector's encoder is **not** an external pinned backbone in the sense the others are: it is a torchvision `resnet18`, requested with `change.pretrained: true` and recorded in the trained artifact's embedded config as `pretrained_used: true` with `encoder: "resnet18"` and `encoder_channels: [64, 128, 256, 512]`. The code does **not** make pretrained weights a hard dependency. Resolution order (`specialists/change/stanet.py`): (1) a local weights file, if given; (2) torchvision's `IMAGENET1K_V1` download; (3) random initialisation **with an explicit warning** — because `pretrained: true` in a config must not become a lie when a download fails. The trained artifact records `pretrained_used: true`, so this run used ImageNet weights. ### 2.6 Backbone licensing Backbones are **not redistributed** by this project. They are fetched from the Hub at run time and carry their own licences (see each model's HF page). See §6. --- ## 3. Per-artifact deep reference Each subsection below gives, for one artifact: its path, byte count, sha256, architecture, **every hyperparameter** (from `configs/base.yaml` and the code), the backbone and its pinned revision, the training data, the evaluation protocol, the measured numbers (with artifact key paths), the acceptance status, and its limitations. ### 3.0 Enforced configuration invariants, with arithmetic Two invariants are **enforced at load time**, not merely documented. Both exist because the failure mode they prevent is a *silent shape error* — a tensor of exactly the right shape that trains to a worse number, the hardest kind of bug to notice. #### 3.0.1 `fusion.input_dim == 2318` ``` input_dim = len(modalities_used) * encoder_dim + optical_channels + sar_channels = 3 * 768 + 12 + 2 = 2304 + 14 = 2318 ``` `configs/base.yaml` states the arithmetic in a comment above `fusion.input_dim: 2318`, and the code recomputes it **twice**, independently: - `core/config.py` recomputes it as `len(modalities_used) * encoder_dim + optical_channels + sar_channels` at load time (finding C-1); - `specialists/optical_sar/fusion_head.py::expected_fusion_dim` returns `int(modalities_used) * int(encoder_dim) + int(optical_channels) + int(sar_channels)`; - `build_fusion_head` **raises `ModelLoadError`** if `int(input_dim) != 2318`, naming the arithmetic (`3*768 + 12 + 2`) in the message; - `assemble_fusion_input` concatenates `[optical_GAP, sar_GAP, joint_GAP, optical_mask, sar_mask]` and **raises `SpecialistError`** if the assembled width is not the expected width. The order of the concatenation is frozen (freeze §2.5) and **asserted rather than assumed**, because a permutation produces a tensor of exactly the right shape that trains to a worse number. > The mask goes **here**, not into CROMA. Freeze §2.5: channel/band dropout during fusion-head > training is **mandatory** — it is what teaches the head to trust the availability mask. With no > dropout the head learns to read channel 4 unconditionally, because in training channel 4 was always > present; on a 4-band sensor channels 5–12 are *always* zero, and a head that never saw a masked > channel treats those zeros as a measurement of blackness rather than as absence. > `channel_dropout()` is a real function, tested directly (measured: `p=1.0` keeps 48/48; `p=0.4` > drops to 18/48; dropped features are exactly `0.0`, not renormalised; a band the sensor never > measured stays dropped even at `p=1.0`; the seeded RNG is deterministic). #### 3.0.2 `grounding_head.feature_dim == 2048` ``` feature_dim = 4 * grounding.encoder_projected_dim = 4 * 512 = 2048 ``` The per-cell feature is ``` f_i = concat([p_i, t, p_i * t, global_pool]) -> 4 * 512 = 2048 ``` where `p_i` is the patch token, `t` the text embedding broadcast to every cell, `p_i * t` the element-wise cross-modal alignment, and `global_pool` the mean over all patches. The global term matters: a per-cell MLP otherwise cannot see anything outside its own patch, and a 1/7-of-image receptive field is too small for objects that span several cells. The invariant is enforced **three** ways: - `core/config.py` **rejects any value other than** `4 * grounding.encoder_projected_dim` at load time — deliberately, so the head can be validated *without importing torch*; - `specialists/grounding/head.py::build_head` recomputes `expected = 4 * VERIFIED_PROJECTED_DIM` and raises `ValueError` naming both numbers if they differ; - `GroundingHead.forward` re-checks `feat.shape[-1] != self.feature_dim` and raises, and `RemoteCLIPEncoder._verify_contract` asserts `visual.proj.shape[1] == 512` against the real model. A mismatch is a silent shape error: torch raises only at the similarity step, after the patch features are already cached. #### 3.0.3 Other enforced invariants (summary) | Invariant | Enforced where | Failure mode prevented | |---|---|---| | `croma.image_resolution % 8 == 0` | `core/config.py`, `CROMAEncoder.__init__`, `OpticalSarSpecialist.__init__` | CROMA asserts it internally (finding C-7) | | `router.max_length <= 256` | `router/encoder.py` | truncating above the tokenizer ceiling is a silent no-op | | `change` spatial dims divisible by 8 | `STANetStyleChangeDetector.forward` | encoder stride; a 1-px-off raster is reflect-padded and cropped back, with a warning | | `sa_mode in {PAM, BAM}`; `BAM` raises | `STANetStyleChangeDetector.__init__` / `.forward` | a config asking for BAM fails visibly rather than silently aliasing PAM | | `fusion_head` device must be `"cpu"` | `build_fusion_head` | a device string that would silently fall back is worse than a refusal | | `task_dim >= 2` | `OpticalSarSpecialist.__init__` | a margin cannot exist over one class | | exactly 2 assets for change / change_vqa / optical_sar; exactly 1 for vqa/caption | each specialist's `validate_request` | a silent "just use the first two" produces a confident answer to a question the caller did not ask | --- ### 3.1 `change` — STANet-style Siamese change detector **Status:** `IMPLEMENTED`, `VERIFIED` — the only task whose headline metric carries the `VERIFIED` tag, because it is the only one measured against a single, immutable public test split with a frozen threshold. #### Identity | Field | Value | Source | |---|---|---| | Manifest `id` | `change_head` | `models/manifest.json` | | Path | `artifacts/change/levir_change_v001/head.pt` | manifest | | Bytes | 63,231,009 | manifest | | sha256 | `c5ef31277b67aa01a593aec0eac503eeaccc6d674349fda20ca44c9cc6f8e9fa` | manifest, `checksums.sha256` | | HF path | `change/head.pt` | manifest | | Architecture | STANet-style Siamese change detector (ResNet-18 + PAM) | manifest, code | | `kind` | `trained_head` | manifest | | `backbone` | `null` (trained in-project; ResNet-18 ImageNet trunk) | manifest | | Config hash | `78f1e3700da15aa1` | manifest | | Parameters | `UNKNOWN — not established from the available evidence` (manifest records `null`) | manifest | #### Architecture (`specialists/change/stanet.py`) A STANet-shaped Siamese network — shared Siamese encoder, spatial-temporal attention over feature differences, feature-difference aggregation decoder — **reimplemented rather than vendored**, per the Phase 0 resolution of finding C-9 (the upstream repo is Python 3.6-era and depends on `visdom`/`apex`). ``` T1 ---> [ shared encoder ] ---> f1 (4 levels) T2 ---> [ shared encoder ] ---> f2 (4 levels) | |f1 - f2| + concat([f1, f2, |f1-f2|]) | spatial attention (PAM) | progressive decoder + skips | 1-channel change logit ``` - **Weights are tied, not copied.** Both branches call the **same** module instance (`SharedResNetEncoder`), so there is no second encoder to fall out of sync — the "shared encoder" requirement from plan §16 enforced by construction, not by convention. - **Measured encoder shapes** (ResNet-18 trunk, 256×256 input): | Level | Channels | Grid | Positions | |---|---|---|---| | `layer1` | 64 | 64×64 | 4096 | | `layer2` | 128 | 32×32 | 1024 | | `layer3` | 256 | 16×16 | 256 | | `layer4` | 512 | 8×8 | 64 | - **The attention memory line is computed, not assumed.** STANet's PAM builds a full `positions × positions` attention matrix. At batch 8, fp32: ``` layer1 8 * 4096^2 * 4 = 537.0 MB <- exceeds a Kaggle session's budget layer2 8 * 1024^2 * 4 = 33.5 MB <- fine layer3 8 * 256^2 * 4 = 2.1 MB <- fine layer4 8 * 64^2 * 4 = 0.13 MB <- fine ``` Attention is therefore applied at **layers 2, 3 and 4** and **skipped at layer 1**, where the difference features are fused directly instead. The decision is **recomputed on every forward pass** from the actual batch size and a byte budget (`DEFAULT_ATTENTION_BUDGET_BYTES = 256 * 1024 * 1024`), so changing batch size or tile size moves the line correctly rather than silently overrunning memory. Applied/skipped levels are returned in the output so the trace can report them (`ChangeOutput.to_trace`). **This is a deviation from a literal STANet reproduction, recorded as one** — attention at 64×64 would not fit. It is not a silent substitution. - **Difference fusion.** `DifferenceFusion` reduces `concat([f1, f2, |f1-f2|])` (3C channels) back to a working width with `Conv2d(3C, out, 3, padding=1, bias=False) → BatchNorm2d → ReLU`. - **Attention module.** `SpatialAttention` is PAM-style: 1×1 `query`/`key` convolutions to `hidden = max(1, channels // reduction)` with `reduction = 8`, a 1×1 `value` to `channels`, and a 1×1 `out`; `softmax(q @ k / sqrt(hidden))` over positions, residual `x + out(attended)`. - **Decoder.** Progressive, coarse-to-fine: `dec3` and `dec2` are `DecoderBlock(width, width, width)`; `dec1` is `DecoderBlock(width, width, width // 2)`; `final_up` is `ConvTranspose2d(width//2, width//2, 4, stride=4)`; `head` is `Conv2d(width//2, 1, 1)` with `FINAL_BIAS_INIT = -2.0` (LEVIR-CD's changed-pixel fraction is roughly 5–15%, so a bias of −2.0 starts the prior near 0.12 rather than 0.5 and stops the first epochs being spent un-learning a saturated sigmoid). - **Loss.** Composite BCE + Dice, `change_loss(logits, target, bce_weight=0.5, dice_weight=0.5, pos_weight=None)`. Dice is computed on soft probabilities so it is differentiable and directly optimises region overlap; `eps = 1e-6` guards the empty-target case (a tile with no change would otherwise be `0/0 = NaN`). `pos_weight` is left as an explicit knob rather than a hidden default, because Dice already addresses the class imbalance. #### Hyperparameters — every value From `configs/base.yaml` (`change:`): | Key | Value | |---|---| | `tile_size` | 256 | | `tile_overlap` | 0 | | `threshold` | 0.50 | | `min_component_pixels` | 32 | | `encoder` | `resnet18` | | `sa_mode` | `PAM` (`BAM` declared but not implemented; raises) | | `pretrained` | true | | `learning_rate` | 0.001 | | `batch_size` | 8 | | `bce_weight` | 0.5 | | `dice_weight` | 0.5 | | `levir_split.train` | 7120 | | `levir_split.val` | 1024 | | `levir_split.test` | 2048 | From the trained artifact's **embedded** config (`artifacts/change/eval_test/eval_result.json` → `checkpoint_embedded_config`), which is the config actually baked into the weights: | Key | Value | |---|---| | `width` | 128 | | `sa_mode` | `PAM` | | `attention_budget_bytes` | 268435456 (256 MB) | | `encoder` | `resnet18` | | `encoder_channels` | `[64, 128, 256, 512]` | | `frozen_encoder` | false | | `pretrained_used` | true | #### Training data LEVIR-CD-256 — the standard 256×256 change-detection benchmark. Split used (from `configs/base.yaml`): train **7,120** / val **1,024** / test **2,048**. The test split is immutable and public. #### Evaluation protocol `artifacts/change/eval_test/eval_result.json`. Environment recorded in the artifact: device `cuda`, torch `2.10.0+cu128`, Python `3.12.13`, platform `Linux-6.12.90+-x86_64-with-glibc2.35`, `cuda_available: true`, `seconds: 55.359`. Config-hash integrity is recorded explicitly: `config_hash: 78f1e3700da15aa1`, `checkpoint_config_hash: 78f1e3700da15aa1`, `checkpoint_config_hash_checked: true`, `config_drift: false`, `config_drift_acknowledged: false`. Evaluation population: `n: 2048`, `n_images_with_change: 935`, `mean_change_fraction: 0.0509`, `n_pixels: 134217728`, `threshold: 0.5`. Change-fraction quantiles (`change_fraction_quantiles`): `min 0.0`, `p50 0.0`, `p90 0.197205`, `max 0.684937`. The median tile has **zero** changed pixels, which is why pooled and macro metrics are both reported. #### Measured numbers All from `artifacts/change/eval_test/eval_result.json`. | Metric | Value | Exact key path | |---|---|---| | **pooled IoU** | **0.8122** | `metrics.pooled.iou` | | **macro IoU** | **0.8457** | `metrics.macro.miou` | | **pooled F1** | **0.8964** | `metrics.pooled.f1` | | pooled mean IoU (`miou`) | 0.9007 | `metrics.pooled.miou` | | pooled precision | 0.9195 | `metrics.pooled.precision` | | pooled recall | 0.8745 | `metrics.pooled.recall` | | macro IoU (`iou`, per-class change IoU) | 0.718 | `metrics.macro.iou` | | macro F1 | 0.7962 | `metrics.macro.f1` | | macro precision | 0.8506 | `metrics.macro.precision` | | macro recall | 0.7757 | `metrics.macro.recall` | | tp | 5,978,997 | `metrics.pooled.tp` | | fp | 523,658 | `metrics.pooled.fp` | | fn | 858,407 | `metrics.pooled.fn` | | tn | 126,856,666 | `metrics.pooled.tn` | The confusion counts are identical between the pooled and macro blocks (as they must be — they are the same pixel population; the two blocks differ only in how the class-wise scores are aggregated). **Pooled vs macro, and why both.** The test split is only ≈ 5 % changed pixels. Pooled IoU answers "how well does the mask overlap overall"; macro IoU answers "how well does each class do, averaged". Quoting one alone would hide the imbalance question. #### Acceptance status `VERIFIED`. This is the only task whose headline number carries the `VERIFIED` tag in [`BENCHMARKS.md`](BENCHMARKS.md) §1, because it is the only one measured against a single, immutable public test split with a frozen threshold. #### Serving wiring (and why the checkpoint is not in the config) A trained head exists and is benchmarked, yet it is **deliberately not wired into serving by default** (`specialists/change/specialist.py`). Populating `change.checkpoint_path` in `configs/base.yaml` would move `Config.hash`, and `scripts/eval_change.py` refuses to score on a hash drift (exit 3) — so that one edit would invalidate the project's own benchmark number. The wiring path is therefore the registry `builders=` override (`core/registry.py`), which injects the checkpoint without touching the config. With no checkpoint wired, a missing artifact is a **deployment** case, not a crash: the specialist runs the randomly-initialised model, marks the result `degraded`, and says so plainly, because a randomly-initialised change map is a map of noise and presenting it as a detection would be exactly the fabrication the evidence system exists to prevent. `has_checkpoint` distinguishes "a checkpoint was loaded" from "we built a random one and are being honest about it". #### Limitations - **Absolute IoU is high, but this is one benchmark.** LEVIR-CD-256 is overhead optical; it says nothing about the hidden ISRO/SAC distribution (Cartosat-2S + RISAT). - **Attention is skipped at layer 1** — a recorded deviation from literal STANet, forced by memory. - **The detector is not the product path by default** (§ above). - **Post-processing suppresses spatial claims on a mis-registered pair.** Poor co-registration drives confidence to 0.0 and withholds region claims; the maps are still written, but the region assertions are not made. This is by design (`architecture freeze §2.4`, plan §40) but it means the reported metric is for a well-registered benchmark, not for arbitrary pairs. - **Minimum region size is 32 px** (`change.min_component_pixels`), so sub-32-px changes are discarded by construction. --- ### 3.2 `change_vqa` — change question answering head **Status:** `IMPLEMENTED`, `MEASURED`, ruling **OPEN**. A head that answers natural-language change questions over a temporal pair. It answers the eight CDVQA question types (`change_or_not`, `change_ratio`, `change_ratio_types`, `change_to_what`, `increase_or_not`, `decrease_or_not`, `largest_change`, `smallest_change`) over a **closed 19-answer space**. #### Identity | Field | Value | Source | |---|---|---| | Manifest `id` | `change_vqa_head` | `models/manifest.json` | | Path | `artifacts/change_vqa/run/head.pt` | manifest | | Bytes | 5,822,809 | manifest | | sha256 | `cfae5e43b97ca930f568dc5b8ae4f36b24e9ff717af226159802206ffd63a82a` | manifest, `PROMOTION.json` | | HF path | `change_vqa/head.pt` | manifest | | Architecture | `change_vqa_head_v1` | `PROMOTION.json` → `artifact.architecture` | | **Parameters** | **1,453,912** | `PROMOTION.json` → `artifact.parameters` | | Backbone | STANet change detector (frozen, backing the head's change features) | manifest | | Config hash | `78f1e3700da15aa1` | manifest | | `satquery_trained` | true | `PROMOTION.json` → `artifact.satquery_trained` | | `eval_mode` | true | `PROMOTION.json` → `artifact.eval_mode` | | `non_finite_tensors` | **0** | `PROMOTION.json` → `artifact.non_finite_tensors` | | `weights_modified` | **false** | `PROMOTION.json` → `artifact.weights_modified` | #### Architecture and the frozen dependency The head consumes **cached features**, not raw imagery: a frozen STANet detector produces a change representation, and a frozen question text encoder (`sentence-transformers/all-MiniLM-L6-v2`) produces a text vector; the head maps those to an answer index. Recorded feature specs (`PROMOTION.json` → `identity`): `feature_spec: change_feat_v1`, `change_cache_spec: c801326f85a185f8`, `text_cache_spec: d2801ea1a314354a`, `preprocessing_version: change_vqa_preproc_v1`. The **frozen dependency** is the STANet detector itself (`PROMOTION.json` → `frozen_dependency`): | Field | Value | |---|---| | role | STANet change detector backing the head's change features | | path | `artifacts/change/levir_change_v001/head.pt` | | sha256 | `c5ef31277b67aa01a593aec0eac503eeaccc6d674349fda20ca44c9cc6f8e9fa` | | bytes | 63,231,009 | | `detector_trained` | true | | `verified_byte_exact_vs_local` | true | This digest is byte-identical to the `change` artifact in §3.1 — the head is wired to the same checkpoint the change task releases. **The detector is frozen and shared in spirit, not in instance** (`specialists/change/vqa_specialist.py`). When the planner routes a change+language request it plans **both** a `change` step and a `change_vqa` step, so the detector runs twice for one request. That is a real cost, stated rather than hidden: the weights are loaded once (the registry caches the specialist instance), so the second cost is a forward pass, not a 60 MB load. The alternative (passing the change map between steps) needs the controller to hand artifacts between steps, which it does not do today. `execute` therefore accepts an optional `change_map` path in `request.params` for a future planner to populate — **unused today and documented as such**, so the hook exists without pretending the wiring does. #### Hyperparameters and selection | Key | Value | Source | |---|---|---| | `apply_type_mask` | true (serving default) | config + `vqa_specialist.py` | | `image_size` | 256 (feature extractor default) | `vqa_specialist.py` (`change_vqa.image_size`) | | `seed` | 42 | `PROMOTION.json` → `identity.seed` | | `epoch_selected` | **8** | `PROMOTION.json` → `identity.epoch_selected` | | `selected_on` | Val answer accuracy | `PROMOTION.json` → `identity.selected_on` | | `val_answer_accuracy` | **0.700018** | `PROMOTION.json` → `identity.val_answer_accuracy` | | `stop_reason` | `early_stopping` | `PROMOTION.json` → `identity.stop_reason` | | `LOW_CONFIDENCE_THRESHOLD` | 0.40 (a reporting threshold, not a calibration) | `vqa_specialist.py` | The **type mask** restricts the answer to those legal for the question type. It is on by default in serving because the type is known from the question and an illegal answer is never right; the evaluation reports masked and unmasked separately so the mask's contribution is visible. `PROMOTION.json` → `verification.mask_gain` is **0.0**. #### Training data CDVQA annotations + SECOND (SCD) imagery (`docs/DATASETS.md` §3). The CDVQA repository publishes **annotations only**; imagery is acquired from SECOND, name-verified **2,968 / 2,968 MATCH**, extracted to `data/cdvqa/{im1,im2,label1,label2}/`. Val images 400; val questions **16,441**; test questions **39,686**. **Temporal and label semantics** — established from evidence, with honest uncertainty (`vqa_specialist.py::TEMPORAL_ORDER_NOTE`): `T1 = pre, T2 = post`; `label1=pre, label2=post` is **proven** (agreement 1.0000 over 2,968 scenes); `im1=pre, im2=post` is **supported statistically, not proven**. The distinction travels in the evidence payload so a reader can see which leg the result rests on. **Trained externally.** The head was trained outside the repository, on an external GPU (Kaggle), following `docs/R02_KAGGLE_TRAINING_GUIDE.md`, whose entry status was `IMPLEMENTATION_READY_FOR_EXTERNAL_TRAINING` and whose explicit contract is: *training produces an artifact, not a verified capability, and the run record says `TRAINED_UNVERIFIED`.* The returned checkpoint was then **promoted** through a byte-identity gate (`PROMOTION.json` → `state`): `before_promotion: TRAINED_UNVERIFIED`, `after_promotion: PROMOTED`. The note is explicit: *promotion records provenance and wires the serving path; it does not itself confer VERIFIED status — that is the maintainer's ruling.* #### Evaluation protocol `artifacts/change_vqa/run/PROMOTION.json` → `verification`. Verification report `ARTIFACT_VERIFICATION.md` (2026-09-22): `checks_passed: 93`, `checks_failed: 0`, `checks_unverified: 0`. **Two test sets are reported. Quoting only the better one would be selective.** | Test set | n | accuracy | macro F1 | |---|---|---|---| | `test` | 39,686 | **0.697626** | **0.378373** | | `test2` | 31,036 | **0.651469** | **0.372309** | | Field | Value | Exact key path | |---|---|---| | test accuracy | 0.697626367 | `verification.test_accuracy` | | test macro F1 | 0.378373275 | `verification.test_macro_f1` | | test2 accuracy | 0.651469262 | `verification.test2_accuracy` | | test2 macro F1 | 0.372308516 | `verification.test2_macro_f1` | | n scored, test | 39,686 | `verification.n_scored_test` | | n scored, test2 | 31,036 | `verification.n_scored_test2` | | global majority baseline, test | 0.311546 | `verification.global_majority_baseline_test` | | global majority baseline, test2 | 0.178728 | `verification.global_majority_baseline_test2` | | mask gain | 0.0 | `verification.mask_gain` | | metric ruling | `OPEN — the plan leaves the accuracy/macro-F1 interpretation owner-gated. No official aggregate metric is asserted here.` | `verification.metric_ruling` | **The wide gap between accuracy and macro-F1 means the head is carried by common classes and performs poorly on rare ones.** The global-majority baselines make the gap concrete: 0.311546 on `test` means a constant predictor would score 0.31, and the head scores 0.70; but the macro-F1 of 0.378 shows the per-class picture is far weaker than the aggregate. #### Acceptance status **OPEN.** No promotion/acceptance decision has been recorded. `PROMOTION.json` → `state.note` states this explicitly. The `metric_ruling` is `OPEN`. #### Serving behaviour — degraded mode produces no answer, on purpose `ChangeSpecialist` degrades to an untrained detector and still emits a change map. This specialist must **not** copy that, because the outputs are not comparable: a random change map is visibly noise, whereas an untrained 19-way classifier still emits a fluent, confident-looking `yes`. A user cannot tell the second from a real answer, so the untrained case returns **no answer at all** — `answer=""`, `degraded=True`, and a warning naming exactly which piece is missing. The controller then falls through to another attributed specialist. There is also a **feature-spec mismatch refusal**: `feature_spec_mismatch()` compares the spec hash the head was trained on against the spec hash this deployment produces, and refuses to answer if they differ. The reason is measured, not hypothetical: `configs/base.yaml`'s `change:` section carries no `checkpoint_path`, so the registry builds the change feature extractor with `checkpoint_path=None` and gets an **untrained** STANet — while `scripts/prepare_change_vqa.py` defaults to the trained LEVIR checkpoint. Training and serving would consume different representations, and the head would still emit a fluent `yes`. So it refuses, and names both specs. Confidence is **uncalibrated and says so**: `ConfidenceBreakdown.method` is `"uncalibrated"`, `raw` is the softmax probability of the chosen answer, `components` carries the top1-top2 margin, the number of distinct answers seen, and the normalised entropy. Nothing is fitted here — the calibration contract (§5) is a separate artefact. #### Limitations - **Weak on rare classes** (macro-F1 0.378 / 0.372 vs accuracy 0.698 / 0.651). - **Two test sets, both reported** — and the second is materially lower on accuracy (0.651 vs 0.698). - **Ruling OPEN** — no official aggregate metric is asserted. - **Serving needs two artefacts to be honest**: a trained head *and* a trained feature extractor; the shipped config wires neither by default. - **`im1=pre, im2=post` is supported statistically but not proven** — a residual provenance risk. --- ### 3.3 `optical_sar` — CROMA-base fusion head **Status:** `IMPLEMENTED`, `MEASURED`, ruling **OPEN**. This is the spec's #1 evaluation priority, and it is the only workflow whose inputs are two **different sensors**. #### Identity | Field | Value | Source | |---|---|---| | Manifest `id` | `optical_sar_fusion_head` | `models/manifest.json` | | Path | `artifacts/optical_sar/fusion_head_production_v001/head.pt` | manifest | | Bytes | 14,427,457 | manifest, `pre_registered_115_metric.json` → `head_bytes` | | sha256 | `785815729a3a39fc34dc41894efaf00d8739365d970a3f830a326e68ae888dab` | manifest, `checksums.sha256`, metric JSON `head_sha256` | | HF path | `optical_sar/head.pt` | manifest | | Architecture | CROMA-base fusion head (input_dim 2318 → hidden 512 → 19 classes) | manifest | | Backbone | `antofuller/CROMA` (`CROMA_base.pt`, rev `0dd28e3d633b`) | manifest | | Config hash | `78f1e3700da15aa1` | manifest | | Parameters | `UNKNOWN — not established from the available evidence` (manifest records `null`) | manifest | #### Architecture (`specialists/optical_sar/fusion_head.py`) CROMA produces three 768-dimensional vectors per sample (an optical GAP, a SAR GAP, and a joint GAP). The fusion head turns those, plus the availability masks, into a task prediction. ``` optical_GAP (B, 768) SAR_GAP (B, 768) joint_GAP (B, 768) optical_mask (B, 12) <- availability, from the sensor adapter sar_mask (B, 2) <- availability, from the sensor adapter --------- concat (B, 2318) ``` The frozen head architecture (`build_fusion_head`, freeze §2.5 / plan §17): ``` LayerNorm -> Linear(input_dim, hidden_dim) -> GELU -> Dropout -> Linear(hidden_dim, task_dim) ``` No activation on the output: this is a logit-producing task head, and a softmax here would be applied twice once a loss function adds its own. **The mask goes here, not into CROMA (finding C-1).** See §3.0.1. `CROMA_GAP_KEYS = ("optical_GAP", "SAR_GAP", "joint_GAP")` is declared once so the rest of the package refers to one spelling. #### Hyperparameters — every value From `configs/base.yaml` (`croma:` and `fusion:`): | Key | Value | |---|---| | `croma.checkpoint_repo` | `antofuller/CROMA` | | `croma.checkpoint_file` | `CROMA_base.pt` | | `croma.checkpoint_revision` | `0dd28e3d633b` | | `croma.variant` | `base` | | `croma.image_resolution` | 120 | | `croma.encoder_dim` | 768 | | `croma.optical_channels` | 12 | | `croma.sar_channels` | 2 | | `croma.modalities` | `[optical, sar, joint]` | | `croma.modalities_used` | `[optical, sar, joint]` | | `fusion.input_dim` | 2318 | | `fusion.hidden_dim` | 512 | | `fusion.dropout` | 0.2 | | `fusion.num_classes` | 19 | **Seed sweep.** Training was run as two arms (**armA**, **armB**) × five seeds (**100–104**), with a per-arm variance report (`armA_seed_variance_report.json`). The **production head** is a distinct, frozen artifact (`fusion_head_production_v001/head.pt`) with its own `production_head_record.json` and a `phase12_rerun_verification.json`. **A/B arm decision.** Made separately on `best_val_accuracy`: **A 0.837100 vs B 0.839100**, floor **0.0285** → **Arm A retained** (owner ruling R-14, `docs/PHASE12_115_METRIC_COMPUTED.md` §5). The metric JSON records `cache_arm: "A"`. **Channel dropout is mandatory** (freeze §2.5). Plan §20's robustness schedule: optical is trained at 100/80/60/40 % channel availability and SAR at 100/50 %. The mask is updated to match the dropped features — that is the entire point: dropping features while leaving the mask saying "present" would teach the head that the mask lies. Dropped channels are zeroed, **not renormalised** — renormalising would fabricate a scale the real missing-channel case does not have. #### Training data BigEarthNet (CLC-19), **single-label subset** (see the caveat below). Measured local subset: 28,000 S2 patches, 98 tiles, 12 bands per patch; full official corpus 480,038 patches. Training consumes cached CROMA features (`fusion_features/`, `fusion_features_armB/`, ~231 MB each), which are **reproducible and not released as model weights**. **Measured context for the single-label policy:** single-label patches are **17.57 %** of the corpus (96,537 of 549,488), and under this policy the rarest class survives as **1 patch** — a **59,204 : 1** imbalance (`docs/PHASE12_LABEL_POLICY_DECISION.md` §3.0). #### Evaluation protocol `artifacts/optical_sar/fusion_head_production_v001/pre_registered_115_metric.json`. This is a **separate, later, read-only** step (`scripts/eval_fusion_115.py`), deliberately **not** part of the trainer: every `run_record.json` carries `pre_registered_metric_computed = false` and `result_status = "PLUMBING_ONLY — fixture/loop evidence, NOT a result; the pre-registered 11.5 metric is not computed"`. That flag is **correct about the trainer** and must not be "fixed" — the trainer fits on `train`/`val` and never opens the test split, deliberately, so the held-out split cannot be contaminated by the search over 10 runs, 2 arms and 5 seeds. The tool **refuses (exit 2)** rather than guessing when: the head or cache is missing; the split is empty; the cache `config_hash` is not `78f1e3700da15aa1` (a different experiment); or the head cannot be built against the frozen `(B, 2318)` input contract. Definition (per `docs/PHASE14_CROMA_NORMALISATION_CHANGE.md` §4): *fusion-head accuracy and macro-F1 over the 19-class label space on the held-out split.* #### Measured numbers All from `artifacts/optical_sar/fusion_head_production_v001/pre_registered_115_metric.json`: | Metric | Value | Exact key path | |---|---|---| | **accuracy** | **0.931** | `accuracy` | | **macro F1** | **0.434161** | `macro_f1` | | loss | 0.254592 | `loss` | | split | `test` | `split` | | n scored | 4,000 | `n_scored` | | num classes | 19 | `num_classes` | | macro-F1 denominator | all 19 classes (absent classes contribute 0.0) | `macro_f1_denominator` | | classes present | `[0,2,3,4,5,6,7,8,9,10,12,13,17,18]` | `classes_present` | | classes absent | `[1,11,14,15,16]` | `classes_absent` | | deciding statistic | **false** | `is_deciding_statistic` | | cache arm | A | `cache_arm` | | cache path | `artifacts/optical_sar/fusion_features/test.npz` | `cache_path` | | cache config hash | `78f1e3700da15aa1` | `cache_config_hash` | **Optical-SAR accuracy 0.931 must ALWAYS travel with macro-F1 0.434161.** The two are recorded side by side in the artifact precisely so neither can be quoted alone. Per-class F1 (`_per_class_f1`, 19 terms, indexed 0–18): ``` [1.000, 0.000, 0.428, 0.571, 0.887, 0.000, 0.000, 0.800, 0.929, 0.940, 0.182, 0.000, 0.200, 0.438, 0.000, 0.000, 0.000, 0.877, 0.998] ``` **Two different averages come from the same per-class vector, and this is a trap:** | Averaged over | Value | |---|---| | **all 19 slots** — the pre-registered definition | **0.434161** | | the 14 present classes only | **0.589218** | A reader who computes the second and compares it to the recorded scalar will wrongly conclude the recorded figure is wrong. It is the 19-slot mean, and the trainer's `_macro_f1` divides by `num_classes` by construction. Both are reproducible: `sum(per_class)/19 = 0.43416082670034284` and `sum(per_class[c] for c in present)/14 = 0.5892182648076082` (`docs/PHASE12_115_METRIC_COMPUTED.md` §3.5). **Five absent classes contribute `0.0` by definition — but two present classes also score `0.0`.** Classes **1, 11, 14, 15, 16** are absent (their `0.0` describes no prediction); classes **5** and **6** are **present** and score `0.0`, i.e. a **real total miss**. So the honest reading is *both*: part of the low macro-F1 is populational, **and** there is genuine per-class failure. High accuracy with a wide per-class spread is the signature of prediction dominated by frequent classes. Measured test-set label distribution (`docs/PHASE12_115_METRIC_COMPUTED.md` §3.4): ``` [13, 0, 63, 39, 846, 4, 1, 6, 453, 246, 4, 0, 9, 18, 0, 0, 0, 34, 2264] ``` 14 of 19 classes present; five have zero samples; top-to-bottom ratio **2,264 : 1** (class 18 vs class 6). The majority class holds 2,264 / 4,000 = **0.566**, and the head scores **0.931** — so the accuracy is not a constant predictor. Per-class recall on the held-out split (selected rows): class 18 n=2264 recall **0.997**; class 9 n=246 recall **0.951**; class 17 n=34 recall **0.941**; class 8 n=453 **0.905**; class 4 n=846 **0.809**; class 3 n=39 **0.410**; class 12 n=9 **0.111**; classes 5 and 6 recall **0.000**. **Four independent verification checks** (`docs/PHASE12_115_METRIC_COMPUTED.md` §3): (1) calling `evaluate_fusion_head` directly returns `n=4000, accuracy=0.931, macro_f1=0.43416082670034284`; (2) rebuilding the input tensor by hand — `concat([optical_gap, sar_gap, joint_gap])` then `concat([…, optical_mask, sar_mask])`, widths `(4000,768)×3 + (4000,12) + (4000,2) = 2318` — gives 0.931 again (this checks feature *ordering*); (3) `sklearn.metrics` `accuracy_score = 0.931`, `f1_score(average="macro", labels=range(19), zero_division=0) = 0.43416082670034284`; (4) the per-class breakdown above. #### Acceptance status **OPEN.** `is_deciding_statistic: false`. The artifact's own `advisory` field states: *"This tool reports ONE head's held-out accuracy and macro-F1. It selects no head, ranks nothing and compares no arms. Whether this constitutes a Phase 12 pass is the owner's ruling."* Ruling 5 attaches **no numerical threshold**; ruling 4 requires it be described as a **single-label subset evaluation, never as a multi-label BigEarthNet/reBEN result**. #### ⚠️ The caveat that governs how these numbers may be used The cache metadata records `label_policy = require_single_label`, `n_skipped_by_policy = 0`. reBEN v2.0 is a **multi-label** corpus while the frozen head is a **single-label 19-class softmax** trained with `cross_entropy`. The extraction therefore used policy (b) — restrict to single-label patches — which preserves the frozen architecture and the 19-class space exactly but **changes the evaluation population**. So this metric **may not** be presented as: a multi-label BigEarthNet/reBEN result; comparable to published BigEarthNet numbers (almost all multi-label); or a statement about all 19 classes (**5 have no test samples here**). It **may** be presented as: *the pre-registered 11.5 metric, as computed under the single-label extraction policy the frozen architecture requires, on the held-out split.* #### Limitations - **Accuracy is carried by common classes** (0.931 vs macro-F1 0.434161; two present classes at 0.0). - **The live service returns a bare class index** (`class_18`), not a human-readable label (`_label_for` falls back to `f"class_{index}"` when `class_labels` is empty). - **Single-label subset** — not comparable to multi-label BigEarthNet numbers. - **Ruling OPEN**; no threshold attached. - **The confidence path floors to 0.0 when there is no prediction or no trained head** — a signal *gap*, not a weak signal. Measured with no checkpoint: `raw = 0.0`, `components = {optical_confidence: 0.333, sar_confidence: 1.0, cross_modal_agreement: 0.0, has_croma: 0.0, trained_head: 0.0, no_prediction: 1.0}`, `degradation_reason: "CROMA is not loaded"`. - **Two assets must be the right two.** `validate_request` rejects 0, 1, 3, and also **two of the same kind** — a plausible mistake that, if accepted, would place a second optical image into the 2-channel SAR slot, zero-fill, run CROMA, and answer about one modality while claiming to fuse two. --- ### 3.4 `grounding` — RemoteCLIP grounding head **Status:** `IMPLEMENTED`, `MEASURED` — **two protocols** and **two decode variants**. #### Identity | Field | Value | Source | |---|---|---| | Manifest `id` | `grounding_head` | `models/manifest.json` | | Path | `artifacts/grounding/remoteclip_grounding_v001/head.pt` | manifest | | Bytes | 12,639,041 | manifest | | sha256 | `93432f7034be91a8ffd9c1a84e3eeec00bed7832c043fe7f83d2be230284c6bb` | manifest, `checksums.sha256` | | HF path | `grounding/head.pt` | manifest | | Architecture | RemoteCLIP ViT-B/32 grounding head (feature_dim 2048, hidden 512) | manifest | | Backbone | `chendelong/RemoteCLIP` (`RemoteCLIP-ViT-B-32.pt`, rev `bf1d8a3ccf2d`) | manifest | | Config hash | `78f1e3700da15aa1` | manifest | | **Parameters** | **1,052,677** | `docs/PHASE8_GROUNDING_HEAD_DECISION.md` | #### Architecture (`specialists/grounding/head.py`) A trainable head over the **frozen** RemoteCLIP ViT-B/32 encoder. Per-cell feature is `concat([patch, text, patch·text, global_pool]) = 4 × 512 = 2048` (enforced at config load — §3.0.2). ``` FROZEN INPUTS (measured, docs/PHASE7_GROUNDING_CONTRACT.md): patch tokens : (B, 49, 512) at 224px -- 7x7 grid, projected dim 512 text emb : (B, 512) ``` The head is `proj: Linear(2048, 512)`, `norm: LayerNorm(512)`, `drop: Dropout(0.10)`, `out: Linear(512, 5)` — five outputs `[tx, ty, tw, th, objectness]`. Forward: `proj → GELU → LayerNorm → Dropout → out`. **Small-std init, and objectness bias low** (`_init_weights`): `proj.weight ~ N(0, 0.02)`, `proj.bias = 0`, `out.weight ~ N(0, 0.01)`, `out.bias = 0` except `out.bias[4] = -2.0`. With one positive cell in 49 the task starts 1:48 imbalanced; a bias of −2.0 starts the objectness prior near 0.12, closer to the truth than 0.5, and stops the first epochs being spent un-learning a saturated sigmoid. **Box parameterisation — cell-relative (YOLO-style).** Cell `(r, c)` of a `grid_h × grid_w` map predicts `[tx, ty, tw, th, obj]`: ``` cx = (c + sigmoid(tx)) / grid_w cy = (r + sigmoid(ty)) / grid_h w = sigmoid(tw) h = sigmoid(th) box = clip((cx - w/2, cy - h/2, cx + w/2, cy + h/2), 0, 1) ``` Cell-relative rather than absolute because 7×7 is coarse: an absolute regressor must learn 49 separate mappings onto the same global coordinates; a cell-relative one only learns a local offset. After clipping, `enforce_order` guarantees `x2 ≥ x1` and `y2 ≥ y1` — without it a decoded box can be inverted, which makes IoU zero and silently kills the gradient. **Positive assignment.** Exactly one cell per target: the one containing the ground-truth box **centre** (standard single-stage-detector convention, YOLO/FCOS). Ties go to the earlier cell so the assignment is deterministic. This makes the 1-of-49 confidence imbalance explicit and therefore correctable. **NMS** is pure torch (no torchvision dependency), so the deployed Space needs no extra package and the behaviour is identical on CPU and GPU. **Loss** (`grounding_loss`): `box = L1(pos_boxes, targets)` (weight 0.5), `giou = 1 - GIoU(pos_boxes, targets)` (weight 0.3), and `conf = BCE_with_logits(obj_logits, conf_targets, weight=weights)` (weight 0.2) where the single positive cell is up-weighted by `positive_confidence_weight = 20.0`. Box and GIoU are computed on the **decoded** coordinates of the single positive cell per sample, so the loss is computed on exactly the coordinates the metric measures. GIoU is used alongside L1 because L1 alone is scale-blind. `LossBreakdown` keeps the components separate so a collapse in one is visible in the logs. #### Hyperparameters — every value From `configs/base.yaml` (`grounding:`, `grounding_head:`, `grounding_training:`): | Key | Value | |---|---| | `grounding.checkpoint_repo` | `chendelong/RemoteCLIP` | | `grounding.checkpoint_file` | `RemoteCLIP-ViT-B-32.pt` | | `grounding.checkpoint_revision` | `bf1d8a3ccf2d` | | `grounding.model_name` | `ViT-B-32` | | `grounding.image_size` | **224** | | `grounding.resolution_frozen` | true | | `grounding.nms_iou` | 0.50 | | `grounding.max_candidates` | 20 | | `grounding.confidence_threshold` | 0.40 | | `grounding.benchmark_box_scale` | 100.0 | | `grounding.coordinate_system` | `normalized_0_1` | | `grounding.encoder_projected_dim` | 512 | | `grounding_head.hidden_dim` | 512 | | `grounding_head.dropout` | 0.10 | | `grounding_head.feature_dim` | 2048 | | `grounding_head.positive_confidence_weight` | **20.0** | | `grounding_head.decode` | `cell_relative` | | `grounding_training.learning_rate` | 0.0001 | | `grounding_training.batch_size` | 16 | | `grounding_training.epochs` | 20 | | `grounding_training.weight_decay` | 0.0001 | | `grounding_training.warmup_ratio` | 0.05 | | `grounding_training.grad_clip` | 1.0 | | `grounding_training.val_fraction` | 0.10 | | `grounding_training.num_workers` | 2 | | `grounding_training.save_every_steps` | 500 | | `grounding_training.box_loss_weight` | 0.5 | | `grounding_training.giou_loss_weight` | 0.3 | | `grounding_training.confidence_loss_weight` | 0.2 | **Why `positive_confidence_weight = 20.0`.** Objectness BCE sees ~1 positive cell out of 49. Unweighted, the optimum is "no object" everywhere; the weight is what stops that collapse. #### Training data VRSBench. The head trains on the train split and is evaluated on the VRSBench **eval** split, n = **16,159** records, scored once, no sampling. VRSBench stores boxes normalised to **0–100**; this project stores boxes normalised to **0–1**, via the declared `benchmark_box_scale: 100.0` so the conversion cannot be applied twice or forgotten. **Validation trajectory** (20 epochs, CPU, ~37 s/epoch) — `docs/PHASE8_GROUNDING_HEAD_DECISION.md`: ``` epoch 1/20 loss 0.4284 box 0.0641 giou 0.7948 conf 0.7897 val_iou 0.0436 epoch 5/20 loss 0.3864 box 0.0489 giou 0.7581 conf 0.6725 val_iou 0.0673 epoch 10/20 loss 0.3653 box 0.0446 giou 0.7313 conf 0.6179 val_iou 0.0831 epoch 15/20 loss 0.3488 box 0.0414 giou 0.7062 conf 0.5811 val_iou 0.0926 epoch 20/20 loss 0.3419 box 0.0401 giou 0.6937 conf 0.5685 val_iou 0.0943 ``` All four loss components move monotonically in the right direction. `conf` falls from an over-confident 0.79 as the positive-cell weighting corrects the objectness prior; `box` and `giou` fall together, which is the healthy pattern (a falling `box` with a rising `giou` would mean the box is drifting in size). **Validation IoU (0.0943 at epoch 20) is much lower than eval IoU (0.2566).** These are not comparable and the gap is expected: validation is a held-out slice of the **train** split, which has noisier ground truth (29.5 % of its boxes are out-of-range and filtered, and the surviving ones come from a different annotation pass). This is a property of VRSBench, not a bug. #### Evaluation protocol — two protocols × two decode variants `artifacts/grounding/remoteclip_grounding_v001/eval_result_canonical.json` (top_k = 20, the config default) and `…/eval_result_matched6.json` (top_k = 6, decode-matched to the baseline). Both record `n_eval_records: 16159`, `resolution: 224`, `grid: 7`, `limited_run: false`, `frozen_config_evaluation: true`, device `cpu`, torch `2.14.0+cpu`, and `config_hash: 78f1e3700da15aa1`, `config_drift: false`. **The `head_decode` block is recorded in each artifact** — `config_default_top_k: 20`, `nms_iou: 0.5`, `score_threshold: 0.4`, `top_k: 20` (canonical) or `top_k: 6` (matched6). Without it the artifact could not say which setting produced its number, and a re-run at the config default would silently yield a different figure. #### Measured numbers **The trained head, under the two protocols:** | Protocol | mean best IoU | recall@0.5 | Artifact key path | |---|---|---|---| | canonical (head threshold decode, top_k = 20) | **0.2838** | **0.2198** | `results.head_threshold.mean_best_iou` / `results.head_threshold.recall.0.50` | | matched6 (head threshold decode, top_k = 6) | **0.2566** | **0.1938** | `results.head_threshold.mean_best_iou` / `results.head_threshold.recall.0.50` | **Two decode variants, reported for completeness** (a reviewer must see the whole grid, not one cell): | Variant | mean best IoU | recall@0.10 | recall@0.25 | recall@0.50 | |---|---|---|---|---| | head argmax decode (canonical) | **0.1215** | 0.3183 | 0.2088 | 0.0795 | | zero-shot matched (no trained head, canonical) | **0.0972** | 0.3298 | 0.1188 | 0.0234 | **Full canonical grid** (`eval_result_canonical.json` → `results`): | Strategy | `mean_best_iou` | `recall.0.10` | `recall.0.25` | `recall.0.50` | `latency_ms_per_image` | `seconds` | |---|---|---|---|---|---|---| | `head_argmax` (strategy `argmax`) | 0.1215 | 0.3183 | 0.2088 | 0.0795 | 0.655 | 10.6 | | `head_threshold` (strategy `threshold`, top_k 20) | 0.2838 | 0.6882 | 0.5047 | 0.2198 | 2.205 | 35.6 | | `zero_shot_matched` | 0.0972 | 0.3298 | 0.1188 | 0.0234 | — | 17.9 | **Full matched6 grid** (`eval_result_matched6.json` → `results`): `head_argmax` identical (0.1215); `head_threshold` (top_k 6) 0.2566 / 0.6315 / 0.4545 / 0.1938 at 2.158 ms; `zero_shot_matched` identical (0.0972). Each artifact also carries `phase7_reference: {mean_best_iou: 0.0972, recall_at_0.50: 0.0234, source: "docs/PHASE7_RESOLUTION_DECISION.md"}`. **Decode-matched delta (head threshold top_k = 6 vs zero-shot): +0.1594 IoU, +0.1704 Recall@0.50.** The pre-registered bar was `MIN_IMPROVEMENT_IOU = 0.02`; the measured margin is **8× the bar**. **`head_argmax` vs zero-shot is NOT apples-to-apples.** 1 box against ~6 boxes flatters the head, because mean best IoU takes the max over predictions. It is reported because it is the number directly comparable to the Phase 7 resolution experiment's zero-shot argmax, not because it decides anything. **A defect this phase had to fix first** (`docs/PHASE8_GROUNDING_HEAD_DECISION.md`). The evaluation script originally built its own single-box baseline with `argmax_candidate`, while Phase 7 measured the baseline through `ground_phrase` (threshold box + up to 5 local maxima). Same 16,159 records, same metric, same cached features — **different decode**: Phase 7 via `ground_phrase` = 0.0972, eval via `argmax_candidate` = 0.0092, a 10× gap. The eval printed `head beats zero-shot: True (+0.1123)` when the matched comparison was +0.0243. Both cleared the bar, but only the second is a claim about the head rather than about the decode. **Fix:** the decode was extracted into one function both paths call — `specialists/grounding/inference.py::decode_candidates_from_features` — so there is exactly one place that turns similarity into boxes. Verification that the fix is real, not cosmetic: re-running the eval reproduced the Phase 7 number exactly (`|difference| = 0.0000`). **A second defect: candidate count changes the number.** Mean best IoU is a max over predictions, so emitting more boxes raises it mechanically. The head's threshold decode defaulted to `max_candidates: 20` while the baseline emits 5.99 — an uncontrolled asymmetry in the head's favour. Fix: `--head-top-k` and `--head-score-threshold` overrides, plus the recorded `head_decode` block. Measured cost of the cap: top_k 20 → 0.2838 (+0.1866 vs zero-shot); top_k 6 → 0.2566 (+0.1594). Capping to the baseline's own budget costs 0.027 IoU — **the win survives.** #### Acceptance status `MEASURED`. Phase 8 verdict: **the trained head beats the zero-shot baseline**; Phase 8 is done. The head is **not yet wired into the specialist** as of the Phase 8 decision record (`specialists/grounding/inference.py` still exposes only the zero-shot path), and that integration step must not change the zero-shot module's behaviour because the resolution experiment depends on it. #### Limitations - **Absolute IoU is low.** 0.2838 / 0.2566 is not "solved". - **Protocol-sensitive.** 0.2838 (canonical) vs 0.1215 (argmax) — absolute values depend on decode. - **Localisation floor.** At 224 each token covers 1/7 (~0.143) of the image width; at 448 it is 1/14. - **`head_threshold` at top_k = 20 was not tuned.** 20 is the config default, not a validation-selected optimum. Selecting it on eval would be benchmark tuning. - **Nothing about the hidden distribution.** VRSBench is overhead optical; the hidden set is Cartosat-2S + RISAT. - **Not a system-level result.** "Beats the zero-shot baseline on VRSBench eval" ≠ "performs well on the hidden ISRO/SAC set". --- ### 3.5 `router` — intent adapter over frozen MiniLM **Status:** `IMPLEMENTED`, `MEASURED`, **TEST NOT RUN**. A **~50,822-parameter adapter** over the frozen MiniLM encoder. Because the encoder is frozen, embeddings are cached and the adapter trains on cached vectors — no GPU required (measured: 20 epochs / 4,096 vectors in 0.28 s on CPU). #### Identity | Field | Value | Source | |---|---|---| | Manifest `id` | `router_adapter` | `models/manifest.json` | | Path | `artifacts/router/router_adapter_v001/adapter.pt` | manifest | | Bytes | 211,961 | manifest | | sha256 | `8527c3ed28a293e13293d48601d48e3ceafa137b9acabddaf5de31a58a509b5c` | manifest, `checksums.sha256` | | HF path | `router/adapter.pt` | manifest | | Architecture | task/modality adapter over frozen MiniLM embeddings (~50,822 params) | manifest | | Backbone | `sentence-transformers/all-MiniLM-L6-v2` (rev `1110a243fdf4`) | manifest | | Config hash | `78f1e3700da15aa1` | manifest | | Parameters | `UNKNOWN — not established from the available evidence` (manifest records `null`) | manifest | > **Parameter-count discrepancy — flagged, not smoothed over.** `specialists/router/adapter.py`'s > docstring and `models/manifest.json` both state **~50,822** parameters. The Phase 4 completion > report (`docs/PHASE4_ROUTER_REPORT.md`) states **51,725** adapter parameters on a frozen > 22,713,216-param encoder and an artifact size of 223.8 KB, whereas the released artifact is > 211,961 bytes. These two figures were produced at different times and have not been reconciled. > The release documents the **manifest** figure (50,822) as the shipped number; which figure is > authoritative is > `UNKNOWN — not established from the available evidence`. #### Architecture (`router/adapter.py`) The only trainable part of the router. It sits on top of the frozen MiniLM embedding and emits **five heads**: ``` embedding (384) | LayerNorm | Linear(384 -> hidden_dim) default hidden_dim = 128 | GELU | Dropout | +--> task_head Linear(hidden, 6) +--> modality_head Linear(hidden, 4) +--> temporal_head Linear(hidden, 1) logit +--> spatial_head Linear(hidden, 1) logit +--> language_head Linear(hidden, 1) logit ``` `_init_weights` uses small-std init (`std = 0.02`) with zero bias on every head, keeping the initial sigmoid near 0.5 — without it the binary heads can start saturated and BCE gradients vanish before the task head has learned anything useful. `forward` asserts the input is 2-D and that `embeddings.shape[1] == input_dim`, and requires embeddings already detached from the frozen encoder (the adapter does not back-propagate into MiniLM). #### Hyperparameters — every value From `configs/base.yaml` (`router:`): | Key | Value | |---|---| | `model` | `sentence-transformers/all-MiniLM-L6-v2` | | `revision` | `1110a243fdf4` | | `max_length` | 128 (asserted ≤ 256) | | `embedding_dim` | 384 | | `device` | `auto` | | `hidden_dim` | 128 | | `dropout` | 0.10 | | `confidence_threshold` | 0.70 | | `num_tasks` | 6 | | `tasks` | `[vqa, caption, grounding, change, optical_sar, unsupported]` | | `training.epochs` | 60 | | `training.batch_size` | 64 | | `training.learning_rate` | 0.001 | | `training.weight_decay` | 0.01 | | `training.task_loss_weight` | 1.0 | | `training.modality_loss_weight` | 0.3 | | `training.binary_loss_weight` | 0.5 | | `training.val_ratio` | 0.15 | | `training.hard_negatives_to_test` | true | **Splits are by GROUP (template / hard-negative family), never by example.** Template-generated queries are near-duplicates; splitting by example would put `"Show me the water body."` in train and `"Show me the road."` in val — one token apart — and report a fake accuracy. Hard-negative families are placed in the **test** split so their accuracy measures generalisation, not memorisation. #### Training data A **synthetic** corpus: 576 examples, 54 groups, hand-written and templated (`threshold_sweep_val.json`: `corpus_total: 576`, `corpus_groups: 54`, `corpus_limited: true`). Split sizes: `train 410`, `val 86`, `test 80`. The plan's minima are `plan_min_val_queries: 500` and `plan_min_hard_negatives: 100`; the corpus is well below both, and the artifact says so. #### Evaluation protocol and measured numbers `artifacts/router/threshold_sweep_val.json`. The sweep iterates thresholds 0.50 → 0.99 and records coverage, covered-task accuracy, fallback rate and `n_covered` per threshold. `select_by: "covered_accuracy"`; `selected.threshold: 0.76` (coverage 0.790698, covered-task accuracy 1.0); `shipped_threshold: 0.70` (coverage 0.848837, covered-task accuracy 0.972603). | Field | Value | Exact key path | |---|---|---| | **overall ungated task accuracy** | **0.965116** | `overall_ungated_accuracy` | | split | `val` | `split` | | n val | **86** | `n_val` | | n val examples scored | 86 | `n_val_examples_scored` | | n test examples scored | **0** | `n_test_examples_scored` | | `test_split_touched` | **false** | `test_split_touched` | | `corpus_limited` | true | `corpus_limited` | | `hard_negatives_in_val` | **0** | `hard_negatives_in_val` | | `val_min_support` | 8 | `val_min_support` | | `val_task_counts` | `{caption: 8, change: 20, grounding: 14, optical_sar: 10, unsupported: 19, vqa: 15}` | `val_task_counts` | | `split_sizes` | `{train: 410, test: 80, val: 86}` | `split_sizes` | | `delta_vs_shipped.coverage` | −0.0581 | `delta_vs_shipped.coverage` | | `delta_vs_shipped.covered_task_accuracy` | 0.0274 | `delta_vs_shipped.covered_task_accuracy` | | adapter encoder params | 22,713,216 | `adapter_encoder.parameters` | | `adapter_config_hash` | `615478910dc266bf` | `adapter_config_hash` | | `config_hash` | `78f1e3700da15aa1` | `config_hash` | **This number is (a) validation-only, (b) ungated, and (c) small (n = 86).** The artifact's own note is blunt: *"corpus-limited: val n=86 vs plan >=500. This is NOT a calibration — the corpus is synthetic and too small (min per-class support 8, caption) and val carries 0 hard negatives (hn_* families are held out to TEST by design). Selecting a threshold here yields a justified default, not a calibrated value. The corpus was NOT padded with generated queries. Backlog P1-9's 'n=80' is the TEST split; the sweep target is val n=86. **The test split was NOT touched.**"* **The router TEST split was NOT RUN.** Do not read 0.965116 as a test result. The sweep scored 86 validation examples and zero test examples, and `test_split_touched` is `false`. > **Historical note, recorded so the two are not conflated.** The Phase 4 completion report > (`docs/PHASE4_ROUTER_REPORT.md`, verified 2026-09-16, Gate 2 PASS) records an **earlier** gate-2 > evaluation that included a test column — test task accuracy **0.975**, macro F1 0.976, per-class test > recall (vqa 1.000/n=15, caption 0.846/n=13, grounding 1.000/n=17, change 1.000/n=13, optical_sar > 1.000/n=12, unsupported 1.000/n=10) and hard-negative accuracy **0.800**. That evaluation predates > the shipped threshold sweep, and its own standing caveats are that the corpus is synthetic and that > *"the 0.975 headline is partly earned on templates the split kept in training. Treat the router as > working, not as benchmarked."* The release's position for the shipped artifact is **TEST NOT RUN**; > the Phase 4 numbers are retained as a historical record, not promoted to a release benchmark. #### Acceptance status `MEASURED`; the headline is validation, ungated, n = 86. No test result is claimed. #### Limitations - **Corpus-limited** (n = 86 val; plan minimum 500; hard negatives in val = 0 by design). - **Not a calibration** — the threshold is a justified default, not a calibrated value. - **Synthetic corpus** — 576 hand-written/templated examples. - **`caption` has the smallest support** (8 in val), the tightest per-class floor. - **Known misroutes** are catalogued in [`LIMITATIONS.md`](LIMITATIONS.md) §2 (e.g. *"What is the new runway?"* reads `change`, not `vqa`; *"How much built-up area was added?"* under-triggers to `vqa`). - **Threshold 0.70 is uncalibrated** — it gates the lexical fallback. --- ### 3.6 `vlm` — SmolVLM LoRA adapter (USABLE_VERIFIED, ACCEPTANCE-REJECTED) **Status:** `IMPLEMENTED`, `MEASURED`, **ACCEPTANCE-REJECTED**. > This is §3.6 because [`TRAINING.md`](TRAINING.md) §7 cross-references it. The section number is kept > stable so that link resolves. A PEFT LoRA adapter on frozen `HuggingFaceTB/SmolVLM-500M-Instruct`. **The artifact's status is `CLOSED` with headline `ACCEPTANCE-REJECTED`.** This is not a contradiction — it is the project's central truthfulness distinction: **`USABLE_VERIFIED`** (the adapter demonstrably works; the metrics are real and reproducible) and **`ACCEPTANCE-REJECTED`** (it is *not accepted* for production promotion). **USABLE ≠ ACCEPTED.** The deployed caption/VQA path therefore uses the **unadapted** SmolVLM. #### Identity | Field | Value | Source | |---|---|---| | Manifest `id` | `vlm_lora_adapter` | `models/manifest.json` | | Path | `.scratch/phase6_real_adapter/phase6_adapter/adapter_model.safetensors` | manifest | | Bytes | 34,798,048 | manifest | | sha256 | `07c76a75fa04624880ed7730590f5fdd7b145a8232e3c0af411c3c545a5adf5e` | manifest, `checksums.sha256` | | HF path | `vlm/adapter_model.safetensors` | manifest | | Architecture | PEFT LoRA (r=16, alpha=32, dropout=0.05) on text_model projections | manifest | | Base model | `HuggingFaceTB/SmolVLM-500M-Instruct` (rev `a7da5b986cb5`) | manifest, closure | | `kind` | `bigearthnet_smolvlm_lora` | closure → `production_adapter.kind` | | **tree sha256** | `5c6b86317d1e65962702dc9e377009b3df41cc13de1b15bceccb70ad977775e7` | closure → `adapter_tree_sha256` | | **file sha256** | `07c76a75fa04624880ed7730590f5fdd7b145a8232e3c0af411c3c545a5adf5e` | closure → `weights_file_sha256` | | Trainable params | **8,683,520** (1.6823 % of 516,165,824) | closure → `trainable_params` | | Config hash | `78f1e3700da15aa1` | manifest | | `retrained_for_closure` | **false** | closure | | `modified_for_closure` | **false** | closure | #### Architecture and target modules LoRA, `peft_type=LORA`, `r=16`, `alpha=32`, `dropout=0.05`, targeting `model.text_model.*.{q,k,v,o,gate,up,down}_proj` — `lora_target_module_count: 224`. **The vision-tower hazard did not occur.** `trainable_subtrees` is exactly `{"model.text_model": 8683520}`; the vision model (86,433,024) and connector (11,796,480) appear in `frozen_params`, which are: | Frozen subtree | Parameters | |---|---| | `model.connector` | 11,796,480 | | `model.text_model` | 361,944,000 | | `model.vision_model` | 86,433,024 | | `other` | 47,308,800 | The `trainable_params` figure was measured two independent ways (closure → `trainable_params.how`): summing `numel` over every tensor in `adapter_model.safetensors` read from the safetensors header (no model load), and loading through the production path and reading `sum(p.numel())` then subtracting the manifest frozen-params sum. Both give 8,683,520; `manifest_matches_measurement: true`. #### Hyperparameters — every value From `configs/base.yaml` (`training:`) and the closure record: | Key | Value | |---|---| | `training.precision` | **fp16** (finding C-6: T4 is SM 7.5 → fp16, NOT bf16) | | `training.vlm_batch_size` | 2 | | `training.vlm_gradient_accumulation` | 8 | | `training.vlm_learning_rate` | 0.0002 | | `training.vlm_epochs` | 1 | | `training.lora_rank` | 16 | | `training.lora_alpha` | 32 | | `training.lora_dropout` | 0.05 | | `training.weight_decay` | 0.01 | | `training.warmup_ratio` | 0.05 | | `training.gradient_checkpointing` | true | | `training.save_every_steps` | 500 | | PEFT version | **0.19.1** | | Seed | 42 | | `precision_recorded` | fp16 | #### Training data BigEarthNet instruction pairs (the same 28,000-patch local subset used for the fusion head), on an external GPU (Kaggle T4-class). This is the only artifact that requires a GPU to train. #### Evaluation protocol `artifacts/vlm/phase6_closure.json` (generated by `scripts/phase6_close.py`, which reads the evidence rather than restating it). The evaluation is on a **frozen 1,000-question subset** of the test split. The subset's identity was proven **without a model** (Gate A″): `available_per_split {val: 6750, test: 7772}`, subset `n = 1000`, and the test per-class counts an **exact** match to the recovery subset (19 classes, sum 1000), `config_hash 78f1e3700da15aa1` identical to Run 1's. Gate D reproduced Run 1's adapted-test control **exactly** — loading the local adapter and evaluating it returned `exact_match 0.963`, `n 1000`, confusion `{tp:500, fp:19, tn:463, fn:18}`, `f1 0.9643201542912246` — identical to Run 1's recorded values, which proves the local artifact *is* Run 1's adapter and that CPU/fp32 reproduces the Kaggle T4 endpoint. #### Measured numbers — the "usable" side `artifacts/vlm/phase6_closure.json` → `why_usable_verified.adapted_test`: | Metric | Value | Exact key path | |---|---|---| | **exact_match** | **0.963** | `why_usable_verified.adapted_test.exact_match` | | **F1** | **0.96432** | `why_usable_verified.adapted_test.f1` | | n | 1000 | `…adapted_test.n` | | precision | 0.963391 | `…adapted_test.precision` | | recall | 0.965251 | `…adapted_test.recall` | | tp / fp / tn / fn | 500 / 19 / 463 / 18 | `…adapted_test.confusion` | | aggregate test delta | **+49.50 pp** (46.80 → 96.30 pp) | `why_usable_verified.aggregate_test_delta_pp` | Per-class accuracy on the adapted test subset ranges from 0.8 (Agro-forestry areas, n=20) to 1.0 (several classes); `Mixed forest` 0.878788 (n=33), `Permanent crops` 0.862069 (n=29), `Broad-leaved forest` 0.898551 (n=69), `Pastures` 0.916667 (n=132), `Marine waters` 1.0 (n=329). #### Measured numbers — the "rejected" side `artifacts/vlm/phase6_closure.json` → `why_acceptance_rejected`. Decision split: **test**. Rule version: **v002**. | | | |---|---| | **V1** | **passes** — test `46.80 → 96.30 pp`, delta **+49.50 pp** (required ≥ +5.00) | | **V2** | **fails** — **Mixed forest**: `n=33`, `100.00 → 87.8788 pp`, `drop 12.1212 pp`, `lost_questions 4`, `z 2.1335` | It fails **both** halves of v002 (`lost ≥ 4` **and** `z ≥ 1.96`). Per item V, a complete run that fails V2 is `REJECTED`. Thresholds used (`why_acceptance_rejected.thresholds_used`): `accept_min_delta_pp: 5.0`, `accept_min_delta_pp_ceiling: 2.0`, `ceiling_baseline_pp: 95.0`, `max_class_drop_pp: 1.0` (legacy v001), `min_class_questions: 20`, `test_val_disagreement_pp: 10.0`; `v2_criterion`: `class_drop_z: 1.96`, `min_class_drop_questions: 4`, `min_class_questions: 20`, with `se_formula: "sqrt((p_b*(1-p_b) + p_a*(1-p_a)) / n)"` and `z_formula: "(baseline - adapted) / se"`. **The rejection is narrow.** `n_classes_failed: 1`, `n_classes_held: 5`, `n_classes_improved: 11`, `n_classes_total: 19`. The next-worst class (Inland wetlands, −6.6667 pp, n=30) lost only 2 questions and therefore sits below V2's materiality floor. **It is not a split artefact.** The same class also degraded on **val** in Run 1 (drop 6.4516 pp, n=31) — the very value that motivated v001's flag. The adapter hurts Mixed forest on **both** splits, so this is a property of the adapter, not an accident of one subset. Mixed forest also sits at a 100.00 pp baseline on test, so any loss is a drop from the ceiling. **Residual risk, reported not resolved** (`residual_risk`): the verdict rests on **4 questions in one class of 33** — the unfloored minimum-size exposure recorded at `PHASE6_AUDIT_AND_CONTRACT.md` §8.6. With no `n ≥ N` floor in V2, a 33-question class can flip the verdict of a run whose aggregate endpoint improved by 49.5 pp. **BERTScore is unavailable**, not zero: `thresholds_used.bertscore.available: false`, reason `roberta-large` is not in the local HuggingFace cache. BLEU/ROUGE are **excluded** because target answers are one token and they are meaningless at that length. #### Both rejection records are preserved Neither is rewritten. A future reader must be able to see what each rule said, on which split, at the time it said it (`preserved_records`): | Record | Rule | Split | Verdict | Source | |---|---|---|---|---| | Run 1's own manifest | `v001` | val | **`REJECTED`** (3 classes) | Run 1's original manifest, preserved verbatim as `preserved_records.v001_val_rejected` in `phase6_closure.json` | | Test-split adjudication | `v002` | **test** (independent) | **`REJECTED`** (1 class) | `artifacts/vlm/run1_test_recovery/test_adjudication.json` | | Recovery manifest | `v002` | val | `ACCEPTED` — **not final acceptance** | `artifacts/vlm/run1_test_recovery/run_manifest.json` | **v001 on val** failed three classes — Mixed forest (6.4516 pp, n=31), Transitional woodland shrub (9.375 pp, n=32), Agro-forestry areas (4.6512 pp, n=43) — with `val_delta_pp 42.0` and `test_delta_pp null`. **The val-split `ACCEPTED` under v002 is recorded for completeness only and is NOT final acceptance** — it decides on the *same val subset that motivated v002*, which `docs/PHASE6_RUN1_REJECTION_DIAGNOSIS.md` §7.4 condition 3 forbids. On data that did not motivate the relaxation, the run does not pass. #### Acceptance status — and what closure does NOT claim `status: CLOSED`; `headline: "Phase 6 is closed. The Run 1 LoRA adapter is promoted to the production VLM adapter: USABLE and VERIFIED, but ACCEPTANCE-REJECTED."` The record keeps verified and accepted separate: *"'Verified' answers: is this artifact the one we trained, and does it work? 'Accepted' answers: did it clear the bar predeclared before we looked? Both are true, and they are different questions."* `what_closure_does_not_claim`: - It does **not** claim Run 1 was accepted — it was rejected by v001 on val and by v002 on the independent test split. - It does **not** claim the Mixed forest regression is resolved. - It does **not** claim a new adapter exists or is planned. - It does **not** alter the v002 rule or its verdict. **Forward rule** (`forward_rule`): any future improved adapter MUST be a **new** experiment/version. It MUST NOT rewrite, amend, or supersede Run 1's records; it must not retrain or modify the Run 1 adapter in place; it must not report a new adapter's metrics under Run 1's identity; it must not edit v001's or v002's recorded verdicts to match a later outcome. Precedent: v002 itself followed this rule. #### How it is enabled, and the two traps The adapter is attached via an **environment setting, not a code change**: `specialists/vqa/model.py` resolves it in the order explicit `adapter_path` argument → **`SATQUERY_VLM_ADAPTER`** (`ADAPTER_ENV_VAR`) → none, then attaches it with `PeftModel.from_pretrained`. A load failure raises rather than silently serving the base model: an inference result attributed to an adapter that did not actually attach is worse than a hard failure. Two traps a future reader will hit (`known_traps`): 1. **`adapter_sha256` names two different values.** `training/vlm/artifact.py` computes a **tree hash** over the `{relpath: sha256}` weight map (`5c6b8631…`), while `specialists/vqa/model.py::_adapter_sha256` computes the **file** sha256 of `adapter_model.safetensors` (`07c76a75…`). Recomputing one and comparing it to the other yields a false "the artifact was altered" conclusion. 2. **The promoted adapter is NOT `checkpoint-2000`.** The three weight files have three different digests: top-level `07c76a75…`, `checkpoint-1500` `7273588e…`, `checkpoint-2000` `bf249943…`. So "just use the last checkpoint" is not equivalent to this artifact. #### Limitations - **Acceptance-rejected** — the deployed path uses the unadapted model. - **Residual risk** — the verdict rests on 4 questions in one class of 33. - **`Mixed forest` regression unresolved**; it degrades on both val and test. - **The path lives under `.scratch`** — `.gitignore` excludes `artifacts/`, `checkpoints/` and `*.safetensors`, so the ~105 MB local directory is not committed; a future cleanup could remove it. It is reconstructible from `phase6_realbundle.zip` and verifiable against the two digests above. - **BERTScore not computed** (unavailable offline); BLEU/ROUGE excluded as meaningless at one-token answers. --- ## 4. Rejected, deferred and open model decisions | Decision | Outcome | Evidence | |---|---|---| | Grounding image resolution 448 vs 224 | **224 chosen; 448 REJECTED** | `docs/PHASE7_RESOLUTION_DECISION.md`; §4.1 below | | VLM adapter promotion | **REJECTED** | metrics usable, acceptance rejected (§3.6) | | Calibration | **kept but ineffective** | ECE worsened (§5) | | Optical-SAR ruling | **OPEN** | no decision recorded; `is_deciding_statistic: false` | | Change-VQA ruling | **OPEN** | `verification.metric_ruling` | | CROMA constructor (`use_croma.PretrainedCROMA`, vendored) | **ACCEPTED** as an implementation detail | `docs/PHASE14_OPTICAL_SAR_DECISIONS.md` §2 (DEV-1) | | CROMA input normalisation | **RESOLVED** 2026-09-18: option (a) adopted; (c) gated experiment; (b) rejected | `docs/PHASE14_CROMA_NORMALISATION_CHANGE.md` (DEV-2) | | CROMA patch count (225) | **CONSISTENT but UNVERIFIED** upstream; checked at load time | `docs/PHASE14_OPTICAL_SAR_DECISIONS.md` §4 (DEV-3) | | `sa_mode: BAM` | **declared, NOT IMPLEMENTED** — raises rather than aliasing PAM | `specialists/change/stanet.py` | | Change head wired into serving by default | **DEFERRED** — registry override, not config | `specialists/change/specialist.py` | | Grounding head wired into `inference.py` | **DEFERRED** (Phase 8 integration step) | `docs/PHASE8_GROUNDING_HEAD_DECISION.md` | | `optical.normalization` / `sar.representation` config keys | **OPEN** — read by no code | `docs/PHASE14_OPTICAL_SAR_DECISIONS.md` §6 item 6 | ### 4.1 The grounding resolution experiment — 448 REJECTED **Rule, fixed before the result was seen** (`docs/PHASE7_RESOLUTION_DECISION.md`; recorded in the artifact as `rule_changed_since_preregistration: false`): ``` 448 WINS if Recall@0.5 improves by >= 0.05 absolute OR mean best IoU improves by >= 0.05 absolute 224 WINS otherwise INCONCLUSIVE if fewer than 30 samples were scored ``` **Run:** full VRSBench eval split, **16,159 / 16,159 records scored at both resolutions**, Tesla T4, `--all --device cuda --tag full`. **Result: 224 WINS.** ``` recall@0.5 gain 448/224 : -0.0022 bestIoU gain 448/224 : -0.0147 latency ratio : 1.59x ``` Neither component came close to the +0.05 margin. Both were **negative**. | Metric | 224 | 448 | delta | |---|---|---|---| | token grid | 7 × 7 = 49 | 14 × 14 = 196 | 4.0× tokens | | attention cost (n²) | 1× | 16× | — | | with boxes | 16159/16159 | 16159/16159 | — | | **mean best IoU** | **0.0972** | 0.0825 | **−0.0147** | | Recall@0.10 | **0.3298** | 0.2599 | **−0.0699** | | Recall@0.25 | **0.1187** | 0.0944 | **−0.0243** | | Recall@0.50 | **0.0234** | 0.0212 | **−0.0022** | | matched IoU | 0.0972 | 0.0825 | −0.0147 | | latency mean | **20.0 ms** | 31.8 ms | **1.59×** | | latency p90 | **20.9 ms** | 32.9 ms | 1.57× | | peak VRAM | **592.1 MB** | 599.8 MB | +7.7 MB | | wall time | **~8.5 min** | ~11.2 min | 1.32× | **448 is worse on every quality metric and slower. There is no axis on which it wins.** **Best-IoU distribution** — a whole-distribution move toward the zero-overlap bucket, not a tail effect: | bucket | 224 | 448 | |---|---|---| | 0.00–0.10 | 10,829 | 11,957 | | 0.10–0.25 | 3,412 | 2,677 | | 0.25–0.50 | 1,540 | 1,182 | | 0.50–0.75 | 336 | 307 | | 0.75–1.01 | 42 | 36 | **Paired analysis — independent confirmation.** Both resolutions scored the **same 16,159 samples**, so the paired test is the stronger statistic: it removes between-object variance. ``` paired samples : 16159 mean 224 : 0.0972 mean 448 : 0.0825 mean paired diff : -0.0147 (95% CI -0.0160 .. -0.0134) t statistic : -22.63 CI excludes zero : True 448 better on : 1371/16159 ( 8.5%) 448 worse on : 3372/16159 (20.9%) identical : 11416/16159 (70.6%) ``` **The pre-registered rule and the paired test AGREE on 224.** There is no rule-versus-evidence disagreement to escalate: both say 224, and the confidence interval excludes zero by a wide margin. The win/loss split is also informative — 448 wins on only 8.5 % of records and loses on 20.9 %; the finer grid is not merely neutral, it is **actively harmful** on a fifth of the corpus. **Paired recall ladder** — the gap **narrows as the threshold rises**, the signature of a method that cannot reach high IoU either way: | threshold | 224 | 448 | diff | 95% CI | |---|---|---|---|---| | 0.10 | 0.3298 | 0.2599 | −0.0699 | excludes zero | | 0.25 | 0.1187 | 0.0944 | −0.0243 | excludes zero | | 0.50 | 0.0234 | 0.0212 | −0.0022 | excludes zero | **Why 448 did not help — the honest reading.** The zero-shot method selects a patch by text similarity and returns that patch's box. At 224 a box is 1/7 of the image; at 448 it is 1/14. A finer grid is only better if the target is small **and** the similarity peak lands on the correct fine cell. Two things work against that here: (1) the peak is not sharper at 448 — splitting each cell into four gives four chances to pick a wrong sub-cell, and the similarity field on frozen features is smooth, so the argmax moves around (losses outnumber wins 2.5 : 1); (2) **Recall@0.10 drops the most (−0.0699)** — if finer tokens genuinely localised better, the *loosest* threshold would benefit most. It degrades most, which means the fine grid adds positional noise rather than positional precision. **This is the zero-shot baseline's limitation, not a property of RemoteCLIP.** **Degeneracy notes: none.** At n=12, n=40 and n=6 the smoke runs reported `Recall@0.5 = 0.0000` at **both** resolutions and emitted a degeneracy warning. At full scale the metric is non-zero (0.0234 / 0.0212), so the note correctly did not fire — a genuine sub-floor artefact that the full run resolved, which is why the sub-floor runs were never treated as evidence. **What this establishes:** grounding runs at 224, frozen in `configs/base.yaml` (`grounding.image_size: 224`, `grounding.resolution_frozen: true`); peak VRAM for the frozen encoder at 224 is 592 MB (inside the ZeroGPU free tier and a T4); encoder latency at 224 on a T4 is 20 ms/image; the 224 localisation floor is 1/7 of image width per token. **What this does NOT establish:** whether the zero-shot baseline is good (it is not — 0.0972 / 0.0234 are weak, an ablation floor for the Phase 8 head); whether a **trained** head has the same resolution sensitivity (re-opening the question after Phase 8 is legitimate *if* the head's validation curve suggests it, and would be a new pre-registered experiment, not a silent retune); anything about hidden ISRO/SAC imagery. --- ## 5. Calibration — a measured negative result **Status:** `MEASURED`; the calibration is **not an improvement**. Temperature scaling is enabled (`confidence.temperature_scaling: true`) with `artifacts/calibration_v001.json` (`schema: calibration_v1`). It is scoped to the **R-02 change-VQA head's answer confidence only** — *other specialists emit their own raw scores and are unaffected* (`scope.note`). | Field | Value | Exact key path | |---|---|---| | **temperature** | **0.9772731820958189** (≈ 0.9772732) | `temperature_scaling.temperature` | | fitted on | `Val` | `temperature_scaling.fitted_on` | | n samples | 16,441 | `temperature_scaling.n_samples` | | method | `temperature_scaling` | `provenance.method` | | objective | `mean_negative_log_likelihood` | `provenance.objective` | | optimizer | `golden_section_on_log_temperature` | `provenance.optimizer` | | space | `multiclass_logits` | `provenance.space` | | iterations | 200 | `provenance.iterations` | | `log_temperature` | −0.022989052824434128 | `fit_diagnostics.log_temperature` | | `hit_bound` | false | `fit_diagnostics.hit_bound` | | `effective` | true | `fit_diagnostics.effective` | | held-out splits excluded | `[Test, Test2]` | `provenance.held_out_splits_excluded` | | checkpoint sha256 | `cfae5e43b97ca930f568dc5b8ae4f36b24e9ff717af226159802206ffd63a82a` | `provenance.checkpoint_sha256` | **The result — ECE got worse:** | Metric | Before | After | Improvement | |---|---|---|---| | **ECE** | **0.013755** | **0.014929** | **−0.001174** | | NLL | 0.689741 | 0.689631 | +0.00011 | `ece_improvement = −0.001174` is **negative**: expected calibration error **increased**. NLL improved by a negligible 0.00011 (fit diagnostics record 0.0001103574982127542). `n_bins: 15`, `n_classes: 19`, `n_samples: 16441`. **The scaling is retained because it is part of the frozen configuration, not because it helped.** This is recorded as a negative result, not smoothed over. **The reliability diagram is explicitly labelled pre-scaling.** `artifacts/calibration_v001.json` → `reliability_diagram.ece` is **0.013755** (the *before* value), and the block carries the note: *"Equal- width bins over predicted-class confidence. ECE is bin-count sensitive and is not an aggregate score."* The diagram on the Benchmark page is labelled **pre-scaling** so a reader cannot mistake it for the calibrated result. The bin populations range from 8 (bin 0.133–0.2) to 2,814 (bin 0.933–1.0), and two bins (0.0–0.0667 and 0.0667–0.1333) are empty. **The consumer contract** (`consumer_contract`): class `TemperatureCalibration` in module `evidence.confidence`; applied as `sigmoid(logit(z) / T)` for a scalar `z` and `softmax(logits / T)` for a distribution; read keys `temperature`, `fitted_on|split`, `artifact`, `n_samples`; resolved by `load_calibration(config, base_dir='configs')`. **Limitations.** ECE is bin-count sensitive and is not an aggregate score. The calibrated curve is not plotted — only the pre-scaling diagram is. The temperature is fitted only on the change-VQA head; no other specialist is calibrated. --- ## 6. Distribution and licensing ### 6.1 Backbones are not redistributed Every backbone is fetched from the Hugging Face Hub at run time, pinned by revision (§2). No backbone weight is included in this release, and no backbone is fine-tuned. Their licences are their own — see each model's HF page: | Backbone | Repository | Pinned revision | |---|---|---| | Router encoder | `sentence-transformers/all-MiniLM-L6-v2` | `1110a243fdf4` | | VLM | `HuggingFaceTB/SmolVLM-500M-Instruct` | `a7da5b986cb5` | | Grounding | `chendelong/RemoteCLIP` | `bf1d8a3ccf2d` | | Optical-SAR | `antofuller/CROMA` | `0dd28e3d633b` | The change encoder's ResNet-18 trunk is a torchvision model with ImageNet weights; it is likewise not redistributed. ### 6.2 The six trained artifacts are published by this project The six trained artifacts (§1) are published on the Hugging Face Hub under `thundercode/SatQuery` (`models/manifest.json` → `release_repo`), **labelled by kind**, each with its backbone dependency documented and each accompanied by a checksum: - [`../models/manifest.json`](../models/manifest.json) — schema `satquery_model_manifest_v1`, generated by `release/tools/generate_model_manifest.py`, `artifact_count: 6`, `generated_utc: 2026-09-25T18:15:38+00:00`, `config_hash: 78f1e3700da15aa1`. Its `note` states: *"No byte count or hash is typed by hand."* - [`../models/checksums.sha256`](../models/checksums.sha256) — six lines, keyed by the path in the repository, verifiable with `sha256sum -c checksums.sha256`. See [`../HF_RELEASE_VERIFICATION.md`](../HF_RELEASE_VERIFICATION.md) for the upload/verification record. ### 6.3 No licence file exists — OPEN **There is no `LICENSE` file in the source repository.** This is an **OPEN** item flagged in [`LIMITATIONS.md`](LIMITATIONS.md) §5 (item 29): a licence must be selected **before any public release of the code**. Model weights carry the terms of their backbone licences; the six trained artifacts are distributed under whatever terms the owner selects. ### 6.4 What this release does not contain - **No backbone weights** (fetched at run time, pinned by revision). - **No training checkpoints** (archived as provenance, not released — §1.2). - **No feature caches** (`fusion_features/`, the grounding feature cache, etc. — reproducible, not released as weights). - **No hidden/private evaluation data** (`evaluation.hidden_data_access: false`). --- ## 7. Status summary — what is NOT established ### 7.1 Status of each artifact | Artifact | Implemented | Measured | Acceptance | Ruling | |---|---|---|---|---| | `change` | yes | yes | **VERIFIED** | closed | | `change_vqa` | yes | yes | MEASURED | **OPEN** | | `optical_sar` | yes | yes | MEASURED | **OPEN** | | `grounding` | yes | yes | MEASURED (2 protocols) | Phase 8 complete; head not wired | | `router` | yes | validation only | MEASURED | **TEST NOT RUN** | | `vlm` | yes | yes | **ACCEPTANCE-REJECTED** | CLOSED | ### 7.2 NOT RUN / OPEN / BLOCKED for this topic | Item | State | |---|---| | **System-level end-to-end accuracy** | **NOT RUN — none exists**; no such number is claimed anywhere | | **Router test split** | **NOT RUN** | | **Grounding resolution at 448 for the trained head** | **NOT RUN** — only the zero-shot baseline was measured at 448; a re-open would be a new pre-registered experiment | | **`grounding_head` wired into `inference.py`** | **DEFERRED** (Phase 8 integration step) | | **`change` head wired into serving by default** | **DEFERRED** (registry override, not config) | | **`change_vqa` serving** | requires both a trained head *and* a trained feature extractor; neither wired by default | | **`sa_mode: BAM`** | **NOT IMPLEMENTED** (raises) | | **Router adapter parameter count** | **UNKNOWN** — 50,822 (manifest/docstring) vs 51,725 (Phase 4 report), unreconciled | | **`optical_sar` / `change` head parameter counts** | `UNKNOWN — not established from the available evidence` (manifest records `null`) | | **`optical.normalization` / `sar.representation` config keys** | read by **no code** — **OPEN** | | **CROMA `number_of_patches`** | **UNVERIFIED** upstream; checked against 225 at load time | | **Grounding validation IoU** | the Phase 8 table records 0.0943 at epoch 20; the prose of the same document rounds it to 0.0946 — the small discrepancy is unreconciled | | **BigEarthNet multi-label evaluation** | **NOT PRODUCED** — the local subset is single-label | | **Cross-dataset generalisation** | **NOT RUN** — each specialist is evaluated only on its own training-family test split | | **`LICENSE` file** | **OPEN** — none exists | | **B-07 tunnel gaps** | patch prepared, **NOT deployed**. **OPEN** | | **B-02 `codespace_name` trailing `\n`** | cosmetic; **OPEN** | ### 7.3 Explicit non-claims - **No claim of state-of-the-art performance** on any benchmark. - **No claim that the trained heads generalise** beyond their training-family test splits. - **No claim that calibration improves confidence** — the measured ECE worsened. - **No claim that the VLM adapter is accepted** for production use. - **No claim of a system-level accuracy**, because no system-level benchmark exists. - **No claim that any artifact's accuracy may be quoted without its companion metric** (optical-SAR accuracy without macro-F1; grounding under one protocol; change-VQA on one test set). --- ## 8. Evidence index | Topic | Evidence | |---|---| | Manifest + checksums | [`../models/manifest.json`](../models/manifest.json), [`../models/checksums.sha256`](../models/checksums.sha256), `release/tools/generate_model_manifest.py`, `release/tools/model_manifest_report.txt` | | Model card | [`../MODEL_CARD.md`](../MODEL_CARD.md) | | HF release verification | [`../HF_RELEASE_VERIFICATION.md`](../HF_RELEASE_VERIFICATION.md) | | Config registry | `configs/base.yaml`, `core/config.py` | | `change` | `specialists/change/stanet.py`, `specialists/change/specialist.py`, `artifacts/change/eval_test/eval_result.json` | | `change_vqa` | `specialists/change/vqa_specialist.py`, `artifacts/change_vqa/run/PROMOTION.json` | | `optical_sar` | `specialists/optical_sar/{croma,fusion_head,specialist}.py`, `artifacts/optical_sar/fusion_head_production_v001/pre_registered_115_metric.json`, `docs/PHASE12_115_METRIC_COMPUTED.md`, `docs/PHASE14_OPTICAL_SAR_DECISIONS.md` | | `grounding` | `specialists/grounding/{head,remoteclip}.py`, `artifacts/grounding/remoteclip_grounding_v001/eval_result_{canonical,matched6}.json`, `docs/PHASE7_RESOLUTION_DECISION.md`, `docs/PHASE8_GROUNDING_HEAD_DECISION.md` | | `router` | `router/adapter.py`, `artifacts/router/threshold_sweep_val.json`, `docs/PHASE4_ROUTER_REPORT.md` | | `vlm` | `specialists/vqa/model.py`, `artifacts/vlm/phase6_closure.json`, `artifacts/vlm/run1_test_recovery/adapter_verification.json`, `docs/PHASE6_CLOSURE.md` | | Calibration | `artifacts/calibration_v001.json` | | Cross-links | [`BENCHMARKS.md`](BENCHMARKS.md) (metrics and their rules), [`EVALUATION.md`](EVALUATION.md) (how each number was produced), [`TRAINING.md`](TRAINING.md) (how each head was trained), [`DATASETS.md`](DATASETS.md) (the corpora), [`LIMITATIONS.md`](LIMITATIONS.md) (the honest catalogue), [`REPRODUCIBILITY.md`](REPRODUCIBILITY.md) (how to reproduce) |