Model card: correct evaluation attribution (hosted best.pt = facade checkpoint; C2-cal release candidate)

README.md

@@ -30,30 +30,24 @@ model-index:
     results:
       - task:
           type: image-segmentation
-          name: Wall / window / other semantic segmentation
+          name: Wall / window / other segmentation (hosted best.pt, real facades)
         dataset:
-          name: painting_vision held-out test split (136 images, 640 px)
+          name: painting_vision held-out test split, real facade subset (76 images)
           type: painting-vision-real-data
           split: test
         metrics:
           - type: mean_iou
-            value: 0.8023
-            name: mIoU (other/wall/window)
-          - type: mean_iou
-            value: 0.6981
-            name: mIoU (full 10-class)
+            value: 0.7717
+            name: mIoU (other/wall/window, real facades)
           - type: iou
-            value: 0.8716
-            name: wall IoU
+            value: 0.838
+            name: wall_painted IoU (real facades)
           - type: iou
-            value: 0.7166
-            name: window IoU
-          - type: precision
-            value: 0.8343
-            name: window precision
+            value: 0.642
+            name: window IoU (real facades)
           - type: recall
-            value: 0.8355
-            name: window recall
+            value: 0.861
+            name: window recall (real facades)
 ---
 
 # Painting Vision Robotics Kit — robot-ready wall-paint coverage segmentation
@@ -69,9 +63,10 @@ live paint-progress grid*, independently cross-checked against depth/LiDAR.
 It is built for the hard part of construction robotics: knowing **exactly where
 the wall is, where the keep-outs are, and when the coat is complete**.
 
-> **Release:** SegFormer-B2 at 768 px (`best.pt`, `config.json`), trained with the open `train.py` on
-> session-separated real capture. The segmentation model, geometry planner, depth validator, safety gate and
-> dataset tooling ship together and are covered by the included test suite.
+> **Release:** `best.pt` is a SegFormer-B2 (768 px) **facade checkpoint** — walls, windows and paint coverage
+> on real exteriors. The interior paint-state and substrate model (**C2-cal**) is in validation and reported
+> below. The geometry planner, depth validator, safety gate and dataset tooling ship together and are covered
+> by the included test suite.
 
 ## The open tier of a larger model family
 
@@ -98,7 +93,8 @@ The kit is designed end-to-end around those questions.
 ## Capabilities
 
 ### 1. Ten-class semantic segmentation + substrate head
-The released model is **SegFormer-B2**; `models.py --arch` also builds MobileNetV3-Large and ResNet-50/101
+The model family is **SegFormer-B2** (the hosted `best.pt` is the facade checkpoint; interior classes are
+C2's); `models.py --arch` also builds MobileNetV3-Large and ResNet-50/101
 DeepLabV3 variants for smaller or larger compute budgets. A **ten-class semantic head** is paired with a
 **separate drywall-material head**, so paint state and substrate are predicted independently:
 
@@ -215,35 +211,54 @@ python3 depth_validation.py --depth depth.npy --intrinsics "fx,fy,cx,cy" \
 
 ## Evaluation
 
-Measured under a **frozen protocol** (`benchmark/SPEC.md`, v1.0) on a held-out `test` split of 136 real
-frames, pinned to a `dataset_id` and the checkpoint hash in `config.json`:
+Frozen protocol (`benchmark/SPEC.md`, v1.0); every number is tied to one checkpoint.
+
+| Checkpoint | In this repo | What it is |
+| --- | --- | --- |
+| `best.pt` (sha256 `3ef02176…`) | **yes** | SegFormer-B2, 768 px; semantic + material (drywall) heads; trained for exterior facades |
+| `C2-cal` | release candidate | SegFormer-B2 lineage with wallpaper and condition heads, painted-wall hard negatives and reviewed real interiors; calibrated operating points |
+
+### This release: `best.pt` on real facades
+
+Held-out real facade frames (76) from the 136-frame test split:
+
+| Metric | Value |
+| --- | --- |
+| mIoU, other / wall / window | **0.772** |
+| wall_painted IoU | **0.838** |
+| window IoU · precision / recall | 0.642 · 0.717 / **0.861** |
+| painted-coverage MAE | **0.114** |
+
+**Scope.** `best.pt` is an exterior model. The other 60 frames of the split are synthetic interiors, where it
+does not predict the interior classes (unpainted wall, skirting, doors, switches) and must not be used to sign
+off paint: on the full 136-frame split its 10-class mIoU is 0.186 and it calls 81% of unfinished interior walls
+complete. Interior work is what C2 is for.
+
+### Release candidate: C2-cal (not in this repo)
 
-| Metric (test split) | Value |
+Same test split: 10-class mIoU **0.848**, painted-coverage MAE **0.088**. On real photographs it was never
+trained on — two independently reviewed sets, with operating points fitted on set 1 and judged on held-out
+set 2:
+
+| Question (set 1 / held-out set 2) | C2-cal |
 | --- | --- |
-| mIoU, other / wall / window | **0.802** |
-| wall / window / other IoU | 0.872 / 0.717 / 0.819 |
-| window precision / recall | 0.834 / 0.836 |
-| mIoU, full 10-class | **0.698** |
-| painted-coverage MAE | 0.162 |
-
-The all-`other` floor is mIoU 0.110. On the validation split, painted-coverage error is 7.5 points with a
-**false-complete rate of 0.0** — the model did not mark an unfinished wall as complete.
-
-**Generalisation to unseen photographs.** On 117 real photographs outside the training distribution, the wall
-is found in **117 / 117**. The substrate signals are less mature and are released as **experimental**: wallpaper
-precision 0.50 (22 positives); painted vs bare drywall separated correctly in 50% of cases raw, 65% after
-calibration; completion sign-off on bare drywall in 1 of 19 walls. These heads should inform, not decide, a
-robot's actions. Surface-condition assessment (peeling,
-rust) is not part of this release. `train.py` reports every metric family above on any dataset you supply.
+| Finding the wall | 117 / 117 |
+| Bare drywall signed off as finished | **0% / 0%** |
+| Bare drywall found (recall) | 0.89 / 0.88 |
+| Painted wall called bare board | 35% / 31% |
+| Wallpaper, held-out precision / recall | 0.64 / 0.70 |
+| Damage (peeling / rust) AUROC | 0.61 / 0.72 — surfaced as a proposal for a person to confirm |
+
+`train.py` reports every metric family above on any dataset you supply.
 
 ## Intended use and limitations
 
-**Intended:** assistive perception and path planning for painting/finishing
-robots; dataset curation and annotation QA for construction vision.
+**Intended:** with `best.pt`, assistive perception and path planning for exterior painting/finishing robots;
+dataset curation and annotation QA for construction vision.
 
 **Not intended:** unattended safety-critical control without calibrated geometry, clearance validation and a
-human-approved deployment; substrate or completion decisions from the experimental heads alone; judging
-*finish quality* (streaking, thin coat, runs) or surface condition, which are outside this release.
+human-approved deployment; interior paint sign-off with `best.pt`; judging *finish quality* (streaking, thin
+coat, runs).
 
 RGB alone cannot always reveal what is under a smooth surface; the model scores
 drywall only where visible evidence supports it and uses `unknown` otherwise.