# Painting Vision Benchmark — specification v1.0

A small, versioned benchmark so results are comparable across teams and across
time. It defines **what** is measured, **how** to reduce a model to the shared
label space, and **what** a submission must contain. Owning the protocol is part
of the point: a benchmark others report against is more durable than a model.

## Task

Given a single RGB frame of a wall, predict per-pixel labels. Models differ in
what they can express, so evaluation is two-tiered:

### Tier 1 — shared subset (any generic model can enter)
The ten-class taxonomy is projected onto `{other, wall, window}`:

| reduced id | name | from the ten-class taxonomy |
| --- | --- | --- |
| 0 | `other` | class 0, and every non-wall, non-window class (skirting, fixtures, obstacles) |
| 1 | `wall` | `wall_unpainted` (1), `wall_painted` (2), `wall_uncertain` (3) |
| 2 | `window` | `window` (8) |

A generic baseline (e.g. an ADE20K model) maps its own labels in: `wall` → 1,
`windowpane`/`window` → 2, everything else → 0.

### Tier 2 — paint state (Painting Vision only, for now)
Reported but not required of baselines:
* `painted_coverage_mae` — mean absolute error of painted fraction of decided wall;
* `mIoU_full_10class` — mean IoU over the ten-class taxonomy.

## Metrics

* **IoU** per reduced class and **mIoU** over classes present in the ground truth.
* **window precision** and **window recall** (binary, reduced class 2) — the
  safety-critical class, reported separately because a false window is expensive.
* Uncertain/void pixels (`255`) are excluded from every metric.

## Protocol

1. **Split.** Evaluate on the held-out `test` split. Walls (buildings) are split,
   never frames; a `wall_id` must not cross splits (the audit enforces this).
2. **Dataset identity.** Every submission cites the `dataset_id` from
   `provenance.py` so the exact bytes evaluated are pinned.
3. **Model identity.** Every submission cites the checkpoint SHA-256.
4. **Inference size.** Reported per submission (default 640).
5. **Resolution independence.** Predictions are compared at ground-truth
   resolution; resize predictions with nearest-neighbour for masks, bilinear for
   logits before argmax.

## Floor

An all-`other` predictor is always reported as the floor. Any model whose mIoU
does not beat the floor has learned nothing.

## Submission record

```sh
python3 benchmark.py --data data --checkpoint artifacts/best.pt \
    --baseline segformer-ade --split test \
    --dataset-manifest data_manifest.json --emit-submission submission.json
```

```json
{
  "benchmark_version": "1.0",
  "taxonomy_version": "1.0",
  "dataset_id": "pv1:...",
  "split": "test",
  "checkpoint_sha256": "...",
  "shared_metrics": {"mIoU": 0.0, "iou": {"other": 0.0, "wall": 0.0, "window": 0.0},
                     "window_precision": 0.0, "window_recall": 0.0},
  "paint_state_metrics": {"painted_coverage_mae": 0.0, "mIoU_full_10class": 0.0}
}
```

## Versioning

`benchmark_version` changes when the protocol changes; `taxonomy_version` changes
when the label set changes. A leaderboard entry is only comparable within a
matching `(benchmark_version, taxonomy_version, dataset_id)` triple.