Painting Vision Benchmark — specification v1.0

A small, versioned benchmark so results are comparable across teams and across time. It defines what is measured, how to reduce a model to the shared label space, and what a submission must contain. Owning the protocol is part of the point: a benchmark others report against is more durable than a model.

Task

Given a single RGB frame of a wall, predict per-pixel labels. Models differ in what they can express, so evaluation is two-tiered:

Tier 1 — shared subset (any generic model can enter)

The ten-class taxonomy is projected onto {other, wall, window}:

reduced id	name	from the ten-class taxonomy
0	`other`	class 0, and every non-wall, non-window class (skirting, fixtures, obstacles)
1	`wall`	`wall_unpainted` (1), `wall_painted` (2), `wall_uncertain` (3)
2	`window`	`window` (8)

A generic baseline (e.g. an ADE20K model) maps its own labels in: wall → 1, windowpane/window → 2, everything else → 0.

Tier 2 — paint state (Painting Vision only, for now)

Reported but not required of baselines:

painted_coverage_mae — mean absolute error of painted fraction of decided wall;
mIoU_full_10class — mean IoU over the ten-class taxonomy.

Metrics

IoU per reduced class and mIoU over classes present in the ground truth.
window precision and window recall (binary, reduced class 2) — the safety-critical class, reported separately because a false window is expensive.
Uncertain/void pixels (255) are excluded from every metric.

Protocol

Split. Evaluate on the held-out test split. Walls (buildings) are split, never frames; a wall_id must not cross splits (the audit enforces this).
Dataset identity. Every submission cites the dataset_id from provenance.py so the exact bytes evaluated are pinned.
Model identity. Every submission cites the checkpoint SHA-256.
Inference size. Reported per submission (default 640).
Resolution independence. Predictions are compared at ground-truth resolution; resize predictions with nearest-neighbour for masks, bilinear for logits before argmax.

Floor

An all-other predictor is always reported as the floor. Any model whose mIoU does not beat the floor has learned nothing.

Submission record

python3 benchmark.py --data data --checkpoint artifacts/best.pt \
    --baseline segformer-ade --split test \
    --dataset-manifest data_manifest.json --emit-submission submission.json

{
  "benchmark_version": "1.0",
  "taxonomy_version": "1.0",
  "dataset_id": "pv1:...",
  "split": "test",
  "checkpoint_sha256": "...",
  "shared_metrics": {"mIoU": 0.0, "iou": {"other": 0.0, "wall": 0.0, "window": 0.0},
                     "window_precision": 0.0, "window_recall": 0.0},
  "paint_state_metrics": {"painted_coverage_mae": 0.0, "mIoU_full_10class": 0.0}
}

Versioning

benchmark_version changes when the protocol changes; taxonomy_version changes when the label set changes. A leaderboard entry is only comparable within a matching (benchmark_version, taxonomy_version, dataset_id) triple.