Image Segmentation
PyTorch
English
painting-vision-robotics-kit
semantic-segmentation
robotics
edge-ai
construction-ai
autonomous-painting
wall-painting-robot
paint-coverage-estimation
building-facade
drywall
skirting-detection
window-detection
lidar
depth-validation
segformer
deeplabv3
Eval Results (legacy)
| # Painting Vision Benchmark — specification v1.0 | |
| A small, versioned benchmark so results are comparable across teams and across | |
| time. It defines **what** is measured, **how** to reduce a model to the shared | |
| label space, and **what** a submission must contain. Owning the protocol is part | |
| of the point: a benchmark others report against is more durable than a model. | |
| ## Task | |
| Given a single RGB frame of a wall, predict per-pixel labels. Models differ in | |
| what they can express, so evaluation is two-tiered: | |
| ### Tier 1 — shared subset (any generic model can enter) | |
| The ten-class taxonomy is projected onto `{other, wall, window}`: | |
| | reduced id | name | from the ten-class taxonomy | | |
| | --- | --- | --- | | |
| | 0 | `other` | class 0, and every non-wall, non-window class (skirting, fixtures, obstacles) | | |
| | 1 | `wall` | `wall_unpainted` (1), `wall_painted` (2), `wall_uncertain` (3) | | |
| | 2 | `window` | `window` (8) | | |
| A generic baseline (e.g. an ADE20K model) maps its own labels in: `wall` → 1, | |
| `windowpane`/`window` → 2, everything else → 0. | |
| ### Tier 2 — paint state (Painting Vision only, for now) | |
| Reported but not required of baselines: | |
| * `painted_coverage_mae` — mean absolute error of painted fraction of decided wall; | |
| * `mIoU_full_10class` — mean IoU over the ten-class taxonomy. | |
| ## Metrics | |
| * **IoU** per reduced class and **mIoU** over classes present in the ground truth. | |
| * **window precision** and **window recall** (binary, reduced class 2) — the | |
| safety-critical class, reported separately because a false window is expensive. | |
| * Uncertain/void pixels (`255`) are excluded from every metric. | |
| ## Protocol | |
| 1. **Split.** Evaluate on the held-out `test` split. Walls (buildings) are split, | |
| never frames; a `wall_id` must not cross splits (the audit enforces this). | |
| 2. **Dataset identity.** Every submission cites the `dataset_id` from | |
| `provenance.py` so the exact bytes evaluated are pinned. | |
| 3. **Model identity.** Every submission cites the checkpoint SHA-256. | |
| 4. **Inference size.** Reported per submission (default 640). | |
| 5. **Resolution independence.** Predictions are compared at ground-truth | |
| resolution; resize predictions with nearest-neighbour for masks, bilinear for | |
| logits before argmax. | |
| ## Floor | |
| An all-`other` predictor is always reported as the floor. Any model whose mIoU | |
| does not beat the floor has learned nothing. | |
| ## Submission record | |
| ```sh | |
| python3 benchmark.py --data data --checkpoint artifacts/best.pt \ | |
| --baseline segformer-ade --split test \ | |
| --dataset-manifest data_manifest.json --emit-submission submission.json | |
| ``` | |
| ```json | |
| { | |
| "benchmark_version": "1.0", | |
| "taxonomy_version": "1.0", | |
| "dataset_id": "pv1:...", | |
| "split": "test", | |
| "checkpoint_sha256": "...", | |
| "shared_metrics": {"mIoU": 0.0, "iou": {"other": 0.0, "wall": 0.0, "window": 0.0}, | |
| "window_precision": 0.0, "window_recall": 0.0}, | |
| "paint_state_metrics": {"painted_coverage_mae": 0.0, "mIoU_full_10class": 0.0} | |
| } | |
| ``` | |
| ## Versioning | |
| `benchmark_version` changes when the protocol changes; `taxonomy_version` changes | |
| when the label set changes. A leaderboard entry is only comparable within a | |
| matching `(benchmark_version, taxonomy_version, dataset_id)` triple. | |