Image Segmentation
PyTorch
English
painting-vision-robotics-kit
semantic-segmentation
robotics
edge-ai
construction-ai
autonomous-painting
wall-painting-robot
paint-coverage-estimation
building-facade
drywall
skirting-detection
window-detection
lidar
depth-validation
segformer
deeplabv3
Eval Results (legacy)
Upload benchmark/SPEC.md with huggingface_hub
Browse files- benchmark/SPEC.md +79 -0
benchmark/SPEC.md
ADDED
|
@@ -0,0 +1,79 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Painting Vision Benchmark — specification v1.0
|
| 2 |
+
|
| 3 |
+
A small, versioned benchmark so results are comparable across teams and across
|
| 4 |
+
time. It defines **what** is measured, **how** to reduce a model to the shared
|
| 5 |
+
label space, and **what** a submission must contain. Owning the protocol is part
|
| 6 |
+
of the point: a benchmark others report against is more durable than a model.
|
| 7 |
+
|
| 8 |
+
## Task
|
| 9 |
+
|
| 10 |
+
Given a single RGB frame of a wall, predict per-pixel labels. Models differ in
|
| 11 |
+
what they can express, so evaluation is two-tiered:
|
| 12 |
+
|
| 13 |
+
### Tier 1 — shared subset (any generic model can enter)
|
| 14 |
+
The ten-class taxonomy is projected onto `{other, wall, window}`:
|
| 15 |
+
|
| 16 |
+
| reduced id | name | from the ten-class taxonomy |
|
| 17 |
+
| --- | --- | --- |
|
| 18 |
+
| 0 | `other` | class 0, and every non-wall, non-window class (skirting, fixtures, obstacles) |
|
| 19 |
+
| 1 | `wall` | `wall_unpainted` (1), `wall_painted` (2), `wall_uncertain` (3) |
|
| 20 |
+
| 2 | `window` | `window` (8) |
|
| 21 |
+
|
| 22 |
+
A generic baseline (e.g. an ADE20K model) maps its own labels in: `wall` → 1,
|
| 23 |
+
`windowpane`/`window` → 2, everything else → 0.
|
| 24 |
+
|
| 25 |
+
### Tier 2 — paint state (Painting Vision only, for now)
|
| 26 |
+
Reported but not required of baselines:
|
| 27 |
+
* `painted_coverage_mae` — mean absolute error of painted fraction of decided wall;
|
| 28 |
+
* `mIoU_full_10class` — mean IoU over the ten-class taxonomy.
|
| 29 |
+
|
| 30 |
+
## Metrics
|
| 31 |
+
|
| 32 |
+
* **IoU** per reduced class and **mIoU** over classes present in the ground truth.
|
| 33 |
+
* **window precision** and **window recall** (binary, reduced class 2) — the
|
| 34 |
+
safety-critical class, reported separately because a false window is expensive.
|
| 35 |
+
* Uncertain/void pixels (`255`) are excluded from every metric.
|
| 36 |
+
|
| 37 |
+
## Protocol
|
| 38 |
+
|
| 39 |
+
1. **Split.** Evaluate on the held-out `test` split. Walls (buildings) are split,
|
| 40 |
+
never frames; a `wall_id` must not cross splits (the audit enforces this).
|
| 41 |
+
2. **Dataset identity.** Every submission cites the `dataset_id` from
|
| 42 |
+
`provenance.py` so the exact bytes evaluated are pinned.
|
| 43 |
+
3. **Model identity.** Every submission cites the checkpoint SHA-256.
|
| 44 |
+
4. **Inference size.** Reported per submission (default 640).
|
| 45 |
+
5. **Resolution independence.** Predictions are compared at ground-truth
|
| 46 |
+
resolution; resize predictions with nearest-neighbour for masks, bilinear for
|
| 47 |
+
logits before argmax.
|
| 48 |
+
|
| 49 |
+
## Floor
|
| 50 |
+
|
| 51 |
+
An all-`other` predictor is always reported as the floor. Any model whose mIoU
|
| 52 |
+
does not beat the floor has learned nothing.
|
| 53 |
+
|
| 54 |
+
## Submission record
|
| 55 |
+
|
| 56 |
+
```sh
|
| 57 |
+
python3 benchmark.py --data data --checkpoint artifacts/best.pt \
|
| 58 |
+
--baseline segformer-ade --split test \
|
| 59 |
+
--dataset-manifest data_manifest.json --emit-submission submission.json
|
| 60 |
+
```
|
| 61 |
+
|
| 62 |
+
```json
|
| 63 |
+
{
|
| 64 |
+
"benchmark_version": "1.0",
|
| 65 |
+
"taxonomy_version": "1.0",
|
| 66 |
+
"dataset_id": "pv1:...",
|
| 67 |
+
"split": "test",
|
| 68 |
+
"checkpoint_sha256": "...",
|
| 69 |
+
"shared_metrics": {"mIoU": 0.0, "iou": {"other": 0.0, "wall": 0.0, "window": 0.0},
|
| 70 |
+
"window_precision": 0.0, "window_recall": 0.0},
|
| 71 |
+
"paint_state_metrics": {"painted_coverage_mae": 0.0, "mIoU_full_10class": 0.0}
|
| 72 |
+
}
|
| 73 |
+
```
|
| 74 |
+
|
| 75 |
+
## Versioning
|
| 76 |
+
|
| 77 |
+
`benchmark_version` changes when the protocol changes; `taxonomy_version` changes
|
| 78 |
+
when the label set changes. A leaderboard entry is only comparable within a
|
| 79 |
+
matching `(benchmark_version, taxonomy_version, dataset_id)` triple.
|