constructelligence commited on
Commit
dee4b0e
·
verified ·
1 Parent(s): a85572a

Upload benchmark/SPEC.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. benchmark/SPEC.md +79 -0
benchmark/SPEC.md ADDED
@@ -0,0 +1,79 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Painting Vision Benchmark — specification v1.0
2
+
3
+ A small, versioned benchmark so results are comparable across teams and across
4
+ time. It defines **what** is measured, **how** to reduce a model to the shared
5
+ label space, and **what** a submission must contain. Owning the protocol is part
6
+ of the point: a benchmark others report against is more durable than a model.
7
+
8
+ ## Task
9
+
10
+ Given a single RGB frame of a wall, predict per-pixel labels. Models differ in
11
+ what they can express, so evaluation is two-tiered:
12
+
13
+ ### Tier 1 — shared subset (any generic model can enter)
14
+ The ten-class taxonomy is projected onto `{other, wall, window}`:
15
+
16
+ | reduced id | name | from the ten-class taxonomy |
17
+ | --- | --- | --- |
18
+ | 0 | `other` | class 0, and every non-wall, non-window class (skirting, fixtures, obstacles) |
19
+ | 1 | `wall` | `wall_unpainted` (1), `wall_painted` (2), `wall_uncertain` (3) |
20
+ | 2 | `window` | `window` (8) |
21
+
22
+ A generic baseline (e.g. an ADE20K model) maps its own labels in: `wall` → 1,
23
+ `windowpane`/`window` → 2, everything else → 0.
24
+
25
+ ### Tier 2 — paint state (Painting Vision only, for now)
26
+ Reported but not required of baselines:
27
+ * `painted_coverage_mae` — mean absolute error of painted fraction of decided wall;
28
+ * `mIoU_full_10class` — mean IoU over the ten-class taxonomy.
29
+
30
+ ## Metrics
31
+
32
+ * **IoU** per reduced class and **mIoU** over classes present in the ground truth.
33
+ * **window precision** and **window recall** (binary, reduced class 2) — the
34
+ safety-critical class, reported separately because a false window is expensive.
35
+ * Uncertain/void pixels (`255`) are excluded from every metric.
36
+
37
+ ## Protocol
38
+
39
+ 1. **Split.** Evaluate on the held-out `test` split. Walls (buildings) are split,
40
+ never frames; a `wall_id` must not cross splits (the audit enforces this).
41
+ 2. **Dataset identity.** Every submission cites the `dataset_id` from
42
+ `provenance.py` so the exact bytes evaluated are pinned.
43
+ 3. **Model identity.** Every submission cites the checkpoint SHA-256.
44
+ 4. **Inference size.** Reported per submission (default 640).
45
+ 5. **Resolution independence.** Predictions are compared at ground-truth
46
+ resolution; resize predictions with nearest-neighbour for masks, bilinear for
47
+ logits before argmax.
48
+
49
+ ## Floor
50
+
51
+ An all-`other` predictor is always reported as the floor. Any model whose mIoU
52
+ does not beat the floor has learned nothing.
53
+
54
+ ## Submission record
55
+
56
+ ```sh
57
+ python3 benchmark.py --data data --checkpoint artifacts/best.pt \
58
+ --baseline segformer-ade --split test \
59
+ --dataset-manifest data_manifest.json --emit-submission submission.json
60
+ ```
61
+
62
+ ```json
63
+ {
64
+ "benchmark_version": "1.0",
65
+ "taxonomy_version": "1.0",
66
+ "dataset_id": "pv1:...",
67
+ "split": "test",
68
+ "checkpoint_sha256": "...",
69
+ "shared_metrics": {"mIoU": 0.0, "iou": {"other": 0.0, "wall": 0.0, "window": 0.0},
70
+ "window_precision": 0.0, "window_recall": 0.0},
71
+ "paint_state_metrics": {"painted_coverage_mae": 0.0, "mIoU_full_10class": 0.0}
72
+ }
73
+ ```
74
+
75
+ ## Versioning
76
+
77
+ `benchmark_version` changes when the protocol changes; `taxonomy_version` changes
78
+ when the label set changes. A leaderboard entry is only comparable within a
79
+ matching `(benchmark_version, taxonomy_version, dataset_id)` triple.