---
license: apache-2.0
language:
  - en
library_name: pytorch
pipeline_tag: image-segmentation
tags:
  - image-segmentation
  - semantic-segmentation
  - robotics
  - edge-ai
  - construction-ai
  - autonomous-painting
  - wall-painting-robot
  - paint-coverage-estimation
  - building-facade
  - drywall
  - skirting-detection
  - window-detection
  - lidar
  - depth-validation
  - segformer
  - deeplabv3
  - pytorch
metrics:
  - mean-iou
  - iou
model-index:
  - name: painting-vision-robotics-kit
    results:
      - task:
          type: image-segmentation
          name: Wall / window / other segmentation (hosted best.pt, real facades)
        dataset:
          name: painting_vision held-out test split, real facade subset (76 images)
          type: painting-vision-real-data
          split: test
        metrics:
          - type: mean_iou
            value: 0.7717
            name: mIoU (other/wall/window, real facades)
          - type: iou
            value: 0.838
            name: wall_painted IoU (real facades)
          - type: iou
            value: 0.642
            name: window IoU (real facades)
          - type: recall
            value: 0.861
            name: window recall (real facades)
---

# Painting Vision Robotics Kit — robot-ready wall-paint coverage segmentation

**Edge-deployable perception and planning for autonomous painting and surface-finishing robots: from pixels to an executable paint plan.**

The Painting Vision Robotics Kit reads a single camera frame of a wall and returns far more than
a mask. It segments the paintable surface, the fixtures and openings that must
not be painted, and the substrate — then hands a downstream planner a *wall
boundary, corners, a two-tool stroke plan, machine-readable waypoints, and a
live paint-progress grid*, independently cross-checked against depth/LiDAR.

It is built for the hard part of construction robotics: knowing **exactly where
the wall is, where the keep-outs are, and when the coat is complete**.

> **Release:** `best.pt` is a SegFormer-B2 (768 px) **facade checkpoint** — walls, windows and paint coverage
> on real exteriors. The interior paint-state and substrate model (**C2-cal**) is in validation and reported
> below. The geometry planner, depth validator, safety gate and dataset tooling ship together and are covered
> by the included test suite.

## The open tier of a larger model family

Constructelligence develops construction-AI models across a range of sizes. This kit is the **open, onboard
tier**: compact enough to run on a painting robot's CPU or a small accelerator, offline, with the full
perception-to-plan pipeline. Our larger models — bigger backbones, higher-resolution and multimodal RGB +
depth/LiDAR input — are a separate class and are not published here.

---

## Why this model exists

General segmentation models tell you a wall is *present*. Painting robots need
to know:

- which pixels are **paintable wall** versus **trim, skirting, openings and fixtures**;
- the **exact boundary and corners** of the wall, so a roller or spray head does not cross an edge;
- a **stroke path** that respects a physical clearance margin and lifts over obstacles;
- the **substrate** (e.g. drywall) so paint behaviour is not confused with surface type;
- **remaining coverage** across repeated observations, so the robot knows when to stop.

The kit is designed end-to-end around those questions.

## Capabilities

![How it works](figures/painting-pipeline.png)


### 1. Ten-class semantic segmentation + substrate head
The model family is **SegFormer-B2** (the hosted `best.pt` is the facade checkpoint; interior classes are
C2's); `models.py --arch` also builds MobileNetV3-Large and ResNet-50/101
DeepLabV3 variants for smaller or larger compute budgets. A **ten-class semantic head** is paired with a
**separate drywall-material head**, so paint state and substrate are predicted independently:

`other · wall_unpainted · wall_painted · wall_uncertain · skirting · switch/outlet · AC unit · door · window · wall_obstacle`

Painted coverage is reported as `wall_painted / (wall_painted + wall_unpainted)`
with lower/upper bounds that include uncertain wall area — a confidence-aware
estimate, not a single false-precise number.

### 2. Window detection you can trust around
Windows are the most safety-critical keep-out in exterior painting. The
`window_postprocess` stage merges mullion-split fragments, fits an **oriented
minimum-area rectangle** with rotating calipers (so a window stays a window at
any viewing angle), and filters specks and non-window shapes by fill ratio and
aspect ratio — producing complete, regularised keep-out geometry.

### 3. Robot paint planner
`robot_planner.py` converts the masks into an executable plan:

- ordered **boundary polygon and corners** (Moore-neighbour contour tracing + Douglas-Peucker — it follows real rooflines and voids, not a convex hull);
- optional **wall-plane rectification** with edge lengths in millimetres;
- a **two-tool plan**: roller passes for open areas plus **trim/brush passes** for edges and around fixtures the roller cannot reach;
- **machine-ready waypoints** with paint on/off flags and travel distance;
- **paint-progress tracking** (`--state`) that targets only what remains and tells the robot when the wall is done.

A documented real-facade run lifted planned coverage from **68% (roller only) to
97.5% (roller + trim)** on the same wall — the residual is the physically
unreachable remainder, reported explicitly.

### 4. Depth / LiDAR validation on top of vision
`depth_validation.py` independently checks the vision output in metric 3D:
RANSAC wall-plane fitting, protrusion/recess detection (missed pipes or windows),
recess-versus-window agreement, and a **metric millimetres-per-pixel scale**
derived from depth instead of guessed — so the planner works in real units.

### 5. Balanced-data tooling
`data_balance.py` and `audit_dataset.py` score coverage and balance across
indoor/exterior, substrate, lighting, weather, paint stage and classes, flag
wall/session leakage and thin classes, and recommend exactly how many more
samples each gap needs. `train.py` supports class-, wall- and hybrid-balanced
sampling so the balanced collection survives into the optimizer.

### 6. Paint colour: named, matched, per face
`paint_color.py` places the painted pixels in CIELAB, trims the darkest and
brightest 10% (shadow, glare, lap marks) and names the coat ("magnolia
#F1EBD6"), reports a k-means palette and a colour per measured wall face — so a
face painted in the wrong colour or a missed second coat is visible in the
report. A specified colour (`--reference-color`) is compared with **CIEDE2000**
and banded (≤1 exact, ≤2 a touch-up match, ≤5 the same at a glance), and
`safety_gate.py` stops the coat when the measured colour is not the specified
one. The colour printed on a paint can's label is read and checked against the
same reference, so the product on site is verified and not assumed.

### 7. The equipment around the wall
`site_objects.py` finds paint cans, trays, brushes, rollers, ladders and dust
sheets around the wall with a classical-vision detector that needs no training data: gradient outlines are cut above the frame's own noise, scene structure
(skirting, ceiling, the wall/floor junction) is removed so a tin standing on the
floor is not welded onto it, outlines are closed and filled back into bodies,
and each body is scored per class from its own cues — aspect, fill, outline
straightness, nap texture, the stripe a tin's rim or a brush's ferrule makes,
the repeats tray ridges or ladder rungs make, frame position, colour. It answers
what a site asks: is the gear the plan needs here, is anything standing where
the paint is going (folded into the keep-out mask), and is the tin the right
colour. Every detection carries the cues that fired and the scores of the
classes it was *not* given; `safety_gate.py` warns by default on clutter and on
missing tools, and stops on a colour mismatch.

## Quickstart

```sh
python3 -m pip install -r requirements.txt

# 1. Segment a frame
python3 predict.py frame.jpg --checkpoint hf://constructelligence/painting-vision-robotics-kit \
    --paintable-mask paintable_wall_mask.png \
    --keepout-mask fixture_keepout_mask.png \
    --json result.json --tta

# 2. Plan the paint path (optional wall-plane rectification + mm scale)
python3 robot_planner.py --paintable paintable_wall_mask.png \
    --keepout fixture_keepout_mask.png \
    --corners "80,60 1180,55 1190,700 70,710" \
    --stroke-width 230 --trim-width 60 --overlap 0.2 --margin 40 --mm-per-unit 3.0 \
    --state wall_014.npz --plan plan.json --overlay plan.png

# 3. Validate against depth / LiDAR if available
python3 depth_validation.py --depth depth.npy --intrinsics "fx,fy,cx,cy" \
    --mask painting_mask.png --report validation.json
```

## Repository layout

| File | Role |
| --- | --- |
| `models.py` | architecture factory: DeepLabV3 (MobileNet/ResNet) + SegFormer, semantic + drywall heads |
| `train.py` | training, balanced sampling, window loss boost, boundary/coverage metrics |
| `predict.py` | inference: masks, coverage, geometry, regularised windows, paint colour, equipment on site |
| `paint_color.py` | the coat's colour named, paletted, per face, and matched with CIEDE2000 |
| `site_objects.py` | model-free detection of paint cans, trays, brushes, rollers, ladders, dust sheets; the colour on a tin's label |
| `robot_planner.py` | boundary/corners, two-tool stroke plan, waypoints, progress state |
| `window_postprocess.py` | fragment merge + oriented rectangle fitting for windows |
| `depth_validation.py` | LiDAR/depth plane fit, protrusion/recess checks, metric scale |
| `data_balance.py` | coverage/balance analytics |
| `audit_dataset.py` | dataset audit: schema, leakage, class and domain balance |
| `fetch_real_data.py` | downloads and converts a real labeled dataset for testing |
| `smoke_test.py` | end-to-end audit → train → predict → plan |
| `benchmark.py` | head-to-head vs a generic baseline on the shared wall/window/other subset |
| `provenance.py` | content-addressed dataset identity, integrity + licence ledger |
| `safety_gate.py` | fail-closed approve/reject deployment gate |
| `flywheel.py` | ingest corrected captures with content + perceptual dedupe |
| `active_learning.py` | rank which frames to label next (novelty + domain gaps + uncertainty) |
| `run_registry.py` | append-only, hash-linked ledger of training/eval runs |
| `test_*.py` | geometry, window, depth and architecture regression tests |

## Evaluation

Frozen protocol (`benchmark/SPEC.md`, v1.0); every number is tied to one checkpoint.

| Checkpoint | In this repo | What it is |
| --- | --- | --- |
| `best.pt` (sha256 `3ef02176…`) | **yes** | SegFormer-B2, 768 px; semantic + material (drywall) heads; trained for exterior facades |
| `C2-cal` | release candidate | SegFormer-B2 lineage with wallpaper and condition heads, painted-wall hard negatives and reviewed real interiors; calibrated operating points |

![Measured results](figures/painting-results.png)

### This release: `best.pt` on real facades

Held-out real facade frames (76) from the 136-frame test split:

| Metric | Value |
| --- | --- |
| mIoU, other / wall / window | **0.772** |
| wall_painted IoU | **0.838** |
| window IoU · precision / recall | 0.642 · 0.717 / **0.861** |
| painted-coverage MAE | **0.114** |

**Scope.** `best.pt` is an exterior model. The other 60 frames of the split are synthetic interiors, where it
does not predict the interior classes (unpainted wall, skirting, doors, switches) and must not be used to sign
off paint: on the full 136-frame split its 10-class mIoU is 0.186 and it calls 81% of unfinished interior walls
complete. Interior work is what C2 is for.

### Release candidate: C2-cal (not in this repo)

Same test split: 10-class mIoU **0.848**, painted-coverage MAE **0.088**. On real photographs it was never
trained on — two independently reviewed sets, with operating points fitted on set 1 and judged on held-out
set 2:

| Question (set 1 / held-out set 2) | C2-cal |
| --- | --- |
| Finding the wall | 117 / 117 |
| Bare drywall signed off as finished | **0% / 0%** |
| Bare drywall found (recall) | 0.89 / 0.88 |
| Painted wall called bare board | 35% / 31% |
| Wallpaper, held-out precision / recall | 0.64 / 0.70 |
| Damage (peeling / rust) AUROC | 0.61 / 0.72 — surfaced as a proposal for a person to confirm |

`train.py` reports every metric family above on any dataset you supply.

## Intended use and limitations

**Intended:** with `best.pt`, assistive perception and path planning for exterior painting/finishing robots;
dataset curation and annotation QA for construction vision.

**Not intended:** unattended safety-critical control without calibrated geometry, clearance validation and a
human-approved deployment; interior paint sign-off with `best.pt`; judging *finish quality* (streaking, thin
coat, runs).

RGB alone cannot always reveal what is under a smooth surface; the model scores
drywall only where visible evidence supports it and uses `unknown` otherwise.
Always validate on held-out physical walls with the production camera.

## Citation

```bibtex
@misc{constructelligence_painting_vision,
  title  = {Painting Vision Robotics Kit: robot-ready wall-paint coverage segmentation},
  author = {Constructelligence},
  year   = {2026},
  note   = {SegFormer-B2 release}
}
```

## Keywords

wall painting robot · autonomous painting · paint coverage estimation ·
construction AI · building facade segmentation · drywall detection · skirting
detection · window detection · semantic segmentation · LiDAR validation ·
depth sensing · robot path planning · paint progress tracking · construction
robotics · BIM · surface finishing automation