painting-vision-robotics-kit / README.md

Add pipeline and results diagrams to model card

90d6cac verified 19 days ago

13.9 kB

license: apache-2.0
language:
  - en
library_name: pytorch
pipeline_tag: image-segmentation
tags:
  - image-segmentation
  - semantic-segmentation
  - robotics
  - edge-ai
  - construction-ai
  - autonomous-painting
  - wall-painting-robot
  - paint-coverage-estimation
  - building-facade
  - drywall
  - skirting-detection
  - window-detection
  - lidar
  - depth-validation
  - segformer
  - deeplabv3
  - pytorch
metrics:
  - mean-iou
  - iou
model-index:
  - name: painting-vision-robotics-kit
    results:
      - task:
          type: image-segmentation
          name: Wall / window / other segmentation (hosted best.pt, real facades)
        dataset:
          name: painting_vision held-out test split, real facade subset (76 images)
          type: painting-vision-real-data
          split: test
        metrics:
          - type: mean_iou
            value: 0.7717
            name: mIoU (other/wall/window, real facades)
          - type: iou
            value: 0.838
            name: wall_painted IoU (real facades)
          - type: iou
            value: 0.642
            name: window IoU (real facades)
          - type: recall
            value: 0.861
            name: window recall (real facades)

Painting Vision Robotics Kit — robot-ready wall-paint coverage segmentation

Edge-deployable perception and planning for autonomous painting and surface-finishing robots: from pixels to an executable paint plan.

The Painting Vision Robotics Kit reads a single camera frame of a wall and returns far more than a mask. It segments the paintable surface, the fixtures and openings that must not be painted, and the substrate — then hands a downstream planner a wall boundary, corners, a two-tool stroke plan, machine-readable waypoints, and a live paint-progress grid, independently cross-checked against depth/LiDAR.

It is built for the hard part of construction robotics: knowing exactly where the wall is, where the keep-outs are, and when the coat is complete.

Release: best.pt is a SegFormer-B2 (768 px) facade checkpoint — walls, windows and paint coverage on real exteriors. The interior paint-state and substrate model (C2-cal) is in validation and reported below. The geometry planner, depth validator, safety gate and dataset tooling ship together and are covered by the included test suite.

The open tier of a larger model family

Constructelligence develops construction-AI models across a range of sizes. This kit is the open, onboard tier: compact enough to run on a painting robot's CPU or a small accelerator, offline, with the full perception-to-plan pipeline. Our larger models — bigger backbones, higher-resolution and multimodal RGB + depth/LiDAR input — are a separate class and are not published here.

Why this model exists

General segmentation models tell you a wall is present. Painting robots need to know:

which pixels are paintable wall versus trim, skirting, openings and fixtures;
the exact boundary and corners of the wall, so a roller or spray head does not cross an edge;
a stroke path that respects a physical clearance margin and lifts over obstacles;
the substrate (e.g. drywall) so paint behaviour is not confused with surface type;
remaining coverage across repeated observations, so the robot knows when to stop.

The kit is designed end-to-end around those questions.

Capabilities

1. Ten-class semantic segmentation + substrate head

The model family is SegFormer-B2 (the hosted best.pt is the facade checkpoint; interior classes are C2's); models.py --arch also builds MobileNetV3-Large and ResNet-50/101 DeepLabV3 variants for smaller or larger compute budgets. A ten-class semantic head is paired with a separate drywall-material head, so paint state and substrate are predicted independently:

other · wall_unpainted · wall_painted · wall_uncertain · skirting · switch/outlet · AC unit · door · window · wall_obstacle

Painted coverage is reported as wall_painted / (wall_painted + wall_unpainted) with lower/upper bounds that include uncertain wall area — a confidence-aware estimate, not a single false-precise number.

2. Window detection you can trust around

Windows are the most safety-critical keep-out in exterior painting. The window_postprocess stage merges mullion-split fragments, fits an oriented minimum-area rectangle with rotating calipers (so a window stays a window at any viewing angle), and filters specks and non-window shapes by fill ratio and aspect ratio — producing complete, regularised keep-out geometry.

3. Robot paint planner

robot_planner.py converts the masks into an executable plan:

ordered boundary polygon and corners (Moore-neighbour contour tracing + Douglas-Peucker — it follows real rooflines and voids, not a convex hull);
optional wall-plane rectification with edge lengths in millimetres;
a two-tool plan: roller passes for open areas plus trim/brush passes for edges and around fixtures the roller cannot reach;
machine-ready waypoints with paint on/off flags and travel distance;
paint-progress tracking (--state) that targets only what remains and tells the robot when the wall is done.

A documented real-facade run lifted planned coverage from 68% (roller only) to 97.5% (roller + trim) on the same wall — the residual is the physically unreachable remainder, reported explicitly.

4. Depth / LiDAR validation on top of vision

depth_validation.py independently checks the vision output in metric 3D: RANSAC wall-plane fitting, protrusion/recess detection (missed pipes or windows), recess-versus-window agreement, and a metric millimetres-per-pixel scale derived from depth instead of guessed — so the planner works in real units.

5. Balanced-data tooling

data_balance.py and audit_dataset.py score coverage and balance across indoor/exterior, substrate, lighting, weather, paint stage and classes, flag wall/session leakage and thin classes, and recommend exactly how many more samples each gap needs. train.py supports class-, wall- and hybrid-balanced sampling so the balanced collection survives into the optimizer.

6. Paint colour: named, matched, per face

paint_color.py places the painted pixels in CIELAB, trims the darkest and brightest 10% (shadow, glare, lap marks) and names the coat ("magnolia #F1EBD6"), reports a k-means palette and a colour per measured wall face — so a face painted in the wrong colour or a missed second coat is visible in the report. A specified colour (--reference-color) is compared with CIEDE2000 and banded (≤1 exact, ≤2 a touch-up match, ≤5 the same at a glance), and safety_gate.py stops the coat when the measured colour is not the specified one. The colour printed on a paint can's label is read and checked against the same reference, so the product on site is verified and not assumed.

7. The equipment around the wall

site_objects.py finds paint cans, trays, brushes, rollers, ladders and dust sheets around the wall with a classical-vision detector that needs no training data: gradient outlines are cut above the frame's own noise, scene structure (skirting, ceiling, the wall/floor junction) is removed so a tin standing on the floor is not welded onto it, outlines are closed and filled back into bodies, and each body is scored per class from its own cues — aspect, fill, outline straightness, nap texture, the stripe a tin's rim or a brush's ferrule makes, the repeats tray ridges or ladder rungs make, frame position, colour. It answers what a site asks: is the gear the plan needs here, is anything standing where the paint is going (folded into the keep-out mask), and is the tin the right colour. Every detection carries the cues that fired and the scores of the classes it was not given; safety_gate.py warns by default on clutter and on missing tools, and stops on a colour mismatch.

Quickstart

python3 -m pip install -r requirements.txt

# 1. Segment a frame
python3 predict.py frame.jpg --checkpoint hf://constructelligence/painting-vision-robotics-kit \
    --paintable-mask paintable_wall_mask.png \
    --keepout-mask fixture_keepout_mask.png \
    --json result.json --tta

# 2. Plan the paint path (optional wall-plane rectification + mm scale)
python3 robot_planner.py --paintable paintable_wall_mask.png \
    --keepout fixture_keepout_mask.png \
    --corners "80,60 1180,55 1190,700 70,710" \
    --stroke-width 230 --trim-width 60 --overlap 0.2 --margin 40 --mm-per-unit 3.0 \
    --state wall_014.npz --plan plan.json --overlay plan.png

# 3. Validate against depth / LiDAR if available
python3 depth_validation.py --depth depth.npy --intrinsics "fx,fy,cx,cy" \
    --mask painting_mask.png --report validation.json

Repository layout

File	Role
`models.py`	architecture factory: DeepLabV3 (MobileNet/ResNet) + SegFormer, semantic + drywall heads
`train.py`	training, balanced sampling, window loss boost, boundary/coverage metrics
`predict.py`	inference: masks, coverage, geometry, regularised windows, paint colour, equipment on site
`paint_color.py`	the coat's colour named, paletted, per face, and matched with CIEDE2000
`site_objects.py`	model-free detection of paint cans, trays, brushes, rollers, ladders, dust sheets; the colour on a tin's label
`robot_planner.py`	boundary/corners, two-tool stroke plan, waypoints, progress state
`window_postprocess.py`	fragment merge + oriented rectangle fitting for windows
`depth_validation.py`	LiDAR/depth plane fit, protrusion/recess checks, metric scale
`data_balance.py`	coverage/balance analytics
`audit_dataset.py`	dataset audit: schema, leakage, class and domain balance
`fetch_real_data.py`	downloads and converts a real labeled dataset for testing
`smoke_test.py`	end-to-end audit → train → predict → plan
`benchmark.py`	head-to-head vs a generic baseline on the shared wall/window/other subset
`provenance.py`	content-addressed dataset identity, integrity + licence ledger
`safety_gate.py`	fail-closed approve/reject deployment gate
`flywheel.py`	ingest corrected captures with content + perceptual dedupe
`active_learning.py`	rank which frames to label next (novelty + domain gaps + uncertainty)
`run_registry.py`	append-only, hash-linked ledger of training/eval runs
`test_*.py`	geometry, window, depth and architecture regression tests

Evaluation

Frozen protocol (benchmark/SPEC.md, v1.0); every number is tied to one checkpoint.

Checkpoint	In this repo	What it is
`best.pt` (sha256 `3ef02176…`)	yes	SegFormer-B2, 768 px; semantic + material (drywall) heads; trained for exterior facades
`C2-cal`	release candidate	SegFormer-B2 lineage with wallpaper and condition heads, painted-wall hard negatives and reviewed real interiors; calibrated operating points

This release: `best.pt` on real facades

Held-out real facade frames (76) from the 136-frame test split:

Metric	Value
mIoU, other / wall / window	0.772
wall_painted IoU	0.838
window IoU · precision / recall	0.642 · 0.717 / 0.861
painted-coverage MAE	0.114

Scope. best.pt is an exterior model. The other 60 frames of the split are synthetic interiors, where it does not predict the interior classes (unpainted wall, skirting, doors, switches) and must not be used to sign off paint: on the full 136-frame split its 10-class mIoU is 0.186 and it calls 81% of unfinished interior walls complete. Interior work is what C2 is for.

Release candidate: C2-cal (not in this repo)

Same test split: 10-class mIoU 0.848, painted-coverage MAE 0.088. On real photographs it was never trained on — two independently reviewed sets, with operating points fitted on set 1 and judged on held-out set 2:

Question (set 1 / held-out set 2)	C2-cal
Finding the wall	117 / 117
Bare drywall signed off as finished	0% / 0%
Bare drywall found (recall)	0.89 / 0.88
Painted wall called bare board	35% / 31%
Wallpaper, held-out precision / recall	0.64 / 0.70
Damage (peeling / rust) AUROC	0.61 / 0.72 — surfaced as a proposal for a person to confirm

train.py reports every metric family above on any dataset you supply.

Intended use and limitations

Intended: with best.pt, assistive perception and path planning for exterior painting/finishing robots; dataset curation and annotation QA for construction vision.

Not intended: unattended safety-critical control without calibrated geometry, clearance validation and a human-approved deployment; interior paint sign-off with best.pt; judging finish quality (streaking, thin coat, runs).

RGB alone cannot always reveal what is under a smooth surface; the model scores drywall only where visible evidence supports it and uses unknown otherwise. Always validate on held-out physical walls with the production camera.

Citation

@misc{constructelligence_painting_vision,
  title  = {Painting Vision Robotics Kit: robot-ready wall-paint coverage segmentation},
  author = {Constructelligence},
  year   = {2026},
  note   = {SegFormer-B2 release}
}

Keywords

wall painting robot · autonomous painting · paint coverage estimation · construction AI · building facade segmentation · drywall detection · skirting detection · window detection · semantic segmentation · LiDAR validation · depth sensing · robot path planning · paint progress tracking · construction robotics · BIM · surface finishing automation