dlrv-mvcnn-v12

Multi-View CNN (MVCNN) 6DoF object pose estimator for the fifteen LineMOD objects. This is the final v12 checkpoint: it renders twelve virtual views of the masked RGB-D observation and is the better of the two project pipelines on real images, used for the head-to-head comparison against the PointNet++ track in the report. Produced for a Deep Learning for Robot Vision project at Hochschule Bonn-Rhein-Sieg.

What it does

Given the masked RGB-D observation of a single object instance, it regresses the object's full 6DoF pose:

• Input — the masked depth is back-projected into a point cloud with RGB colours, and the pipeline renders twelve orthographic virtual views of that observed geometry (azimuths 0–330°, nearest-first z-buffer, object scaled to roughly 80% of the 224×224 frame). The views are ordered and all come from the same single instance, so the bag has exactly one pose target.
• Rotation in the continuous 6D representation of Zhou et al., converted to a rotation matrix with Gram–Schmidt.
• Translation as a residual from the observed depth centroid (t = centroid + t_residual, residual normalised by 100 mm) — the same parameterisation as the PointNet++ track, so the translation is predicted, not supplied.
• An auxiliary classifier on the pooled descriptor acts as a diagnostic of object-identity information; it is not part of the pose task.

Architecture

Backbone	ResNet-18 (ImageNet-pretrained) shared over the 12 views, per-view 512-d descriptor, max-pooled over views (MVCNN view pooling)
Head	shared PoseHead (identical to PointNet++'s), 512-d feature, rotation branch 256→128→6, translation branch 256→128→3
Auxiliary classifier	dropout 0.3 + Linear(512, 15)
Parameters	11,383,512
Input	[B, 12, 3, 224, 224], ImageNet-normalised; renderer capped at 10,000 points

Results

Trained on the synthetic PBR split only (635,614 usable instances after visibility filtering), with sparse-observation augmentation as the only domain-adaptation measure. Model selection used a stratified 990-instance subset of the real test split; best checkpoint at epoch 26 (58.28% on that subset).

Split	@0.05d	@0.1d	@0.15d	@0.2d	Mean ADD	Median ADD
Synthetic (model-selection subset)	—	58.28%	—	—	—	—
Real BOP test, 3,000 images	15.33%	56.33%	73.20%	80.27%	26.38 mm	15.55 mm

Mean translation error 11.88 mm, mean rotation error 51.61°.

The symmetric objects (bowl, eggbox, glue, scored with ADD-S) rank highest (93–100%); the weakest are cup 12.0%, duck 28.0%, driller 30.5% — the objects whose real observations are sparsest. Unlike the point cloud track, accuracy is largely insensitive to object diameter (Pearson r = +0.21 against the point cloud pipeline's +0.92 on real data): the renderer normalises apparent scale, so a fixed pixel precision corresponds to a fixed fraction of object extent — exactly what the ADD threshold measures. For the point cloud counterpart, see jan024/dlrv-pointnet-exp5 (36.23%) and jan024/dlrv-ppointnet-exp6.

Training

30 epochs, batch 16, AdamW with differential learning rates (pose head 1e-3, backbone 2e-4), weight decay 1e-4, gradient clipping at 1.0, StepLR halving every 20 epochs, AMP mixed precision. Each epoch trains on a rotating 100,000-instance window of the usable PBR instances (6,250 batches), with deterministic per-epoch ordering so mid-epoch resume skips completed batches.

The objective is the differentiable ADD/ADD-S distance on 500 sampled CAD vertices (ADD-S for bowl, eggbox, glue) plus a small auxiliary cross-entropy term (weight 0.1, label smoothing 0.05). Augmentation: brightness/contrast/saturation jitter, point dropout, structured occlusion, sparse-observation simulation (p = 0.35, keep 30–70%), XYZ sensor noise, pose-consistent rotation (±20°/±30°/±20°) and adaptive splat densification of views below 6% occupancy.

Usage

The checkpoint is a dict with keys model, epoch, val_add10 and loss. Load it with the model definitions from the project repository:

import torch
from mvcnn.model import MVCNNWithPose

ckpt = torch.load('mvcnn_v12_best.pth', map_location='cuda')

model = MVCNNWithPose(num_classes=15, pretrained=False, trans_scale_mm=100.0).cuda()
model.load_state_dict(ckpt['model'])
model.eval()

The virtual views are rendered by mvcnn.data.MVCNNBOPDataset / render_virtual_views from the same repository. To reproduce the headline figure end to end on the real test split:

python examples/run_real_eval.py \
    --data-root /path/to/lm \
    --mvcnn-checkpoint mvcnn_v12_best.pth \
    --out results.json

LineMOD in BOP format is required and is not distributed here; get it from https://bop.felk.cvut.cz/datasets/. mvcnn_v12_results.json in this repository holds the full metric dict of the reported run.

Limitations

• Trained on synthetic data only. Real test observations carry only 20–53% of the training point density per object; that observation-density gap, not the pose targets, is the dominant residual error source.
• Requires a ground-truth object mask. No detection or segmentation stage is included.
• Small, sparsely visible objects fail: cup 12.0%, duck 28.0%, driller 30.5%.
• Evaluation subsamples the CAD vertices without a seed, so per-object figures move by a few points between runs; aggregate figures are stable to the precision quoted.
• At 11.4 M parameters with ImageNet pretraining it is not budget-matched to the 0.86 M point cloud pipeline; the report separates these confounds from the representation comparison.