CloudDiffusion: Diffusion-Based Scene Completion in the Point Cloud Domain

IEEE International Conference on Systems, Man, and Cybernetics (SMC) 2026, presented October 2026. Paper: https://arxiv.org/abs/2606.16048 (v3, 7 Aug 2026, the accepted version) · Code and ground-truth pipeline: https://github.com/A-C-Simon/sonata_ws (sonata-workspace/), development copy https://github.com/msannikov03/sonata_ws

Chidera Agbasiere*, Mikhail Sannikov*, Faith Ogunwoye, Erik Shaikhiev, Alex Kozinov, Ilya Mikhalchuk, Iana Zhura, Dzmitry Tsetserukou. * equal contribution. Intelligent Space Robotics Laboratory, Skoltech.

This repository was previously named businesslion/scaffdiff and its card described a planned journal follow-up ("ScaffDiff"). That follow-up is discontinued. This card now covers only the SMC 2026 paper; see "What changed" at the bottom.

What the paper does

LiDAR scene completion on SemanticKITTI with three independent components:

1. Multi-token Gaussian VAE. A PointNet encoder and a 32-query cross-attention pooler produce 32 × 32-d Gaussian latent tokens (1,024-d total); a transformer decoder with 5 cross-attention blocks and 8,000 learned point queries reconstructs the scene. Squared Chamfer distance 0.120 ± 0.026 m² on 50 seq. 08 validation frames at 7.1 M parameters and 1.6 ms/frame, with none of the global-pooling or codebook-collapse failures of prior point-cloud autoencoders.
2. Anchor-based ICP ground-truth refinement. Neighbouring scans are placed by pose, gathered over a sliding temporal window, aligned by anchor ICP against a rolling reference, and cropped; ICP transforms with implausible displacement are rejected in favour of the pose-only placement. Rebuilding the references this way over 23,201 training frames reduces the single-step diffusion teacher's squared Chamfer distance on seq. 08 (4,071 frames, 20,000-point subsample) from 0.396 ± 0.090 m² (standard pose-only accumulation, "v1 GT") to 0.024 ± 0.005 m² ("v2 GT"), a ~16× gain with no model change. Part of that gain comes from the refined references being denser and more compact, so v2-GT numbers are not unit-comparable with published baselines evaluated on standard SemanticKITTI ground truth (see Table III in the paper).
3. Single-step coordinate-space completion. The diffusion teacher (frozen Point Transformer V3 / Sonata encoder, 108 M parameters, plus an 8.9 M-parameter denoiser) completes a scene in one x₀ step directly in coordinate space, not in the VAE latent, starting from a noised scaffold at t = 200 of a 1,000-step cosine schedule. 209 ms/frame on one RTX 4090, which is 23–98× lower inference latency than the iterative diffusion baselines LiDiff (20.6 s/frame) and ScoreLiDAR (4.7 s/frame) re-timed on the same GPU.

The main finding is that data quality dominates model design at this scale: the ground-truth pipeline, not the architecture, produced the largest gain.

Checkpoints

File	Size	What it is	Where it appears
teacher_v2gt/best_model.pth	542 MB	Single-step diffusion teacher trained on the ICP-refined (v2) ground truth, 30 epochs	Paper Tables II and III, v2 row (0.024 m²)
teacher_v1gt/best_model.pth	542 MB	Same architecture trained on standard pose-only (v1) ground truth	Paper Tables II and III, v1 row (0.396 m²)
vae_v3/best_point_vae.pth	85 MB	Multi-token Gaussian VAE (7.1 M parameters, 32 tokens × 32-d, 8,000 output points)	Paper Table I

Legacy files kept for provenance, not used in the paper:

File	What it is
teacher_v2gt_ft/best_model.pth	A 3-epoch mixed-scaffold fine-tune of teacher_v2gt made for the discontinued follow-up study.
student_v1gt/best_model.pth, student_v2gt/best_model.pth	Student models from an earlier teacher-student framing that the paper does not use.

All teacher checkpoints share the frozen Sonata/PTv3 encoder loaded from the upstream Sonata release; only the denoiser (and, separately, the VAE) are trained.

Loading

• Teacher checkpoints are standard PyTorch state dicts of the diffusion teacher (frozen Sonata/PTv3 encoder + 8.9 M-parameter denoiser). Build the model with the teacher constructor in sonata-workspace/ of the code repository and load the state dict; inference uses single-step x₀ prediction at t = 200 from a noised scaffold.
• The VAE (vae_v3/best_point_vae.pth) is a separate PointCloudVAE (no Sonata encoder). Build it from models/point_cloud_vae.py with the architecture settings stored in the checkpoint dict (num_latent_tokens=32, num_dec_blocks=5, num_decoded_points=8000, …); the bare defaults differ and will not load. Apply the train-time per-sample normalisation (mean-centre, isotropic rescale to [-1, 1]) and invert it before computing Chamfer distance; the 0.120 m² figure is only reproduced with that normalisation. eval_vae_multitoken.py in the code repository does this end to end.

Data and compute

• SemanticKITTI sequences 00–07 and 09–10 for training, 08 for validation. The v2 ground truth (23,201 frames) is produced by the anchor-based ICP pipeline described in Section III of the paper; the scripts live under sonata-workspace/ in the code repository.
• One NVIDIA RTX 4090 (24 GB) is enough for training (about 10 hours, 30 epochs) and for inference at 209 ms/frame.

Figures

images/vae_v3_sample.png (VAE reconstruction), images/teacher_v2gt_sample1.png and images/teacher_v2gt_sample2.png (teacher completions on v2 GT). The other images in images/ belong to the discontinued follow-up and are not referenced by the paper.

Citation

@inproceedings{agbasiere2026clouddiffusion,
  title     = {CloudDiffusion: Diffusion-Based Scene Completion in the Point Cloud Domain},
  author    = {Agbasiere, Chidera and Sannikov, Mikhail and Ogunwoye, Faith and
               Shaikhiev, Erik and Kozinov, Alex and Mikhalchuk, Ilya and
               Zhura, Iana and Tsetserukou, Dzmitry},
  booktitle = {Proc. IEEE Int. Conf. on Systems, Man, and Cybernetics (SMC)},
  year      = {2026}
}

What changed

• Renamed from businesslion/scaffdiff (Hugging Face redirects the old name).
• The earlier card mixed this paper with a planned RA-L follow-up ("ScaffDiff", scaffold-dominance ablations, a matched-protocol comparison against LiDiff/ScoreLiDAR, and a 138×/65× latency claim). That follow-up is discontinued and those claims are withdrawn from this card. The latency comparison here follows the accepted paper: 23–98× against LiDiff and ScoreLiDAR re-timed on the same RTX 4090.
• Earlier still, the card described a teacher-student distillation story with student checkpoints; the paper contains no student model. The student files remain only for provenance.

License: MIT.