NoPoSplat TartanAir (Robotics Trajectories)
Motivation & Background
Feed-forward generalizable 3D Gaussian Splatting models like NoPoSplat (Ye et al., ICLR 2025) reconstruct 3D scenes from unposed image pairs without test-time camera poses or COLMAP optimization. However, most pre-trained foundation models are trained on forward-facing real estate sequences (RealEstate10K) or gentle automotive driving datasets. When applied to aerial drones or ground robotics, they often degrade due to rapid angular rotations, aggressive pitch/roll changes, and open industrial indoor geometry.
I set up this run to fine-tune NoPoSplat on the TartanAir robotics dataset to adapt the pose-free transformer encoder and splat decoder to 6-DoF robotics trajectories.
Training Setup & Compute Constraints
The model was fine-tuned on a single 80GB NVIDIA H100 PCIe node using custom chunked trajectories:
- Pretrained Foundation: Warm-started from
re10k.ckpt(NoPoSplat RE10K weights) alongside MASt3R ViT-Large backbone features. - Data & Trajectories: 39 TartanAir trajectories across 5 diverse environments (
office,office2,abandonedfactory,carwelding,oldtown). Trajectory NED frames were converted to standard OpenCV coordinates (+Xright,+Ydown,+Zforward). - VRAM & Batching: Batch size 16 at 256Γ256 resolution, sustaining ~50β52GB VRAM across the training run.
- Objectives: Joint MSE and VGG-based LPIPS perceptual loss on rendered novel views.
Compute Budget & Checkpoint Milestones
The project operated under a hard deadline on compute credits expiring on an H100 instance. Rather than reaching the full 150,000-step target, training was completed in two stages:
- Initial Run: Completed 80,000 steps (
checkpoints/epoch_9999-step_80000.ckpt). - Extended Run: Resumed with an automated 6-hour watchdog timer to maximize steps within budget, gracefully halting at step ~121,670 (
checkpoints/epoch_14998-step_120000.ckpt).
Note on quantitative metrics: Because the run prioritised training steps within the remaining credit window, offline evaluation scripts (compute_metrics.py for formal PSNR/SSIM/LPIPS splits) were not run before compute shutdown. I plan on uploading the benchmark evaluation metrics soon. In the meantime, I have provided raw training logs (train.log, train_extended.log) and GPU telemetry directly in the repo for transparency.
Pre-Rendered Splats & Visual Sanity
To verify geometry qualitatively, I extracted sample 3D Gaussian Splat scenes directly from the checkpoint across unseen trajectory frames. These are saved in standard 3DGS .ply format under splats/:
splats/carwelding_Hard_*.ply: Fast rotational motion in low-light industrial environments.splats/office_Easy_*.ply: Wide baseline indoor office sequences.
You can inspect any of these .ply files directly using WebGL splat viewers (such as PlayCanvas or SuperSplat).
Repository Structure
.
βββ checkpoints/
β βββ epoch_14998-step_120000.ckpt # Step 120k checkpoint (~2.4 GB)
β βββ epoch_9999-step_80000.ckpt # Step 80k checkpoint (~2.4 GB)
βββ splats/ # 47 pre-rendered 3D Gaussian .ply scenes
βββ configs/
β βββ tartanair.yaml # Dataset coordinate loader config
β βββ tartanair_h100.yaml # H100 experiment definition
βββ datasets/tartanair/ # Processed training/testing chunks
βββ logs/
βββ train.log # Initial run log (0β80k)
βββ train_extended.log # Extended resume log (80kβ121k)
βββ gpu_telemetry.csv # H100 memory & power trace
βββ auto_stop_6h.log # Watchdog timer log
References & Citation
If using this work, please cite the original NoPoSplat and TartanAir publications:
@inproceedings{ye2024noposplat,
title = {NoPoSplat: High-Quality Novel View Synthesis from Unposed Images via Generalizable 3D Gaussian Splatting},
author = {Ye, Botao and Li, Chenghao and Lee, Gim Hee},
booktitle = {ICLR},
year = {2025}
}
@article{wang2020tartanair,
title = {TartanAir: A Dataset to Push the Limits of Visual SLAM},
author = {Wang, Wenshan and Zhu, Delong and Wang, Xiangwei and Hu, Yaoyu and Qiu, Yuheng and Wang, Chen and Hu, Yafei and Kapoor, Ashish and Scherer, Sebastian},
journal = {IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
year = {2020}
}