NoPoSplat TartanAir (Robotics Trajectories)

License: Apache 2.0 Base Model: NoPoSplat (re10k) Hardware: 1x NVIDIA H100 (80GB) Dataset: TartanAir


Motivation & Background

Feed-forward generalizable 3D Gaussian Splatting models like NoPoSplat (Ye et al., ICLR 2025) reconstruct 3D scenes from unposed image pairs without test-time camera poses or COLMAP optimization. However, most pre-trained foundation models are trained on forward-facing real estate sequences (RealEstate10K) or gentle automotive driving datasets. When applied to aerial drones or ground robotics, they often degrade due to rapid angular rotations, aggressive pitch/roll changes, and open industrial indoor geometry.

I set up this run to fine-tune NoPoSplat on the TartanAir robotics dataset to adapt the pose-free transformer encoder and splat decoder to 6-DoF robotics trajectories.


Training Setup & Compute Constraints

The model was fine-tuned on a single 80GB NVIDIA H100 PCIe node using custom chunked trajectories:

  • Pretrained Foundation: Warm-started from re10k.ckpt (NoPoSplat RE10K weights) alongside MASt3R ViT-Large backbone features.
  • Data & Trajectories: 39 TartanAir trajectories across 5 diverse environments (office, office2, abandonedfactory, carwelding, oldtown). Trajectory NED frames were converted to standard OpenCV coordinates (+X right, +Y down, +Z forward).
  • VRAM & Batching: Batch size 16 at 256Γ—256 resolution, sustaining ~50–52GB VRAM across the training run.
  • Objectives: Joint MSE and VGG-based LPIPS perceptual loss on rendered novel views.

Compute Budget & Checkpoint Milestones

The project operated under a hard deadline on compute credits expiring on an H100 instance. Rather than reaching the full 150,000-step target, training was completed in two stages:

  1. Initial Run: Completed 80,000 steps (checkpoints/epoch_9999-step_80000.ckpt).
  2. Extended Run: Resumed with an automated 6-hour watchdog timer to maximize steps within budget, gracefully halting at step ~121,670 (checkpoints/epoch_14998-step_120000.ckpt).

Note on quantitative metrics: Because the run prioritised training steps within the remaining credit window, offline evaluation scripts (compute_metrics.py for formal PSNR/SSIM/LPIPS splits) were not run before compute shutdown. I plan on uploading the benchmark evaluation metrics soon. In the meantime, I have provided raw training logs (train.log, train_extended.log) and GPU telemetry directly in the repo for transparency.


Pre-Rendered Splats & Visual Sanity

To verify geometry qualitatively, I extracted sample 3D Gaussian Splat scenes directly from the checkpoint across unseen trajectory frames. These are saved in standard 3DGS .ply format under splats/:

  • splats/carwelding_Hard_*.ply: Fast rotational motion in low-light industrial environments.
  • splats/office_Easy_*.ply: Wide baseline indoor office sequences.

You can inspect any of these .ply files directly using WebGL splat viewers (such as PlayCanvas or SuperSplat).


Repository Structure

.
β”œβ”€β”€ checkpoints/
β”‚   β”œβ”€β”€ epoch_14998-step_120000.ckpt   # Step 120k checkpoint (~2.4 GB)
β”‚   └── epoch_9999-step_80000.ckpt     # Step 80k checkpoint (~2.4 GB)
β”œβ”€β”€ splats/                            # 47 pre-rendered 3D Gaussian .ply scenes
β”œβ”€β”€ configs/
β”‚   β”œβ”€β”€ tartanair.yaml                 # Dataset coordinate loader config
β”‚   └── tartanair_h100.yaml            # H100 experiment definition
β”œβ”€β”€ datasets/tartanair/                # Processed training/testing chunks
└── logs/
    β”œβ”€β”€ train.log                      # Initial run log (0–80k)
    β”œβ”€β”€ train_extended.log             # Extended resume log (80k–121k)
    β”œβ”€β”€ gpu_telemetry.csv              # H100 memory & power trace
    └── auto_stop_6h.log               # Watchdog timer log

References & Citation

If using this work, please cite the original NoPoSplat and TartanAir publications:

@inproceedings{ye2024noposplat,
  title     = {NoPoSplat: High-Quality Novel View Synthesis from Unposed Images via Generalizable 3D Gaussian Splatting},
  author    = {Ye, Botao and Li, Chenghao and Lee, Gim Hee},
  booktitle = {ICLR},
  year      = {2025}
}

@article{wang2020tartanair,
  title     = {TartanAir: A Dataset to Push the Limits of Visual SLAM},
  author    = {Wang, Wenshan and Zhu, Delong and Wang, Xiangwei and Hu, Yaoyu and Qiu, Yuheng and Wang, Chen and Hu, Yafei and Kapoor, Ashish and Scherer, Sebastian},
  journal   = {IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
  year      = {2020}
}
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support