license: mit
tags:
- novel-view-synthesis
- lact
- test-time-training
- dl3dv
test_lact
A LaCT (Large-Chunk Test-Time Training) novel-view-synthesis model, trained from scratch on DL3DV at 256×256 with sparse inputs (2–6 context views).
The architecture is unmodified LaCTLVSM from a1600012888/LaCT (commit a648340), using config lact_l24_d768_ttt2x: 24 blocks, dim 768, 312.3M parameters.
This is not the official LaCT checkpoint. The official scene models use 32–64 input views, batch 768 and far more compute. This run used a small sparse-view recipe, so the numbers below are much lower than the paper's.
Files
| file | content |
|---|---|
model.safetensors |
Model weights at iteration 65,000, fp32. Keys match LaCTLVSM exactly. No optimizer state. |
configs/lact_l24_d768_ttt2x.yaml |
Model config, copied from the LaCT repo. |
configs/train_abl_lact_l24.yaml |
Training config used for this run. |
code/encoder_decoder_lact.py |
Wrapper used in training. It maps the training code's camera inputs (w2c + K) to LaCT's c2w / fxfycxcy. |
eval_results.json |
PSNR / SSIM / LPIPS every 5k iterations, 0–65k. |
logs/train.log |
Full training log. |
LICENSE-LaCT |
MIT license of the LaCT code. |
Usage
import sys, torch
from omegaconf import OmegaConf
from safetensors.torch import load_file
# git clone https://github.com/a1600012888/LaCT && git -C LaCT checkout a648340
sys.path.insert(0, "LaCT/lact_nvs")
from model import LaCTLVSM
cfg = OmegaConf.to_container(OmegaConf.load("configs/lact_l24_d768_ttt2x.yaml"), resolve=True)
model = LaCTLVSM(**cfg)
model.load_state_dict(load_file("model.safetensors"), strict=True)
model = model.cuda().eval()
# input_data = {"image": [B, N, 3, 256, 256] in [0, 1],
# "fxfycxcy": [B, N, 4] in pixels at 256x256,
# "c2w": [B, N, 4, 4]}
# target_data = {"fxfycxcy": [B, T, 4], "c2w": [B, T, 4, 4]}
with torch.no_grad(), torch.autocast("cuda", dtype=torch.bfloat16):
rendered = model(input_data, target_data) # [B, T, 3, 256, 256], values in [0, 1]
LaCT's lact_ttt.py uses torch.compile, so the first call compiles.
Camera convention
Cameras use the OpenCV convention for c2w. The training poses were normalized as follows:
- Re-center the scene on the first context camera.
- Divide translations by
1.35 × max(distance of the context cameras from the first camera).
After this, the farthest context camera is at distance 1/1.35. Use the same normalization at inference.
Training
- Data: DL3DV,
images_8(480×270, distorted, as released), resized and cropped to 256×256. - Views per sample: 2–6 context views (weights 2:8, 3:2, 4:8, 5:2, 6:8) plus 8 target views.
- Context views are evenly spaced 10–20 frames apart, with jitter.
- Targets are drawn from the context span, extended by one spacing on each side.
- Optimization:
- Batch size 16. The training code batches by view count: with c context views, each step has floor(2 × 16 / c) scenes.
- AdamW, lr 4e-4, betas (0.9, 0.95), weight decay 0.05.
- Warmup 3k iterations, then cosine decay to 0 at 65k.
- Grad clip 3.0.
- Loss: L2 + 0.5 × perceptual.
- Hardware: one GPU (B300), bf16 autocast. Activation checkpointing on the first 12 blocks; per-block
torch.compile. About 24 hours.
Results
The test set is 140 DL3DV benchmark scenes at 256×256, with fixed context/target view indices. "Nv" means N context views.
| iter | 2v PSNR | 4v PSNR | 6v PSNR | 2v / 4v / 6v SSIM | 2v / 4v / 6v LPIPS |
|---|---|---|---|---|---|
| 5k | 16.14 | 17.67 | 18.24 | 0.416 / 0.440 / 0.448 | 0.665 / 0.642 / 0.637 |
| 25k | 16.86 | 19.05 | 20.02 | 0.464 / 0.526 / 0.556 | 0.533 / 0.453 / 0.421 |
| 45k | 17.42 | 20.20 | 21.24 | 0.493 / 0.584 / 0.619 | 0.491 / 0.381 / 0.345 |
| 65k | 17.65 | 20.68 | 21.78 | 0.504 / 0.607 / 0.644 | 0.475 / 0.358 / 0.320 |
eval_results.json has every 5k step.