| --- |
| license: mit |
| tags: |
| - novel-view-synthesis |
| - lact |
| - test-time-training |
| - dl3dv |
| --- |
| |
| # test_lact |
| |
| A LaCT (Large-Chunk Test-Time Training) novel-view-synthesis model, trained from scratch on DL3DV at 256×256 with **sparse inputs (2–6 context views)**. |
| |
| The architecture is unmodified `LaCTLVSM` from [a1600012888/LaCT](https://github.com/a1600012888/LaCT) (commit `a648340`), using config `lact_l24_d768_ttt2x`: 24 blocks, dim 768, 312.3M parameters. |
|
|
| This is **not** the official LaCT checkpoint. The official scene models use 32–64 input views, batch 768 and far more compute. This run used a small sparse-view recipe, so the numbers below are much lower than the paper's. |
|
|
| ## Files |
|
|
| | file | content | |
| |---|---| |
| | `model.safetensors` | Model weights at iteration 65,000, fp32. Keys match `LaCTLVSM` exactly. No optimizer state. | |
| | `configs/lact_l24_d768_ttt2x.yaml` | Model config, copied from the LaCT repo. | |
| | `configs/train_abl_lact_l24.yaml` | Training config used for this run. | |
| | `code/encoder_decoder_lact.py` | Wrapper used in training. It maps the training code's camera inputs (w2c + K) to LaCT's `c2w` / `fxfycxcy`. | |
| | `eval_results.json` | PSNR / SSIM / LPIPS every 5k iterations, 0–65k. | |
| | `logs/train.log` | Full training log. | |
| | `LICENSE-LaCT` | MIT license of the LaCT code. | |
|
|
| ## Usage |
|
|
| ```python |
| import sys, torch |
| from omegaconf import OmegaConf |
| from safetensors.torch import load_file |
| |
| # git clone https://github.com/a1600012888/LaCT && git -C LaCT checkout a648340 |
| sys.path.insert(0, "LaCT/lact_nvs") |
| from model import LaCTLVSM |
| |
| cfg = OmegaConf.to_container(OmegaConf.load("configs/lact_l24_d768_ttt2x.yaml"), resolve=True) |
| model = LaCTLVSM(**cfg) |
| model.load_state_dict(load_file("model.safetensors"), strict=True) |
| model = model.cuda().eval() |
| |
| # input_data = {"image": [B, N, 3, 256, 256] in [0, 1], |
| # "fxfycxcy": [B, N, 4] in pixels at 256x256, |
| # "c2w": [B, N, 4, 4]} |
| # target_data = {"fxfycxcy": [B, T, 4], "c2w": [B, T, 4, 4]} |
| with torch.no_grad(), torch.autocast("cuda", dtype=torch.bfloat16): |
| rendered = model(input_data, target_data) # [B, T, 3, 256, 256], values in [0, 1] |
| ``` |
|
|
| LaCT's `lact_ttt.py` uses `torch.compile`, so the first call compiles. |
|
|
| ### Camera convention |
|
|
| Cameras use the OpenCV convention for `c2w`. The training poses were normalized as follows: |
|
|
| 1. Re-center the scene on the first context camera. |
| 2. Divide translations by `1.35 × max(distance of the context cameras from the first camera)`. |
|
|
| After this, the farthest context camera is at distance 1/1.35. Use the same normalization at inference. |
|
|
| ## Training |
|
|
| - **Data:** DL3DV, `images_8` (480×270, distorted, as released), resized and cropped to 256×256. |
| - **Views per sample:** 2–6 context views (weights 2:8, 3:2, 4:8, 5:2, 6:8) plus 8 target views. |
| - Context views are evenly spaced 10–20 frames apart, with jitter. |
| - Targets are drawn from the context span, extended by one spacing on each side. |
| - **Optimization:** |
| - Batch size 16. The training code batches by view count: with c context views, each step has floor(2 × 16 / c) scenes. |
| - AdamW, lr 4e-4, betas (0.9, 0.95), weight decay 0.05. |
| - Warmup 3k iterations, then cosine decay to 0 at 65k. |
| - Grad clip 3.0. |
| - **Loss:** L2 + 0.5 × perceptual. |
| - **Hardware:** one GPU (B300), bf16 autocast. Activation checkpointing on the first 12 blocks; per-block `torch.compile`. About 24 hours. |
|
|
| ## Results |
|
|
| The test set is 140 DL3DV benchmark scenes at 256×256, with fixed context/target view indices. "Nv" means N context views. |
|
|
| | iter | 2v PSNR | 4v PSNR | 6v PSNR | 2v / 4v / 6v SSIM | 2v / 4v / 6v LPIPS | |
| |---|---|---|---|---|---| |
| | 5k | 16.14 | 17.67 | 18.24 | 0.416 / 0.440 / 0.448 | 0.665 / 0.642 / 0.637 | |
| | 25k | 16.86 | 19.05 | 20.02 | 0.464 / 0.526 / 0.556 | 0.533 / 0.453 / 0.421 | |
| | 45k | 17.42 | 20.20 | 21.24 | 0.493 / 0.584 / 0.619 | 0.491 / 0.381 / 0.345 | |
| | **65k** | **17.65** | **20.68** | **21.78** | **0.504 / 0.607 / 0.644** | **0.475 / 0.358 / 0.320** | |
|
|
| `eval_results.json` has every 5k step. |
|
|
