LaCT l24_d768_ttt2x, sparse-view DL3DV 256x256, iter 65000

LICENSE-LaCT

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Tianyuan Zhang, Hao Tan
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the “Software”), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

README.md

@@ -0,0 +1,90 @@
+---
+license: mit
+tags:
+- novel-view-synthesis
+- lact
+- test-time-training
+- dl3dv
+---
+
+# test_lact
+
+A LaCT (Large-Chunk Test-Time Training) novel-view-synthesis model, trained from scratch on DL3DV at 256×256 with **sparse inputs (2–6 context views)**.
+
+The architecture is unmodified `LaCTLVSM` from [a1600012888/LaCT](https://github.com/a1600012888/LaCT) (commit `a648340`), using config `lact_l24_d768_ttt2x`: 24 blocks, dim 768, 312.3M parameters.
+
+This is **not** the official LaCT checkpoint. The official scene models use 32–64 input views, batch 768 and far more compute. This run used a small sparse-view recipe, so the numbers below are much lower than the paper's.
+
+## Files
+
+| file | content |
+|---|---|
+| `model.safetensors` | Model weights at iteration 65,000, fp32. Keys match `LaCTLVSM` exactly. No optimizer state. |
+| `configs/lact_l24_d768_ttt2x.yaml` | Model config, copied from the LaCT repo. |
+| `configs/train_abl_lact_l24.yaml` | Training config used for this run. |
+| `code/encoder_decoder_lact.py` | Wrapper used in training. It maps the training code's camera inputs (w2c + K) to LaCT's `c2w` / `fxfycxcy`. |
+| `eval_results.json` | PSNR / SSIM / LPIPS every 5k iterations, 0–65k. |
+| `logs/train.log` | Full training log. |
+| `LICENSE-LaCT` | MIT license of the LaCT code. |
+
+## Usage
+
+```python
+import sys, torch
+from omegaconf import OmegaConf
+from safetensors.torch import load_file
+
+# git clone https://github.com/a1600012888/LaCT && git -C LaCT checkout a648340
+sys.path.insert(0, "LaCT/lact_nvs")
+from model import LaCTLVSM
+
+cfg = OmegaConf.to_container(OmegaConf.load("configs/lact_l24_d768_ttt2x.yaml"), resolve=True)
+model = LaCTLVSM(**cfg)
+model.load_state_dict(load_file("model.safetensors"), strict=True)
+model = model.cuda().eval()
+
+# input_data = {"image": [B, N, 3, 256, 256] in [0, 1],
+#               "fxfycxcy": [B, N, 4] in pixels at 256x256,
+#               "c2w": [B, N, 4, 4]}
+# target_data = {"fxfycxcy": [B, T, 4], "c2w": [B, T, 4, 4]}
+with torch.no_grad(), torch.autocast("cuda", dtype=torch.bfloat16):
+    rendered = model(input_data, target_data)  # [B, T, 3, 256, 256], values in [0, 1]
+```
+
+LaCT's `lact_ttt.py` uses `torch.compile`, so the first call compiles.
+
+### Camera convention
+
+Cameras use the OpenCV convention for `c2w`. The training poses were normalized as follows:
+
+1. Re-center the scene on the first context camera.
+2. Divide translations by `1.35 × max(distance of the context cameras from the first camera)`.
+
+After this, the farthest context camera is at distance 1/1.35. Use the same normalization at inference.
+
+## Training
+
+- **Data:** DL3DV, `images_8` (480×270, distorted, as released), resized and cropped to 256×256.
+- **Views per sample:** 2–6 context views (weights 2:8, 3:2, 4:8, 5:2, 6:8) plus 8 target views.
+  - Context views are evenly spaced 10–20 frames apart, with jitter.
+  - Targets are drawn from the context span, extended by one spacing on each side.
+- **Optimization:**
+  - Batch size 16. The training code batches by view count: with c context views, each step has floor(2 × 16 / c) scenes.
+  - AdamW, lr 4e-4, betas (0.9, 0.95), weight decay 0.05.
+  - Warmup 3k iterations, then cosine decay to 0 at 65k.
+  - Grad clip 3.0.
+- **Loss:** L2 + 0.5 × perceptual.
+- **Hardware:** one GPU (B300), bf16 autocast. Activation checkpointing on the first 12 blocks; per-block `torch.compile`. About 24 hours.
+
+## Results
+
+The test set is 140 DL3DV benchmark scenes at 256×256, with fixed context/target view indices. "Nv" means N context views.
+
+| iter | 2v PSNR | 4v PSNR | 6v PSNR | 2v / 4v / 6v SSIM | 2v / 4v / 6v LPIPS |
+|---|---|---|---|---|---|
+| 5k | 16.14 | 17.67 | 18.24 | 0.416 / 0.440 / 0.448 | 0.665 / 0.642 / 0.637 |
+| 25k | 16.86 | 19.05 | 20.02 | 0.464 / 0.526 / 0.556 | 0.533 / 0.453 / 0.421 |
+| 45k | 17.42 | 20.20 | 21.24 | 0.493 / 0.584 / 0.619 | 0.491 / 0.381 / 0.345 |
+| **65k** | **17.65** | **20.68** | **21.78** | **0.504 / 0.607 / 0.644** | **0.475 / 0.358 / 0.320** |
+
+`eval_results.json` has every 5k step.

code/encoder_decoder_lact.py

@@ -0,0 +1,73 @@
+"""LaCT (Large-Chunk TTT, LVSM 계열) 를 abl kit 학습 루프에 끼우는 wrapper.
+
+모델 코드는 git clone 한 LaCT repo (lact_nvs/model.py, lact_ttt.py) 를 수정 없이 그대로 쓴다.
+kit 의 입력 (images, rays, camera_cond, num_cond_views) 을 LaCT 입력으로 바꿀 뿐이다:
+  camera_cond["extrinsics"] = 정규화된 c2w 의 역행렬 (w2c, normalize_for_da3=False)
+  camera_cond["intrinsics"] = 학습 해상도 (256x256) 기준 K
+-> LaCT 의 c2w / fxfycxcy. kit 의 renderer 가 쓰는 Plucker ray 와 같은 카메라 정규화
+   (첫 context 카메라 기준, 가장 먼 context 카메라가 1/1.35) 를 그대로 받는다.
+출력은 kit 모델과 같이 [context 이미지, 렌더링한 target] 을 view 축으로 붙여 돌려준다.
+"""
+
+import os
+import sys
+
+import torch
+import torch.nn as nn
+import torch.utils.checkpoint
+from omegaconf import OmegaConf
+
+
+def _import_lact(lact_nvs_dir):
+    # LaCT config 의 block type 이 "model.SelfAttention", "lact_ttt.FastWeightGluMLPMultihead"
+    # 처럼 top-level 모듈 이름이라 lact_nvs 폴더를 sys.path 에 넣어야 한다.
+    lact_nvs_dir = os.path.abspath(lact_nvs_dir)
+    if lact_nvs_dir not in sys.path:
+        sys.path.insert(0, lact_nvs_dir)
+    import model as lact_model  # noqa: E402  (LaCT/lact_nvs/model.py)
+    assert os.path.dirname(os.path.abspath(lact_model.__file__)) == lact_nvs_dir, lact_model.__file__
+    return lact_model
+
+
+class LaCTWrapper(nn.Module):
+    def __init__(self, lact_nvs_dir, lact_config, ckpt_blocks=0, compile_blocks=False):
+        super().__init__()
+        lact_model = _import_lact(lact_nvs_dir)
+        cfg = OmegaConf.to_container(OmegaConf.load(lact_config), resolve=True)
+        self.lact = lact_model.LaCTLVSM(**cfg)
+        # 학습 중 앞에서부터 ckpt_blocks 개 Block 을 activation checkpointing 한다 (0 = 끔).
+        # Block.forward 만 바꾸므로 state_dict key 는 LaCTLVSM 그대로다.
+        self.ckpt_blocks = int(ckpt_blocks)
+        for i, blk in enumerate(self.lact.blocks):
+            if i < self.ckpt_blocks:
+                blk.forward = self._ckpt_forward(blk.forward)
+        # LaCT README: TTT 모델은 compile 해야 제 속도가 난다. Block 을 하나씩 in-place 로 compile
+        # 하고 checkpoint 는 compile 안에서 돈다 (2026-11-13 memtest, B300):
+        #   LaCTLVSM 전체를 compile -> inductor 커널이 "misaligned address" 로 죽는다.
+        #   checkpoint 바깥 + compile 안쪽 -> backward 재계산에서 CheckpointError.
+        self.compile_blocks = bool(compile_blocks)
+        if self.compile_blocks:
+            # block 마다 guard (forward closure id) 가 달라 block 수만큼 재컴파일된다 (기본 한도 8).
+            torch._dynamo.config.recompile_limit = 128
+            torch._dynamo.config.accumulated_recompile_limit = 1024
+            for blk in self.lact.blocks:
+                blk.compile()
+
+    def _ckpt_forward(self, fwd):
+        def run(x, info):
+            if self.training and torch.is_grad_enabled():
+                return torch.utils.checkpoint.checkpoint(fwd, x, info, use_reentrant=False)
+            return fwd(x, info)
+        return run
+
+    def forward(self, images, rays, camera_cond, num_cond_views, timeit=False):
+        n = int(num_cond_views)
+        w2c = camera_cond["extrinsics"].to(images.device).float()
+        K = camera_cond["intrinsics"].to(images.device).float()
+        c2w = torch.linalg.inv(w2c)
+        fxfycxcy = torch.stack([K[..., 0, 0], K[..., 1, 1], K[..., 0, 2], K[..., 1, 2]], dim=-1)
+
+        input_data = {"image": images[:, :n].float(), "fxfycxcy": fxfycxcy[:, :n], "c2w": c2w[:, :n]}
+        target_data = {"fxfycxcy": fxfycxcy[:, n:], "c2w": c2w[:, n:]}
+        rendered = self.lact(input_data, target_data)
+        return torch.cat([images[:, :n], rendered.to(images.dtype)], dim=1)

configs/lact_l24_d768_ttt2x.yaml

@@ -0,0 +1,21 @@
+patch_size: 8
+dim: 768
+layers: 24
+
+block_config:
+  - type: model.SelfAttention
+    length_dim: l
+    params:
+      head_dim: 64
+      use_qk_norm: true
+  - type: lact_ttt.FastWeightGluMLPMultihead
+    length_dim: vl
+    params:
+      head_dim: 768 # 256
+      inter_multi: 2
+      base_lr: 0.01
+      muon_update_steps: 5
+  - type: model.MLP
+    length_dim: vl
+    params:
+      inter_multi: 4

configs/train_abl_lact_l24.yaml

@@ -0,0 +1,101 @@
+# abl_lact_l24: LaCT baseline (git clone https://github.com/a1600012888/LaCT @ a648340, lact_nvs 의
+# LaCTLVSM + config/lact_l24_d768_ttt2x.yaml, 처음부터 학습) 를 abl_da3_l12 와 같은 조건으로 돌린다 (2026-11-13, B300 GPU 3).
+#   abl_da3_l12.yaml 과 model 블록만 다르다: 데이터 / view (context 2-6 + target 8) / batch 16 / 256x256 /
+#   lr 4e-4, warmup 3000, cosine 65k / loss (L2 + 0.5 perceptual) / eval 5k / W&B abl_enc6 이 모두 같다.
+#   LaCT 는 encoder 가 없어서 카메라는 da3 형식 (w2c + K) 으로 받아 c2w / fxfycxcy 로 바꿔 쓴다
+#   (models/encoder_decoder_lact.py). LaCT repo 코드는 수정하지 않았다.
+#   memtest batch 16 (B300, 컨테이너 yglee_lact): checkpointing 없음 -> cond 2 OOM.
+#     ckpt 24: cond 2 37,862 MiB / 2.78 s | ckpt 12: cond 2 220,166 MiB / 2.58 s
+#     ckpt 12 + compile (채택): cond 2 179,558 MiB / 1.87 s, cond 4 109,538 MiB / 1.05 s, cond 6 81,436 MiB / 0.72 s
+module: train
+seed: 0
+
+opt:
+  lr: 4e-4
+  weight_decay: 0.05
+  num_iter_total: 65001     # GPU 2,3 를 이틀 쓰는 길이 (VGGT 2.3 s/iter + eval 5 분/2.5k -> 약 44 h)
+  lr_scheduler_iter: 65001
+  warmup_steps: 3000
+  batch_size: 16            # memtest 2026-11-08: 두 arm 공통 최대 (cond 2 가 최악: DA3 165.5 GB, VGGT 168.3 GB; 24 는 VGGT OOM)
+  batch_size_test: 16
+  target_has_input_p: 1.0
+  l2_loss_weight: 1.0
+  perceptual_loss_weight: 0.5
+  grad_clip_norm: 3.0
+  allowed_gradnorm_factor: 5
+  betas:
+  - 0.9
+  - 0.95
+  lr_scheduler: cosine
+  hold_steps: 0
+  cosine_min_range: 0.0
+  freeze_da3: false         # encoder 도 학습 (논문 256 모델과 같다)
+  print_every: 50
+
+model:
+  model_type: regressive
+  encoder: lact
+  lact_nvs_dir: /data/models/yglee_lager/LaCT/lact_nvs
+  lact_config: /data/models/yglee_lager/LaCT/lact_nvs/config/lact_l24_d768_ttt2x.yaml
+  lact_ckpt_blocks: 12        # activation checkpointing 하는 앞쪽 Block 수 (계산 결과는 같다)
+  lact_compile: true          # LaCT Block 을 torch.compile (LaCT README 권장)
+  da_model_weights_path: 'pretrained_weights/model_giant.safetensors'   # run.sh 가 넘기는 값. LaCT 는 쓰지 않는다
+
+data:
+  camera_condition_type: da3
+  subdataset_list:
+  - name: dl3dv
+    view_sampler_range:
+    - 10
+    - 20
+    expansion_factor: 1.0
+    equalization_length: 200000
+  num_cond_views_aug:
+  - 2
+  - 6
+  aspect_ratio_range:
+  - 1.0
+  - 1.0
+  im_size_hw:
+  - 256
+  - 256
+  num_views: 10
+  zero_out_cam_cond_p: 0.0
+  num_tgt_views: 8
+  view_to_sample_prob:
+  - 2,8
+  - 3,2
+  - 4,8
+  - 5,2
+  - 6,8
+
+test_data:
+  dataset_names: [dl3dv, dl3dv, dl3dv]
+  dataset_names_log: [dl3dv_2view, dl3dv_4view, dl3dv_6view]
+  zero_out_cam_cond_p: [0.0, 0.0, 0.0]
+  test_view_indices_paths: [assets/dl3dv_2v.json, assets/dl3dv_4v.json, assets/dl3dv_6v.json]
+  num_cond_views: [2, 4, 6]
+  camera_condition_type: da3
+
+view_sampler:
+  train_method: bounded_interp
+  test_method: fixed
+
+log_dir: ./output
+
+wandb:
+  api_key_path: ./config/api_keys.yaml
+  enabled: true
+  project: abl_enc6
+  entity: dudrms514
+  name: null
+  resume: allow
+  id: null
+  log_iter: 50
+  vis_iter: 500
+
+eval:
+  log_iter: 50
+  eval_at_start: true
+  eval_iter: 5000
+  ckpt_iter: 100000000      # 번호 붙은 체크포인트는 만들지 않는다 (latest 는 1000 마다 덮어쓴다)

eval_results.json

@@ -0,0 +1,254 @@
+[
+ {
+  "iter": 0,
+  "dl3dv_2view": {
+   "psnr": 11.421739,
+   "ssim": 0.065604,
+   "lpips": 0.788644
+  },
+  "dl3dv_4view": {
+   "psnr": 11.422578,
+   "ssim": 0.065605,
+   "lpips": 0.788819
+  },
+  "dl3dv_6view": {
+   "psnr": 11.422861,
+   "ssim": 0.065604,
+   "lpips": 0.788792
+  }
+ },
+ {
+  "iter": 5000,
+  "dl3dv_2view": {
+   "psnr": 16.143566,
+   "ssim": 0.416415,
+   "lpips": 0.66512
+  },
+  "dl3dv_4view": {
+   "psnr": 17.673761,
+   "ssim": 0.440066,
+   "lpips": 0.642324
+  },
+  "dl3dv_6view": {
+   "psnr": 18.242579,
+   "ssim": 0.44798,
+   "lpips": 0.636527
+  }
+ },
+ {
+  "iter": 10000,
+  "dl3dv_2view": {
+   "psnr": 16.489031,
+   "ssim": 0.439963,
+   "lpips": 0.605489
+  },
+  "dl3dv_4view": {
+   "psnr": 18.298374,
+   "ssim": 0.4804,
+   "lpips": 0.557058
+  },
+  "dl3dv_6view": {
+   "psnr": 19.074331,
+   "ssim": 0.499484,
+   "lpips": 0.538382
+  }
+ },
+ {
+  "iter": 15000,
+  "dl3dv_2view": {
+   "psnr": 16.557905,
+   "ssim": 0.446517,
+   "lpips": 0.564823
+  },
+  "dl3dv_4view": {
+   "psnr": 18.557508,
+   "ssim": 0.497642,
+   "lpips": 0.498675
+  },
+  "dl3dv_6view": {
+   "psnr": 19.409191,
+   "ssim": 0.521651,
+   "lpips": 0.473214
+  }
+ },
+ {
+  "iter": 20000,
+  "dl3dv_2view": {
+   "psnr": 16.463772,
+   "ssim": 0.448511,
+   "lpips": 0.54851
+  },
+  "dl3dv_4view": {
+   "psnr": 18.719131,
+   "ssim": 0.51003,
+   "lpips": 0.470424
+  },
+  "dl3dv_6view": {
+   "psnr": 19.628149,
+   "ssim": 0.536896,
+   "lpips": 0.442171
+  }
+ },
+ {
+  "iter": 25000,
+  "dl3dv_2view": {
+   "psnr": 16.858255,
+   "ssim": 0.46427,
+   "lpips": 0.533192
+  },
+  "dl3dv_4view": {
+   "psnr": 19.045271,
+   "ssim": 0.526044,
+   "lpips": 0.453275
+  },
+  "dl3dv_6view": {
+   "psnr": 20.020243,
+   "ssim": 0.555756,
+   "lpips": 0.420685
+  }
+ },
+ {
+  "iter": 30000,
+  "dl3dv_2view": {
+   "psnr": 16.958929,
+   "ssim": 0.469906,
+   "lpips": 0.521318
+  },
+  "dl3dv_4view": {
+   "psnr": 19.342415,
+   "ssim": 0.54211,
+   "lpips": 0.430941
+  },
+  "dl3dv_6view": {
+   "psnr": 20.310455,
+   "ssim": 0.573381,
+   "lpips": 0.396702
+  }
+ },
+ {
+  "iter": 35000,
+  "dl3dv_2view": {
+   "psnr": 16.954477,
+   "ssim": 0.474047,
+   "lpips": 0.5025
+  },
+  "dl3dv_4view": {
+   "psnr": 19.558046,
+   "ssim": 0.555949,
+   "lpips": 0.406279
+  },
+  "dl3dv_6view": {
+   "psnr": 20.597551,
+   "ssim": 0.58913,
+   "lpips": 0.372302
+  }
+ },
+ {
+  "iter": 40000,
+  "dl3dv_2view": {
+   "psnr": 17.270464,
+   "ssim": 0.484966,
+   "lpips": 0.493843
+  },
+  "dl3dv_4view": {
+   "psnr": 19.936426,
+   "ssim": 0.570776,
+   "lpips": 0.393177
+  },
+  "dl3dv_6view": {
+   "psnr": 20.957508,
+   "ssim": 0.604687,
+   "lpips": 0.356841
+  }
+ },
+ {
+  "iter": 45000,
+  "dl3dv_2view": {
+   "psnr": 17.417568,
+   "ssim": 0.492875,
+   "lpips": 0.491123
+  },
+  "dl3dv_4view": {
+   "psnr": 20.203033,
+   "ssim": 0.584476,
+   "lpips": 0.381447
+  },
+  "dl3dv_6view": {
+   "psnr": 21.237524,
+   "ssim": 0.618938,
+   "lpips": 0.344819
+  }
+ },
+ {
+  "iter": 50000,
+  "dl3dv_2view": {
+   "psnr": 17.475805,
+   "ssim": 0.496668,
+   "lpips": 0.479948
+  },
+  "dl3dv_4view": {
+   "psnr": 20.386845,
+   "ssim": 0.594407,
+   "lpips": 0.369613
+  },
+  "dl3dv_6view": {
+   "psnr": 21.483255,
+   "ssim": 0.631511,
+   "lpips": 0.331847
+  }
+ },
+ {
+  "iter": 55000,
+  "dl3dv_2view": {
+   "psnr": 17.616961,
+   "ssim": 0.502485,
+   "lpips": 0.475959
+  },
+  "dl3dv_4view": {
+   "psnr": 20.577383,
+   "ssim": 0.601876,
+   "lpips": 0.362314
+  },
+  "dl3dv_6view": {
+   "psnr": 21.660151,
+   "ssim": 0.638468,
+   "lpips": 0.324519
+  }
+ },
+ {
+  "iter": 60000,
+  "dl3dv_2view": {
+   "psnr": 17.627119,
+   "ssim": 0.502907,
+   "lpips": 0.475652
+  },
+  "dl3dv_4view": {
+   "psnr": 20.662294,
+   "ssim": 0.605849,
+   "lpips": 0.3587
+  },
+  "dl3dv_6view": {
+   "psnr": 21.750998,
+   "ssim": 0.642899,
+   "lpips": 0.320617
+  }
+ },
+ {
+  "iter": 65000,
+  "dl3dv_2view": {
+   "psnr": 17.649261,
+   "ssim": 0.50394,
+   "lpips": 0.474991
+  },
+  "dl3dv_4view": {
+   "psnr": 20.684305,
+   "ssim": 0.607008,
+   "lpips": 0.358075
+  },
+  "dl3dv_6view": {
+   "psnr": 21.778204,
+   "ssim": 0.644274,
+   "lpips": 0.319716
+  }
+ }
+]

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cc6ad4805ee577beaacbfccc97d3cc8da3cb20d3685c077aa7745cf94efc913c
+size 1249236840