---
license: other
tags:
- world-model
- robotics
- tactile
- cosmos
---
# ACTS world model: sensor-only / no-force ablation (code + data + checkpoints)
This repo contains everything needed to run three action-conditioned visuo-tactile world-model runs on a new machine, and to train
the four follow-up experiments listed under **TODO**. **1116 TODO** (below): val-set inference of pushT / rope at epochs 14 and 15. It holds the code, the preprocessed data, the raw media and the trained
checkpoints. Only the two public base models are downloaded from their own repos.
| run | task | what it is | ckpt in this repo | status |
|---|---|---|---|---|
| `nano_action_causal_umi_sensoronly_noforce_v1` | motherboard | **Slurm job 884599** (`rebuttal_2`): sensor-only (no view-action streams) + no force | `checkpoints/…/epoch_013.pt` | training was still running at upload (epoch 14/20) |
| `nano_action_causal_pusht_v1` | pushT | full model (sensor + view actions, with force) | `checkpoints/…/epoch_019.pt` (+ `epoch_014/015.pt` for the 1116 val TODO) | done |
| `nano_action_causal_rope_v1` | rope | full model (sensor + view actions, with force) | `checkpoints/…/epoch_019.pt` (+ `epoch_014/015.pt` for the 1116 val TODO) | done |
## TODO experiments
The same ablation that was run on motherboard (sensor-only, with and without force), now on **pushT** and **rope**.
**pushT and rope are two separate tasks**: each is trained only on its own data, and each run is evaluated with **its own
inference script on its own test set**. Nothing is shared between the two tasks except the code.
| | **pushT** | **rope** |
|---|---|---|
| experiments | 1a `nano_action_causal_pusht_sensoronly_v1` (with force)
1b `nano_action_causal_pusht_sensoronly_noforce_v1` (no force) | 2a `nano_action_causal_rope_sensoronly_v1` (with force)
2b `nano_action_causal_rope_sensoronly_noforce_v1` (no force) |
| configs | `configs/nano_action_causal_pusht_sensoronly{,_noforce}_v1.yaml` | `configs/nano_action_causal_rope_sensoronly{,_noforce}_v1.yaml` |
| training data | `data/mixed_pushT_{latents,actions}`, split `splits/mixed_pusht_wrist_holdout.json`: 56 train / 4 val episodes, 55,199 train windows | `data/mixed_rope_{latents,actions}`, split `splits/mixed_rope_wrist_holdout.json`: 53 train / 8 val episodes, 77,188 train windows |
| training time (4× H100, 20 epochs) | ≈ 2.3 h/epoch (6,900 steps) → ~2 days per experiment | ≈ 3.3 h/epoch (9,648 steps) → ~3 days per experiment |
| **inference script** | **`scripts/infer_pusht.sh`** | **`scripts/infer_rope.sh`** |
| **test set** | test split of the pushT split file: 8 episodes → `eval/cases/pusht_test_cases.json`, **30 cases** (28 with a long rollout; cases 5 and 29 are short-only, the segment is too short) | test split of the rope split file: 9 episodes → `eval/cases/rope_test_cases.json`, **34 cases** (all with a long rollout) |
| compared against | `nano_action_causal_pusht_v1` epoch 019 (`eval/cases/pusht_reference_metrics.json`) | `nano_action_causal_rope_v1` epoch 019 (`eval/cases/rope_reference_metrics.json`) |
Each config differs from its parent (`nano_action_causal_pusht_v1` / `nano_action_causal_rope_v1`) only in `run_name`,
`model.active_streams` (no `view_action_*`), and for the no-force variants `data.action_file: sensor_actions_fps15_noforce.pt`.
### Requirements (all 4 experiments)
* **Each experiment is its own training job on 4× H100 80 GB (one node)**: one DDP rank per GPU, batch 2 per GPU = global batch 8,
~31 GiB per GPU. Do not change the GPU count or `training.batch_size`; the parent runs and job 884599 used exactly 4×2.
* **Train for the full 20 epochs** (`training.epochs: 20` → `epoch_000.pt` … `epoch_019.pt`). Do not stop early and do not use
`best.pt`. The training is finished only when `runs//epoch_019.pt` exists.
* If a Slurm job hits its time limit or is preempted, submit **the same command again**. It resumes from `runs//last.pt`;
do not change the run name.
* **Only after epoch 20**: run the inference script **of that task** on **that task's test set** (pushT run → `infer_pusht.sh`,
rope run → `infer_rope.sh`). It uploads the checkpoint and the test results to this HF repo, so it needs `HF_TOKEN` with
**write** access to `WitneyWW/acts-sensoronly-ablation`.
### Step 1: setup and data (once per machine)
```bash
bash scripts/setup_env.sh && source env.sh && export HF_TOKEN=hf_... # see section 1
$PYTHON_BIN scripts/download.py --what data raw ckpts nano wan_vae --tasks pusht # pushT: train + val + test data
$PYTHON_BIN scripts/download.py --what data raw ckpts nano wan_vae --tasks rope # rope: train + val + test data
```
### pushT (experiments 1a, 1b)
```bash
source env.sh
# train: two separate jobs, 4x H100 each, 20 epochs
sbatch -N1 --ntasks-per-node=4 --gres=gpu:4 scripts/train.sh configs/nano_action_causal_pusht_sensoronly_v1.yaml # 1a
sbatch -N1 --ntasks-per-node=4 --gres=gpu:4 scripts/train.sh configs/nano_action_causal_pusht_sensoronly_noforce_v1.yaml # 1b
# (without Slurm, on a 4x H100 machine: NGPU=4 bash scripts/train.sh configs/.yaml)
# after runs//epoch_019.pt exists: pushT inference script on the pushT test set (30 cases) + upload, 4 GPUs, ~1 h
sbatch -N1 --ntasks=1 --gres=gpu:4 scripts/infer_pusht.sh configs/nano_action_causal_pusht_sensoronly_v1.yaml # 1a
sbatch -N1 --ntasks=1 --gres=gpu:4 scripts/infer_pusht.sh configs/nano_action_causal_pusht_sensoronly_noforce_v1.yaml # 1b
```
### rope (experiments 2a, 2b)
```bash
source env.sh
# train: two separate jobs, 4x H100 each, 20 epochs
sbatch -N1 --ntasks-per-node=4 --gres=gpu:4 scripts/train.sh configs/nano_action_causal_rope_sensoronly_v1.yaml # 2a
sbatch -N1 --ntasks-per-node=4 --gres=gpu:4 scripts/train.sh configs/nano_action_causal_rope_sensoronly_noforce_v1.yaml # 2b
# after runs//epoch_019.pt exists: rope inference script on the rope test set (34 cases) + upload, 4 GPUs, ~1 h
sbatch -N1 --ntasks=1 --gres=gpu:4 scripts/infer_rope.sh configs/nano_action_causal_rope_sensoronly_v1.yaml # 2a
sbatch -N1 --ntasks=1 --gres=gpu:4 scripts/infer_rope.sh configs/nano_action_causal_rope_sensoronly_noforce_v1.yaml # 2b
```
Sanity checks at the start of training (`runs//train.log`): `[data] train episodes=56 windows=55199` (pushT) or
`train episodes=53 windows=77188` (rope), and `trainable=45,301,952`. Progress: `ls runs//epoch_*.pt`.
### What the inference scripts do
`scripts/infer_pusht.sh` / `scripts/infer_rope.sh` (both call `scripts/todo_eval_upload.py --task `):
1. refuse a config of the other task, and refuse to run unless `runs//epoch_019.pt` exists (20 epochs finished);
2. check that the test cases come from the test split of the same split file the model was trained with;
3. run `eval/run_test_case.py` with `epoch_019.pt` on **all test cases of that task** (each = short 8→8 prediction + 15-window long
rollout, UniPC 35 steps, seed 0), spread over the 4 GPUs. If interrupted, run the same command again: finished cases are skipped;
4. check every case finished and write `summary.md` (PSNR/SSIM/LPIPS + paired Δ vs `pusht_v1` / `rope_v1` epoch 019);
5. upload to this repo:
* `checkpoints//`: `epoch_019.pt`, `config.yaml`, `action_normalizer.json`, `train.log`, `metrics.jsonl`, …
* `results//test_epoch_019/`: every test case (metrics json, predicted latents, GT|PRED PNG strips, `short_gt_pred.mp4`,
`long_gt_pred.mp4`, ~5 GB per experiment) + `summary.md`.
### Checklist
| # | task | trained 20 epochs (`epoch_019.pt`) | inference on own test set | uploaded `checkpoints/` + `results/` |
|---|---|---|---|---|
| 1a sensor-only | pushT | ☐ | ☐ `infer_pusht.sh` (30 cases) | ☐ |
| 1b sensor-only no-force | pushT | ☐ | ☐ `infer_pusht.sh` (30 cases) | ☐ |
| 2a sensor-only | rope | ☐ | ☐ `infer_rope.sh` (34 cases) | ☐ |
| 2b sensor-only no-force | rope | ☐ | ☐ `infer_rope.sh` (34 cases) | ☐ |
### What the configs change
* **sensor-only** = the two per-hand view-action streams are not modelled (not tokenised; `view_action2llm9` frozen). The dataset
still reads `view_actions_fps15.pt`, so the window list and the action normaliser are identical to the parent run's. Trainable
params 45.30 M (vs 45.34 M); 1,310 tokens/sample (vs 1,342).
* **with force** = the sensor action is `sensor_actions_with_dforce_fps15.pt`: `dp = R_kᵀΔp + k_f·Δf·n`, so the change of normal
force enters as a displacement along the gel normal.
* **no force** = `sensor_actions_fps15_noforce.pt`: the same file with the `k_f·Δf·n` term removed (pure geometry); it differs from
the force file only in `dp_y`.
* Everything else = parent recipe: lr 5e-5 (100 warm-up steps), wd 0.01, bf16, LoRA r16/α32 on the GEN tower, Cosmos3-Nano init, seed 0.
## 1116 TODO: validation set (pushT / rope) at epochs 14 and 15
Added 2026-11-16. Run the **val** split of pushT and rope on **epoch 14 and epoch 15** (`epoch_014.pt`, `epoch_015.pt`) for
the parent runs and for the four TODO runs, so the sensor-only / no-force ablation can be compared on held-out val episodes
at matched epochs. This does not replace the epoch-19 test inference above; it is an extra check.
### The validation sets
Each task's val split is the `val` list of its own split file (the same file the run trains with; val ≠ test, whole source
episodes held out). The latents, actions and raw media of these episodes are in this repo under the same `data/` trees as
train/test, so nothing extra has to be prepared.
| task | split file | val episodes (segments) | val windows (stride 1, valid actions) | val cases | file |
|---|---|---|---:|---:|---|
| pushT | `splits/mixed_pusht_wrist_holdout.json` | 0910_ep002, 0911_ep003 (seg00, seg01), 0912_ep000 → 4 segments | 4,439 | **14** (all with a 15-window long rollout) | `eval/cases/pusht_val_cases.json` |
| rope | `splits/mixed_rope_wrist_holdout.json` | 0911_ep001 (2 segs), 0914_ep003 (4 segs), 0915_ep002 (2 segs) → 8 segments | 6,174 | **30** (all with a long rollout) | `eval/cases/rope_val_cases.json` |
Val cases use the test-case protocol: one case per (val segment × contact class `none`/`onset`/`release`/`full`, from
`max(F_L, F_R) > 0.5 N` on the initial 16-frame window) where such a window exists, the window drawn with seed 0 among those
with room for the 15-window rollout. Built by `scripts/build_val_cases.py`, which also checks that the parent config and both
TODO configs of the task give the **identical** val window index, so the same case file serves all three runs. (pushT val
has no `none` window in 0911_ep003; rope 0911_ep001_seg01 and 0914_ep003_seg02 have no `none` window.)
**Download** (only the val episodes of one task, + checkpoints incl. the parent `epoch_014.pt` / `epoch_015.pt`):
```bash
$PYTHON_BIN scripts/download.py --what data raw ckpts nano wan_vae --tasks pusht --splits val
$PYTHON_BIN scripts/download.py --what data raw ckpts nano wan_vae --tasks rope --splits val
```
**Load in Python** (the dataset must be built exactly like this, otherwise the case `index` points at a different window):
```python
import json
from omegaconf import OmegaConf
from vtm_cosmos.data.dataset import VtmWindowDataset
from vtm_cosmos.data.splits import load_split
cfg = OmegaConf.load("configs/nano_action_causal_rope_sensoronly_v1.yaml") # any pushT/rope config; needs `source env.sh`
episodes = load_split(cfg.data.split_file)["val"] # ['rope_0911_episode_001_seg00.pt', ...]
ds = VtmWindowDataset(cfg.data.latent_root, cfg.data.action_root, episodes,
action_file=cfg.data.action_file, window_stride=1,
include_view_actions=True, view_action_file=cfg.data.view_action_file,
require_valid_actions=True, # action_full drops invalid transitions
active_streams=tuple(cfg.model.active_streams) if cfg.model.get("active_streams") else None)
print(len(ds)) # rope 6174, pushT 4439
item = ds[0] # latents.{video,wrist_left,wrist_right,tactile_left,tactile_right}: (48,4,16,16) = 16 frames @ 15 fps
# actions.action_{left,right}: (15,9); actions.view_action_{left,right}: (16,9); action_valid.*; stream_present
cases = json.load(open("eval/cases/rope_val_cases.json"))["cases"]
c = cases[0]; assert ds.window_meta(c["index"]) == (c["episode"], c["t_start"])
```
Raw media of a val episode: `data/raw/videos//{view_middle,tactile_left,tactile_right,wrist_left,wrist_right}.mp4`,
`data/raw/parquet/.parquet`. During training the val split is only used for the per-epoch flow loss
(`data.val_window_stride: 50`).
### What to run (4× H100 per command, resumable, uploads to this repo)
`scripts/infer_val.sh ` = for epoch 14 and 15: all val cases of that task on that epoch's checkpoint
(`todo_eval_upload.py --split val --epoch E`) → `runs//val_epoch_01{4,5}/summary.md` (paired Δ vs the parent run
at the same epoch, if the parent has already been run) → upload `results//val_epoch_01{4,5}/` and, for the TODO runs,
`checkpoints//epoch_01{4,5}.pt`. It can start as soon as `runs//epoch_014.pt` exists (training can keep
running on another node; epoch 15 needs `epoch_015.pt`). **Run the parent first** so the TODO summaries get the paired column.
```bash
source env.sh && export HF_TOKEN=hf_... # write access
# pushT (14 val cases; ~30 min per epoch on 4 GPUs)
sbatch -N1 --ntasks=1 --gres=gpu:4 scripts/infer_val.sh configs/nano_action_causal_pusht_v1.yaml # parent, ckpts shipped
sbatch -N1 --ntasks=1 --gres=gpu:4 scripts/infer_val.sh configs/nano_action_causal_pusht_sensoronly_v1.yaml # 1a
sbatch -N1 --ntasks=1 --gres=gpu:4 scripts/infer_val.sh configs/nano_action_causal_pusht_sensoronly_noforce_v1.yaml # 1b
# rope (30 val cases; ~1 h per epoch on 4 GPUs)
sbatch -N1 --ntasks=1 --gres=gpu:4 scripts/infer_val.sh configs/nano_action_causal_rope_v1.yaml # parent, ckpts shipped
sbatch -N1 --ntasks=1 --gres=gpu:4 scripts/infer_val.sh configs/nano_action_causal_rope_sensoronly_v1.yaml # 2a
sbatch -N1 --ntasks=1 --gres=gpu:4 scripts/infer_val.sh configs/nano_action_causal_rope_sensoronly_noforce_v1.yaml # 2b
# one epoch only / no upload: EPOCHS=14 bash scripts/infer_val.sh --no-upload
# re-summarise later (e.g. after the parent finished):
$PYTHON_BIN eval/summarize.py --task rope --split val --root runs/nano_action_causal_rope_sensoronly_v1/val_epoch_014 \
--ref-root runs/nano_action_causal_rope_v1/val_epoch_014
```
Single case by hand: `$PYTHON_BIN eval/run_test_case.py --task pusht --split val --case 3 --ckpt `
(output `runs//val_epoch_014/03_.../`).
### 1116 checklist
| run | task | epoch 14 val | epoch 15 val | uploaded `results//val_epoch_01{4,5}/` |
|---|---|---|---|---|
| `nano_action_causal_pusht_v1` (parent) | pushT | ☐ 14 cases | ☐ 14 cases | ☐ |
| 1a `pusht_sensoronly_v1` | pushT | ☐ | ☐ | ☐ (+ ckpts) |
| 1b `pusht_sensoronly_noforce_v1` | pushT | ☐ | ☐ | ☐ (+ ckpts) |
| `nano_action_causal_rope_v1` (parent) | rope | ☐ 30 cases | ☐ 30 cases | ☐ |
| 2a `rope_sensoronly_v1` | rope | ☐ | ☐ | ☐ (+ ckpts) |
| 2b `rope_sensoronly_noforce_v1` | rope | ☐ | ☐ | ☐ (+ ckpts) |
## 1. Setup on a new machine
Tested on Linux, NVIDIA H100 80 GB, a driver that supports CUDA 12.8, and conda.
```bash
export HF_TOKEN=hf_... # this repo is private
# code only (skip the ~150 GB of data/ and checkpoints/ for now)
hf download WitneyWW/acts-sensoronly-ablation --exclude "data/*" "checkpoints/*" --local-dir acts-sensoronly-ablation
cd acts-sensoronly-ablation
bash scripts/setup_env.sh # conda env `acts_wm` (override ENV_NAME=..., CONDA_ROOT=...): torch 2.10.0+cu128,
# pinned deps, and NVIDIA/cosmos-framework @ 3f94d25 cloned into third_party/
source env.sh # paths + PYTHONPATH; PYTHON_BIN defaults to $CONDA_ROOT/envs/acts_wm/bin/python
$PYTHON_BIN scripts/download.py --what all # everything (~175 GB incl. Cosmos3-Nano 29 GB)
# or only what you need, e.g. evaluating pushT on its test split:
$PYTHON_BIN scripts/download.py --what data raw ckpts nano wan_vae --tasks pusht --splits test
```
`download.py` places files in `./data` and `./checkpoints`, which are the `env.sh` defaults of `ACTS_DATA_ROOT` / `ACTS_CKPT_ROOT`.
`--tasks` ∈ `mb` (motherboard), `pusht`, `rope`. Training needs the train and val splits (the default fetches all three).
If the env is not at `~/miniconda3/envs/acts_wm`, export `PYTHON_BIN=/path/to/env/bin/python` before `source env.sh`.
| variable (env.sh) | default | content |
|---|---|---|
| `ACTS_DATA_ROOT` | `./data` | `mixed_*_latents`, `mixed_*_actions`, `raw/` |
| `ACTS_CKPT_ROOT` | `./checkpoints` | `Cosmos3-Nano/`, `Wan2.2_VAE.pth`, `vae/lora_last.pt`, `nano_action_causal_*/epoch_XXX.pt` |
| `ACTS_RUNS_ROOT` | `./runs` | new training runs and eval outputs |
| `VTM_SOURCE_VIDEO_ROOT` | `$ACTS_DATA_ROOT/raw/videos` | mp4s used to rebuild the VAE anchor frame at decode time |
## 2. What is in this repo
```
configs/ exact configs of the 3 runs above + umi_sensoronly_v1 (force arm of 884599) + the 4 TODO configs
splits/ train/val/test episode lists (mixed_{mb,pusht,rope}_wrist_holdout.json)
vtm_cosmos/ model / data / train / infer package (Cosmos3 Generator adapter, causal mask, dataset, rollout)
eval/ run_test_case.py, summarize.py, cases/_test_cases.json (+ reference metrics),
cases/{pusht,rope}_val_cases.json (1116 TODO)
scripts/ setup_env.sh, download.py, train.sh, infer_pusht.sh / infer_rope.sh (TODO: per-task test inference + upload,
via todo_eval_upload.py), infer_val.sh (1116 TODO: val at epochs 14/15), build_val_cases.py,
select_contact_windows.py
results/ (filled by the TODO runs) /test_epoch_019/: per-case test outputs + summary.md;
/val_epoch_01{4,5}/ (1116 TODO)
preprocessing/ raw mp4 + parquet -> latents + actions (see section 5)
third_party/wan22_vae/ standalone Wan2.2 VAE + tactile LoRA + encoder + pad_latent.py
data/
mixed_{mb,pushT,rope}_latents/{tac,visual,wrist}/.pt Wan2.2 VAE latents, (N_windows,48,4,16,16) fp16 per stream
mixed_{mb,pushT,rope}_actions//
sensor_actions_with_dforce_fps15.pt per-hand 9-D sensor action, force as k_f·Δf·n (pusht_v1, rope_v1, TODO 1a/2a)
sensor_actions_fps15_noforce.pt same without the force term (884599, TODO 1b/2b)
view_actions_fps15.pt per-hand view action (read by every run; modelled only by pusht_v1 / rope_v1)
raw/videos//{view_middle,tactile_left,tactile_right,wrist_left,wrist_right}.mp4 30 fps source
raw/parquet/.parquet, raw/episodes.json poses + normal force
checkpoints/
vae/lora_last.pt tactile (GelSight) LoRA of the Wan2.2 VAE
nano_action_causal_umi_sensoronly_noforce_v1/epoch_013.pt (+ config.yaml, action_normalizer.json, train.log, metrics.jsonl)
nano_action_causal_pusht_v1/epoch_{014,015,019}.pt (+ same files)
nano_action_causal_rope_v1/epoch_{014,015,019}.pt (+ same files)
```
Data = exactly the episodes of the three split files (motherboard 21/2/5, pushT 56/4/8, rope 53/8/9 train/val/test = 166 episodes;
latents + actions 117 GB, raw 16 GB). Training windows (stride 1, after dropping windows with an invalid action transition):
motherboard 71,386, pushT 55,199, rope 77,188.
## 3. Inference / evaluation
Test protocol: per task, a fixed set of contact-stratified cases (`none` / `onset` / `release` / `full`, from the GelSight normal force,
`max(F_L, F_R) > 0.5 N`). Each case = one short prediction (8 observed → 8 predicted frames) plus one autoregressive rollout of 15
windows (120 frames) from the same start, conditioned on the recorded actions; seed 0, UniPC 35 steps per latent block.
```bash
source env.sh
$PYTHON_BIN eval/run_test_case.py --task rope --case 0 # one case (~7 min on H100; ~40 s with --short-only)
$PYTHON_BIN eval/run_test_case.py --task rope --case all # 34 cases, sequential, resumable
$PYTHON_BIN eval/summarize.py --task rope --reference # table + paired Δ vs the shipped reference
$PYTHON_BIN eval/run_test_case.py --task umi_sensoronly_noforce --case all # job 884599's checkpoint, 20 motherboard cases
```
| `--task` | cases | default checkpoint |
|---|---:|---|
| `pusht` | 30 (28 with a long rollout) | `nano_action_causal_pusht_v1/epoch_019.pt` |
| `rope` | 34 | `nano_action_causal_rope_v1/epoch_019.pt` |
| `umi_sensoronly_noforce` | 20 | `nano_action_causal_umi_sensoronly_noforce_v1/epoch_013.pt` |
Outputs: `$ACTS_RUNS_ROOT//test_//` with `window_eval.json` (short metrics),
`reference_rollout.json` (per-window long metrics), `*_pred.pt`, GT/PRED strip PNGs, `short_gt_pred.mp4`, `long_gt_pred.mp4`.
Cases are independent, so on a cluster run them as an array job, e.g.
`sbatch --array=0-33 --gres=gpu:1 --wrap "source env.sh; \$PYTHON_BIN eval/run_test_case.py --task rope --case \$SLURM_ARRAY_TASK_ID"`.
Lower-level CLI for arbitrary windows (the checkpoint stores the original absolute paths, hence the overrides):
```bash
$PYTHON_BIN -m vtm_cosmos.infer.rollout --ckpt $ACTS_CKPT_ROOT/nano_action_causal_rope_v1/epoch_019.pt \
--split test --window-stride 1 --windows 0 500 --rollout-windows 15 --steps 35 --solver unipc --seed 0 \
--out-dir runs/my_eval --override data.latent_root=$ACTS_DATA_ROOT/mixed_rope_latents \
data.action_root=$ACTS_DATA_ROOT/mixed_rope_actions data.split_file=splits/mixed_rope_wrist_holdout.json \
model.nano_checkpoint=$ACTS_CKPT_ROOT/Cosmos3-Nano
```
### Reference results (test split, PSNR dB)
| run | n | horizon | video | wrist_left | wrist_right | tactile_left | tactile_right |
|---|---:|---|---:|---:|---:|---:|---:|
| rope_v1 e019 | 34 | short | 28.92 | 22.30 | 23.70 | 46.08 | 46.51 |
| rope_v1 e019 | 34 | long | 23.68 | 16.40 | 18.22 | 38.26 | 40.09 |
| pusht_v1 e019 | 30 | short | 29.68 | 22.86 | 22.76 | 46.43 | 48.37 |
| pusht_v1 e019 | 28 | long | 23.28 | 14.77 | 15.50 | 39.21 | 39.65 |
| umi_sensoronly_noforce e013 | 20 | short | 27.54 | 17.56 | 20.49 | 41.56 | 45.94 |
| umi_sensoronly_noforce e013 | 20 | long | 21.02 | 13.53 | 14.35 | 34.53 | 35.51 |
SSIM / LPIPS and per-case numbers: `eval/cases/_reference_metrics.json`.
## 4. Training
```bash
source env.sh
bash scripts/train.sh configs/nano_action_causal_rope_v1.yaml # all local GPUs (torchrun)
NGPU=1 bash scripts/train.sh configs/nano_action_causal_rope_sensoronly_v1.yaml \
run_name=smoke training.max_steps=20 # quick smoke test (~31 GiB)
sbatch -N1 --ntasks-per-node=4 --gres=gpu:4 scripts/train.sh configs/.yaml # Slurm
```
Any `key=value` after the config is an OmegaConf dot-list override. The run directory gets `config.yaml`, `train.log`, `metrics.jsonl`,
`action_normalizer.json` (quantile normaliser fitted on the train split), `epoch_XXX.pt`, `last.pt`, `best.pt` (lowest val flow loss;
only a monitor: report the last or a matched epoch). Re-launching with the same `run_name` resumes from `last.pt`.
## 5. Preprocessing (raw → latents + actions)
Only needed to rebuild or extend the data; `data/` already holds its output.
```bash
bash preprocessing/run_preprocess.sh rope # -> $ACTS_DATA_ROOT/rebuilt/mixed_rope_{latents,actions}
```
(`run_preprocess.sh` task names: `umi_all` = motherboard, `rope`, `pusht`.) Latents: 30 → 15 fps, 256² crop, sliding 16-frame windows,
Wan2.2 VAE (tactile with the GelSight LoRA), anchor latent dropped. Sensor actions: `build_force_actions.py --force-term delta`
(→ `..._with_dforce_fps15.pt`). View actions: `build_view_actions.py --stride 2`.
## Notes and pitfalls
* Always `source env.sh` first. It sets `PYTHONNOUSERSITE=1` and puts `third_party/cosmos-framework` + `third_party/wan22_vae` on `PYTHONPATH`.
* `setup_env.sh` pins torch 2.10.0+cu128 and removes `nvidia-*-cu13` libraries that some PyPI deps pull in, because they break cuDNN
convolutions (`CUDNN_STATUS_NOT_INITIALIZED`). If you install extra packages, re-run its last block.
* `imageio-ffmpeg` is pinned to 0.6.0 (ffmpeg 7.0.2). Bit-exact decoding of the anchor frame depends on the ffmpeg build.
* The first training / inference step compiles the FlexAttention Triton kernels, which takes a few minutes.
* Code relative to the 11-08 release `WitneyWW/acts-world-model`: `streams.py`, `model/layout.py` and `model/generator_adapter.py`
allow `action_full` runs without the view-action pair (sensor-only), and the eval scripts take any checkpoint.
Checks done before upload: `REPRO_CHECKS.md`.