WitneyWW's picture
1116 TODO: pushT/rope val cases + val inference at epochs 14/15 (infer_val.sh), README
fc95770 verified
|
Raw History Blame Contribute Delete
25.3 kB
metadata
license: other
tags:
  - world-model
  - robotics
  - tactile
  - cosmos

ACTS world model: sensor-only / no-force ablation (code + data + checkpoints)

This repo contains everything needed to run three action-conditioned visuo-tactile world-model runs on a new machine, and to train the four follow-up experiments listed under TODO. 1116 TODO (below): val-set inference of pushT / rope at epochs 14 and 15. It holds the code, the preprocessed data, the raw media and the trained checkpoints. Only the two public base models are downloaded from their own repos.

run task what it is ckpt in this repo status
nano_action_causal_umi_sensoronly_noforce_v1 motherboard Slurm job 884599 (rebuttal_2): sensor-only (no view-action streams) + no force checkpoints/…/epoch_013.pt training was still running at upload (epoch 14/20)
nano_action_causal_pusht_v1 pushT full model (sensor + view actions, with force) checkpoints/…/epoch_019.pt (+ epoch_014/015.pt for the 1116 val TODO) done
nano_action_causal_rope_v1 rope full model (sensor + view actions, with force) checkpoints/…/epoch_019.pt (+ epoch_014/015.pt for the 1116 val TODO) done

TODO experiments

The same ablation that was run on motherboard (sensor-only, with and without force), now on pushT and rope. pushT and rope are two separate tasks: each is trained only on its own data, and each run is evaluated with its own inference script on its own test set. Nothing is shared between the two tasks except the code.

pushT rope
experiments 1a nano_action_causal_pusht_sensoronly_v1 (with force)
1b nano_action_causal_pusht_sensoronly_noforce_v1 (no force)
2a nano_action_causal_rope_sensoronly_v1 (with force)
2b nano_action_causal_rope_sensoronly_noforce_v1 (no force)
configs configs/nano_action_causal_pusht_sensoronly{,_noforce}_v1.yaml configs/nano_action_causal_rope_sensoronly{,_noforce}_v1.yaml
training data data/mixed_pushT_{latents,actions}, split splits/mixed_pusht_wrist_holdout.json: 56 train / 4 val episodes, 55,199 train windows data/mixed_rope_{latents,actions}, split splits/mixed_rope_wrist_holdout.json: 53 train / 8 val episodes, 77,188 train windows
training time (4Γ— H100, 20 epochs) β‰ˆ 2.3 h/epoch (6,900 steps) β†’ ~2 days per experiment β‰ˆ 3.3 h/epoch (9,648 steps) β†’ ~3 days per experiment
inference script scripts/infer_pusht.sh scripts/infer_rope.sh
test set test split of the pushT split file: 8 episodes β†’ eval/cases/pusht_test_cases.json, 30 cases (28 with a long rollout; cases 5 and 29 are short-only, the segment is too short) test split of the rope split file: 9 episodes β†’ eval/cases/rope_test_cases.json, 34 cases (all with a long rollout)
compared against nano_action_causal_pusht_v1 epoch 019 (eval/cases/pusht_reference_metrics.json) nano_action_causal_rope_v1 epoch 019 (eval/cases/rope_reference_metrics.json)

Each config differs from its parent (nano_action_causal_pusht_v1 / nano_action_causal_rope_v1) only in run_name, model.active_streams (no view_action_*), and for the no-force variants data.action_file: sensor_actions_fps15_noforce.pt.

Requirements (all 4 experiments)

  • Each experiment is its own training job on 4Γ— H100 80 GB (one node): one DDP rank per GPU, batch 2 per GPU = global batch 8, ~31 GiB per GPU. Do not change the GPU count or training.batch_size; the parent runs and job 884599 used exactly 4Γ—2.
  • Train for the full 20 epochs (training.epochs: 20 β†’ epoch_000.pt … epoch_019.pt). Do not stop early and do not use best.pt. The training is finished only when runs/<run_name>/epoch_019.pt exists.
  • If a Slurm job hits its time limit or is preempted, submit the same command again. It resumes from runs/<run_name>/last.pt; do not change the run name.
  • Only after epoch 20: run the inference script of that task on that task's test set (pushT run β†’ infer_pusht.sh, rope run β†’ infer_rope.sh). It uploads the checkpoint and the test results to this HF repo, so it needs HF_TOKEN with write access to WitneyWW/acts-sensoronly-ablation.

Step 1: setup and data (once per machine)

bash scripts/setup_env.sh && source env.sh && export HF_TOKEN=hf_...         # see section 1
$PYTHON_BIN scripts/download.py --what data raw ckpts nano wan_vae --tasks pusht   # pushT: train + val + test data
$PYTHON_BIN scripts/download.py --what data raw ckpts nano wan_vae --tasks rope    # rope:  train + val + test data

pushT (experiments 1a, 1b)

source env.sh
# train: two separate jobs, 4x H100 each, 20 epochs
sbatch -N1 --ntasks-per-node=4 --gres=gpu:4 scripts/train.sh configs/nano_action_causal_pusht_sensoronly_v1.yaml          # 1a
sbatch -N1 --ntasks-per-node=4 --gres=gpu:4 scripts/train.sh configs/nano_action_causal_pusht_sensoronly_noforce_v1.yaml  # 1b
#   (without Slurm, on a 4x H100 machine: NGPU=4 bash scripts/train.sh configs/<config>.yaml)

# after runs/<run_name>/epoch_019.pt exists: pushT inference script on the pushT test set (30 cases) + upload, 4 GPUs, ~1 h
sbatch -N1 --ntasks=1 --gres=gpu:4 scripts/infer_pusht.sh configs/nano_action_causal_pusht_sensoronly_v1.yaml           # 1a
sbatch -N1 --ntasks=1 --gres=gpu:4 scripts/infer_pusht.sh configs/nano_action_causal_pusht_sensoronly_noforce_v1.yaml   # 1b

rope (experiments 2a, 2b)

source env.sh
# train: two separate jobs, 4x H100 each, 20 epochs
sbatch -N1 --ntasks-per-node=4 --gres=gpu:4 scripts/train.sh configs/nano_action_causal_rope_sensoronly_v1.yaml           # 2a
sbatch -N1 --ntasks-per-node=4 --gres=gpu:4 scripts/train.sh configs/nano_action_causal_rope_sensoronly_noforce_v1.yaml   # 2b

# after runs/<run_name>/epoch_019.pt exists: rope inference script on the rope test set (34 cases) + upload, 4 GPUs, ~1 h
sbatch -N1 --ntasks=1 --gres=gpu:4 scripts/infer_rope.sh configs/nano_action_causal_rope_sensoronly_v1.yaml             # 2a
sbatch -N1 --ntasks=1 --gres=gpu:4 scripts/infer_rope.sh configs/nano_action_causal_rope_sensoronly_noforce_v1.yaml     # 2b

Sanity checks at the start of training (runs/<run_name>/train.log): [data] train episodes=56 windows=55199 (pushT) or train episodes=53 windows=77188 (rope), and trainable=45,301,952. Progress: ls runs/<run_name>/epoch_*.pt.

What the inference scripts do

scripts/infer_pusht.sh / scripts/infer_rope.sh (both call scripts/todo_eval_upload.py --task <task>):

  1. refuse a config of the other task, and refuse to run unless runs/<run_name>/epoch_019.pt exists (20 epochs finished);
  2. check that the test cases come from the test split of the same split file the model was trained with;
  3. run eval/run_test_case.py with epoch_019.pt on all test cases of that task (each = short 8β†’8 prediction + 15-window long rollout, UniPC 35 steps, seed 0), spread over the 4 GPUs. If interrupted, run the same command again: finished cases are skipped;
  4. check every case finished and write summary.md (PSNR/SSIM/LPIPS + paired Ξ” vs pusht_v1 / rope_v1 epoch 019);
  5. upload to this repo:
    • checkpoints/<run_name>/: epoch_019.pt, config.yaml, action_normalizer.json, train.log, metrics.jsonl, …
    • results/<run_name>/test_epoch_019/: every test case (metrics json, predicted latents, GT|PRED PNG strips, short_gt_pred.mp4, long_gt_pred.mp4, ~5 GB per experiment) + summary.md.

Checklist

# task trained 20 epochs (epoch_019.pt) inference on own test set uploaded checkpoints/ + results/
1a sensor-only pushT ☐ ☐ infer_pusht.sh (30 cases) ☐
1b sensor-only no-force pushT ☐ ☐ infer_pusht.sh (30 cases) ☐
2a sensor-only rope ☐ ☐ infer_rope.sh (34 cases) ☐
2b sensor-only no-force rope ☐ ☐ infer_rope.sh (34 cases) ☐

What the configs change

  • sensor-only = the two per-hand view-action streams are not modelled (not tokenised; view_action2llm9 frozen). The dataset still reads view_actions_fps15.pt, so the window list and the action normaliser are identical to the parent run's. Trainable params 45.30 M (vs 45.34 M); 1,310 tokens/sample (vs 1,342).
  • with force = the sensor action is sensor_actions_with_dforce_fps15.pt: dp = R_kα΅€Ξ”p + k_fΒ·Ξ”fΒ·n, so the change of normal force enters as a displacement along the gel normal.
  • no force = sensor_actions_fps15_noforce.pt: the same file with the k_fΒ·Ξ”fΒ·n term removed (pure geometry); it differs from the force file only in dp_y.
  • Everything else = parent recipe: lr 5e-5 (100 warm-up steps), wd 0.01, bf16, LoRA r16/Ξ±32 on the GEN tower, Cosmos3-Nano init, seed 0.

1116 TODO: validation set (pushT / rope) at epochs 14 and 15

Added 2026-11-16. Run the val split of pushT and rope on epoch 14 and epoch 15 (epoch_014.pt, epoch_015.pt) for the parent runs and for the four TODO runs, so the sensor-only / no-force ablation can be compared on held-out val episodes at matched epochs. This does not replace the epoch-19 test inference above; it is an extra check.

The validation sets

Each task's val split is the val list of its own split file (the same file the run trains with; val β‰  test, whole source episodes held out). The latents, actions and raw media of these episodes are in this repo under the same data/ trees as train/test, so nothing extra has to be prepared.

task split file val episodes (segments) val windows (stride 1, valid actions) val cases file
pushT splits/mixed_pusht_wrist_holdout.json 0910_ep002, 0911_ep003 (seg00, seg01), 0912_ep000 β†’ 4 segments 4,439 14 (all with a 15-window long rollout) eval/cases/pusht_val_cases.json
rope splits/mixed_rope_wrist_holdout.json 0911_ep001 (2 segs), 0914_ep003 (4 segs), 0915_ep002 (2 segs) β†’ 8 segments 6,174 30 (all with a long rollout) eval/cases/rope_val_cases.json

Val cases use the test-case protocol: one case per (val segment Γ— contact class none/onset/release/full, from max(F_L, F_R) > 0.5 N on the initial 16-frame window) where such a window exists, the window drawn with seed 0 among those with room for the 15-window rollout. Built by scripts/build_val_cases.py, which also checks that the parent config and both TODO configs of the task give the identical val window index, so the same case file serves all three runs. (pushT val has no none window in 0911_ep003; rope 0911_ep001_seg01 and 0914_ep003_seg02 have no none window.)

Download (only the val episodes of one task, + checkpoints incl. the parent epoch_014.pt / epoch_015.pt):

$PYTHON_BIN scripts/download.py --what data raw ckpts nano wan_vae --tasks pusht --splits val
$PYTHON_BIN scripts/download.py --what data raw ckpts nano wan_vae --tasks rope  --splits val

Load in Python (the dataset must be built exactly like this, otherwise the case index points at a different window):

import json
from omegaconf import OmegaConf
from vtm_cosmos.data.dataset import VtmWindowDataset
from vtm_cosmos.data.splits import load_split

cfg = OmegaConf.load("configs/nano_action_causal_rope_sensoronly_v1.yaml")       # any pushT/rope config; needs `source env.sh`
episodes = load_split(cfg.data.split_file)["val"]                              # ['rope_0911_episode_001_seg00.pt', ...]
ds = VtmWindowDataset(cfg.data.latent_root, cfg.data.action_root, episodes,
                      action_file=cfg.data.action_file, window_stride=1,
                      include_view_actions=True, view_action_file=cfg.data.view_action_file,
                      require_valid_actions=True,                               # action_full drops invalid transitions
                      active_streams=tuple(cfg.model.active_streams) if cfg.model.get("active_streams") else None)
print(len(ds))                                                                  # rope 6174, pushT 4439
item = ds[0]   # latents.{video,wrist_left,wrist_right,tactile_left,tactile_right}: (48,4,16,16) = 16 frames @ 15 fps
               # actions.action_{left,right}: (15,9); actions.view_action_{left,right}: (16,9); action_valid.*; stream_present
cases = json.load(open("eval/cases/rope_val_cases.json"))["cases"]
c = cases[0]; assert ds.window_meta(c["index"]) == (c["episode"], c["t_start"])

Raw media of a val episode: data/raw/videos/<episode>/{view_middle,tactile_left,tactile_right,wrist_left,wrist_right}.mp4, data/raw/parquet/<episode>.parquet. During training the val split is only used for the per-epoch flow loss (data.val_window_stride: 50).

What to run (4Γ— H100 per command, resumable, uploads to this repo)

scripts/infer_val.sh <config> = for epoch 14 and 15: all val cases of that task on that epoch's checkpoint (todo_eval_upload.py --split val --epoch E) β†’ runs/<run_name>/val_epoch_01{4,5}/summary.md (paired Ξ” vs the parent run at the same epoch, if the parent has already been run) β†’ upload results/<run_name>/val_epoch_01{4,5}/ and, for the TODO runs, checkpoints/<run_name>/epoch_01{4,5}.pt. It can start as soon as runs/<run_name>/epoch_014.pt exists (training can keep running on another node; epoch 15 needs epoch_015.pt). Run the parent first so the TODO summaries get the paired column.

source env.sh && export HF_TOKEN=hf_...     # write access
# pushT (14 val cases; ~30 min per epoch on 4 GPUs)
sbatch -N1 --ntasks=1 --gres=gpu:4 scripts/infer_val.sh configs/nano_action_causal_pusht_v1.yaml                   # parent, ckpts shipped
sbatch -N1 --ntasks=1 --gres=gpu:4 scripts/infer_val.sh configs/nano_action_causal_pusht_sensoronly_v1.yaml        # 1a
sbatch -N1 --ntasks=1 --gres=gpu:4 scripts/infer_val.sh configs/nano_action_causal_pusht_sensoronly_noforce_v1.yaml  # 1b
# rope (30 val cases; ~1 h per epoch on 4 GPUs)
sbatch -N1 --ntasks=1 --gres=gpu:4 scripts/infer_val.sh configs/nano_action_causal_rope_v1.yaml                    # parent, ckpts shipped
sbatch -N1 --ntasks=1 --gres=gpu:4 scripts/infer_val.sh configs/nano_action_causal_rope_sensoronly_v1.yaml         # 2a
sbatch -N1 --ntasks=1 --gres=gpu:4 scripts/infer_val.sh configs/nano_action_causal_rope_sensoronly_noforce_v1.yaml   # 2b
# one epoch only / no upload:  EPOCHS=14 bash scripts/infer_val.sh <config> --no-upload
# re-summarise later (e.g. after the parent finished):
$PYTHON_BIN eval/summarize.py --task rope --split val --root runs/nano_action_causal_rope_sensoronly_v1/val_epoch_014 \
    --ref-root runs/nano_action_causal_rope_v1/val_epoch_014

Single case by hand: $PYTHON_BIN eval/run_test_case.py --task pusht --split val --case 3 --ckpt <epoch_014.pt> (output runs/<run dir>/val_epoch_014/03_.../).

1116 checklist

run task epoch 14 val epoch 15 val uploaded results/<run>/val_epoch_01{4,5}/
nano_action_causal_pusht_v1 (parent) pushT ☐ 14 cases ☐ 14 cases ☐
1a pusht_sensoronly_v1 pushT ☐ ☐ ☐ (+ ckpts)
1b pusht_sensoronly_noforce_v1 pushT ☐ ☐ ☐ (+ ckpts)
nano_action_causal_rope_v1 (parent) rope ☐ 30 cases ☐ 30 cases ☐
2a rope_sensoronly_v1 rope ☐ ☐ ☐ (+ ckpts)
2b rope_sensoronly_noforce_v1 rope ☐ ☐ ☐ (+ ckpts)

1. Setup on a new machine

Tested on Linux, NVIDIA H100 80 GB, a driver that supports CUDA 12.8, and conda.

export HF_TOKEN=hf_...             # this repo is private
# code only (skip the ~150 GB of data/ and checkpoints/ for now)
hf download WitneyWW/acts-sensoronly-ablation --exclude "data/*" "checkpoints/*" --local-dir acts-sensoronly-ablation
cd acts-sensoronly-ablation
bash scripts/setup_env.sh          # conda env `acts_wm` (override ENV_NAME=..., CONDA_ROOT=...): torch 2.10.0+cu128,
                                   # pinned deps, and NVIDIA/cosmos-framework @ 3f94d25 cloned into third_party/
source env.sh                      # paths + PYTHONPATH; PYTHON_BIN defaults to $CONDA_ROOT/envs/acts_wm/bin/python
$PYTHON_BIN scripts/download.py --what all                       # everything (~175 GB incl. Cosmos3-Nano 29 GB)
# or only what you need, e.g. evaluating pushT on its test split:
$PYTHON_BIN scripts/download.py --what data raw ckpts nano wan_vae --tasks pusht --splits test

download.py places files in ./data and ./checkpoints, which are the env.sh defaults of ACTS_DATA_ROOT / ACTS_CKPT_ROOT. --tasks ∈ mb (motherboard), pusht, rope. Training needs the train and val splits (the default fetches all three). If the env is not at ~/miniconda3/envs/acts_wm, export PYTHON_BIN=/path/to/env/bin/python before source env.sh.

variable (env.sh) default content
ACTS_DATA_ROOT ./data mixed_*_latents, mixed_*_actions, raw/
ACTS_CKPT_ROOT ./checkpoints Cosmos3-Nano/, Wan2.2_VAE.pth, vae/lora_last.pt, nano_action_causal_*/epoch_XXX.pt
ACTS_RUNS_ROOT ./runs new training runs and eval outputs
VTM_SOURCE_VIDEO_ROOT $ACTS_DATA_ROOT/raw/videos mp4s used to rebuild the VAE anchor frame at decode time

2. What is in this repo

configs/        exact configs of the 3 runs above + umi_sensoronly_v1 (force arm of 884599) + the 4 TODO configs
splits/         train/val/test episode lists (mixed_{mb,pusht,rope}_wrist_holdout.json)
vtm_cosmos/     model / data / train / infer package (Cosmos3 Generator adapter, causal mask, dataset, rollout)
eval/           run_test_case.py, summarize.py, cases/<task>_test_cases.json (+ reference metrics),
                cases/{pusht,rope}_val_cases.json (1116 TODO)
scripts/        setup_env.sh, download.py, train.sh, infer_pusht.sh / infer_rope.sh (TODO: per-task test inference + upload,
                via todo_eval_upload.py), infer_val.sh (1116 TODO: val at epochs 14/15), build_val_cases.py,
                select_contact_windows.py
results/        (filled by the TODO runs) <run_name>/test_epoch_019/: per-case test outputs + summary.md;
                <run_name>/val_epoch_01{4,5}/ (1116 TODO)
preprocessing/  raw mp4 + parquet -> latents + actions (see section 5)
third_party/wan22_vae/   standalone Wan2.2 VAE + tactile LoRA + encoder + pad_latent.py
data/
  mixed_{mb,pushT,rope}_latents/{tac,visual,wrist}/<episode>.pt    Wan2.2 VAE latents, (N_windows,48,4,16,16) fp16 per stream
  mixed_{mb,pushT,rope}_actions/<episode>/
      sensor_actions_with_dforce_fps15.pt   per-hand 9-D sensor action, force as k_fΒ·Ξ”fΒ·n   (pusht_v1, rope_v1, TODO 1a/2a)
      sensor_actions_fps15_noforce.pt       same without the force term                    (884599, TODO 1b/2b)
      view_actions_fps15.pt                 per-hand view action (read by every run; modelled only by pusht_v1 / rope_v1)
  raw/videos/<episode>/{view_middle,tactile_left,tactile_right,wrist_left,wrist_right}.mp4   30 fps source
  raw/parquet/<episode>.parquet, raw/episodes.json                                           poses + normal force
checkpoints/
  vae/lora_last.pt                          tactile (GelSight) LoRA of the Wan2.2 VAE
  nano_action_causal_umi_sensoronly_noforce_v1/epoch_013.pt   (+ config.yaml, action_normalizer.json, train.log, metrics.jsonl)
  nano_action_causal_pusht_v1/epoch_{014,015,019}.pt          (+ same files)
  nano_action_causal_rope_v1/epoch_{014,015,019}.pt           (+ same files)

Data = exactly the episodes of the three split files (motherboard 21/2/5, pushT 56/4/8, rope 53/8/9 train/val/test = 166 episodes; latents + actions 117 GB, raw 16 GB). Training windows (stride 1, after dropping windows with an invalid action transition): motherboard 71,386, pushT 55,199, rope 77,188.

3. Inference / evaluation

Test protocol: per task, a fixed set of contact-stratified cases (none / onset / release / full, from the GelSight normal force, max(F_L, F_R) > 0.5 N). Each case = one short prediction (8 observed β†’ 8 predicted frames) plus one autoregressive rollout of 15 windows (120 frames) from the same start, conditioned on the recorded actions; seed 0, UniPC 35 steps per latent block.

source env.sh
$PYTHON_BIN eval/run_test_case.py --task rope --case 0                    # one case (~7 min on H100; ~40 s with --short-only)
$PYTHON_BIN eval/run_test_case.py --task rope --case all                  # 34 cases, sequential, resumable
$PYTHON_BIN eval/summarize.py --task rope --reference                     # table + paired Ξ” vs the shipped reference
$PYTHON_BIN eval/run_test_case.py --task umi_sensoronly_noforce --case all    # job 884599's checkpoint, 20 motherboard cases
--task cases default checkpoint
pusht 30 (28 with a long rollout) nano_action_causal_pusht_v1/epoch_019.pt
rope 34 nano_action_causal_rope_v1/epoch_019.pt
umi_sensoronly_noforce 20 nano_action_causal_umi_sensoronly_noforce_v1/epoch_013.pt

Outputs: $ACTS_RUNS_ROOT/<ckpt run dir>/test_<ckpt stem>/<NN_episode_class>/ with window_eval.json (short metrics), reference_rollout.json (per-window long metrics), *_pred.pt, GT/PRED strip PNGs, short_gt_pred.mp4, long_gt_pred.mp4. Cases are independent, so on a cluster run them as an array job, e.g. sbatch --array=0-33 --gres=gpu:1 --wrap "source env.sh; \$PYTHON_BIN eval/run_test_case.py --task rope --case \$SLURM_ARRAY_TASK_ID".

Lower-level CLI for arbitrary windows (the checkpoint stores the original absolute paths, hence the overrides):

$PYTHON_BIN -m vtm_cosmos.infer.rollout --ckpt $ACTS_CKPT_ROOT/nano_action_causal_rope_v1/epoch_019.pt \
  --split test --window-stride 1 --windows 0 500 --rollout-windows 15 --steps 35 --solver unipc --seed 0 \
  --out-dir runs/my_eval --override data.latent_root=$ACTS_DATA_ROOT/mixed_rope_latents \
  data.action_root=$ACTS_DATA_ROOT/mixed_rope_actions data.split_file=splits/mixed_rope_wrist_holdout.json \
  model.nano_checkpoint=$ACTS_CKPT_ROOT/Cosmos3-Nano

Reference results (test split, PSNR dB)

run n horizon video wrist_left wrist_right tactile_left tactile_right
rope_v1 e019 34 short 28.92 22.30 23.70 46.08 46.51
rope_v1 e019 34 long 23.68 16.40 18.22 38.26 40.09
pusht_v1 e019 30 short 29.68 22.86 22.76 46.43 48.37
pusht_v1 e019 28 long 23.28 14.77 15.50 39.21 39.65
umi_sensoronly_noforce e013 20 short 27.54 17.56 20.49 41.56 45.94
umi_sensoronly_noforce e013 20 long 21.02 13.53 14.35 34.53 35.51

SSIM / LPIPS and per-case numbers: eval/cases/<task>_reference_metrics.json.

4. Training

source env.sh
bash scripts/train.sh configs/nano_action_causal_rope_v1.yaml                    # all local GPUs (torchrun)
NGPU=1 bash scripts/train.sh configs/nano_action_causal_rope_sensoronly_v1.yaml \
     run_name=smoke training.max_steps=20                                         # quick smoke test (~31 GiB)
sbatch -N1 --ntasks-per-node=4 --gres=gpu:4 scripts/train.sh configs/<config>.yaml  # Slurm

Any key=value after the config is an OmegaConf dot-list override. The run directory gets config.yaml, train.log, metrics.jsonl, action_normalizer.json (quantile normaliser fitted on the train split), epoch_XXX.pt, last.pt, best.pt (lowest val flow loss; only a monitor: report the last or a matched epoch). Re-launching with the same run_name resumes from last.pt.

5. Preprocessing (raw β†’ latents + actions)

Only needed to rebuild or extend the data; data/ already holds its output.

bash preprocessing/run_preprocess.sh rope                       # -> $ACTS_DATA_ROOT/rebuilt/mixed_rope_{latents,actions}

(run_preprocess.sh task names: umi_all = motherboard, rope, pusht.) Latents: 30 β†’ 15 fps, 256Β² crop, sliding 16-frame windows, Wan2.2 VAE (tactile with the GelSight LoRA), anchor latent dropped. Sensor actions: build_force_actions.py --force-term delta (β†’ ..._with_dforce_fps15.pt). View actions: build_view_actions.py --stride 2.

Notes and pitfalls

  • Always source env.sh first. It sets PYTHONNOUSERSITE=1 and puts third_party/cosmos-framework + third_party/wan22_vae on PYTHONPATH.
  • setup_env.sh pins torch 2.10.0+cu128 and removes nvidia-*-cu13 libraries that some PyPI deps pull in, because they break cuDNN convolutions (CUDNN_STATUS_NOT_INITIALIZED). If you install extra packages, re-run its last block.
  • imageio-ffmpeg is pinned to 0.6.0 (ffmpeg 7.0.2). Bit-exact decoding of the anchor frame depends on the ffmpeg build.
  • The first training / inference step compiles the FlexAttention Triton kernels, which takes a few minutes.
  • Code relative to the 11-08 release WitneyWW/acts-world-model: streams.py, model/layout.py and model/generator_adapter.py allow action_full runs without the view-action pair (sensor-only), and the eval scripts take any checkpoint.

Checks done before upload: REPRO_CHECKS.md.