Instructions to use WitneyWW/acts-sensoronly-ablation with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Cosmos
How to use WitneyWW/acts-sensoronly-ablation with Cosmos:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
|
Download README.md from WitneyWW/acts-sensoronly-ablation: direct link, hf CLI and curl.
- Browser
- Download file 25.3 kB
-
https://huggingface.co/WitneyWW/acts-sensoronly-ablation/resolve/main/README.md
- Command line
-
hf download hf://WitneyWW/acts-sensoronly-ablation/README.md
-
curl -L -o README.md https://huggingface.co/WitneyWW/acts-sensoronly-ablation/resolve/main/README.md
25.3 kB
| license: other | |
| tags: | |
| - world-model | |
| - robotics | |
| - tactile | |
| - cosmos | |
| # ACTS world model: sensor-only / no-force ablation (code + data + checkpoints) | |
| This repo contains everything needed to run three action-conditioned visuo-tactile world-model runs on a new machine, and to train | |
| the four follow-up experiments listed under **TODO**. **1116 TODO** (below): val-set inference of pushT / rope at epochs 14 and 15. It holds the code, the preprocessed data, the raw media and the trained | |
| checkpoints. Only the two public base models are downloaded from their own repos. | |
| | run | task | what it is | ckpt in this repo | status | | |
| |---|---|---|---|---| | |
| | `nano_action_causal_umi_sensoronly_noforce_v1` | motherboard | **Slurm job 884599** (`rebuttal_2`): sensor-only (no view-action streams) + no force | `checkpoints/β¦/epoch_013.pt` | training was still running at upload (epoch 14/20) | | |
| | `nano_action_causal_pusht_v1` | pushT | full model (sensor + view actions, with force) | `checkpoints/β¦/epoch_019.pt` (+ `epoch_014/015.pt` for the 1116 val TODO) | done | | |
| | `nano_action_causal_rope_v1` | rope | full model (sensor + view actions, with force) | `checkpoints/β¦/epoch_019.pt` (+ `epoch_014/015.pt` for the 1116 val TODO) | done | | |
| ## TODO experiments | |
| The same ablation that was run on motherboard (sensor-only, with and without force), now on **pushT** and **rope**. | |
| **pushT and rope are two separate tasks**: each is trained only on its own data, and each run is evaluated with **its own | |
| inference script on its own test set**. Nothing is shared between the two tasks except the code. | |
| | | **pushT** | **rope** | | |
| |---|---|---| | |
| | experiments | 1a `nano_action_causal_pusht_sensoronly_v1` (with force)<br>1b `nano_action_causal_pusht_sensoronly_noforce_v1` (no force) | 2a `nano_action_causal_rope_sensoronly_v1` (with force)<br>2b `nano_action_causal_rope_sensoronly_noforce_v1` (no force) | | |
| | configs | `configs/nano_action_causal_pusht_sensoronly{,_noforce}_v1.yaml` | `configs/nano_action_causal_rope_sensoronly{,_noforce}_v1.yaml` | | |
| | training data | `data/mixed_pushT_{latents,actions}`, split `splits/mixed_pusht_wrist_holdout.json`: 56 train / 4 val episodes, 55,199 train windows | `data/mixed_rope_{latents,actions}`, split `splits/mixed_rope_wrist_holdout.json`: 53 train / 8 val episodes, 77,188 train windows | | |
| | training time (4Γ H100, 20 epochs) | β 2.3 h/epoch (6,900 steps) β ~2 days per experiment | β 3.3 h/epoch (9,648 steps) β ~3 days per experiment | | |
| | **inference script** | **`scripts/infer_pusht.sh`** | **`scripts/infer_rope.sh`** | | |
| | **test set** | test split of the pushT split file: 8 episodes β `eval/cases/pusht_test_cases.json`, **30 cases** (28 with a long rollout; cases 5 and 29 are short-only, the segment is too short) | test split of the rope split file: 9 episodes β `eval/cases/rope_test_cases.json`, **34 cases** (all with a long rollout) | | |
| | compared against | `nano_action_causal_pusht_v1` epoch 019 (`eval/cases/pusht_reference_metrics.json`) | `nano_action_causal_rope_v1` epoch 019 (`eval/cases/rope_reference_metrics.json`) | | |
| Each config differs from its parent (`nano_action_causal_pusht_v1` / `nano_action_causal_rope_v1`) only in `run_name`, | |
| `model.active_streams` (no `view_action_*`), and for the no-force variants `data.action_file: sensor_actions_fps15_noforce.pt`. | |
| ### Requirements (all 4 experiments) | |
| * **Each experiment is its own training job on 4Γ H100 80 GB (one node)**: one DDP rank per GPU, batch 2 per GPU = global batch 8, | |
| ~31 GiB per GPU. Do not change the GPU count or `training.batch_size`; the parent runs and job 884599 used exactly 4Γ2. | |
| * **Train for the full 20 epochs** (`training.epochs: 20` β `epoch_000.pt` β¦ `epoch_019.pt`). Do not stop early and do not use | |
| `best.pt`. The training is finished only when `runs/<run_name>/epoch_019.pt` exists. | |
| * If a Slurm job hits its time limit or is preempted, submit **the same command again**. It resumes from `runs/<run_name>/last.pt`; | |
| do not change the run name. | |
| * **Only after epoch 20**: run the inference script **of that task** on **that task's test set** (pushT run β `infer_pusht.sh`, | |
| rope run β `infer_rope.sh`). It uploads the checkpoint and the test results to this HF repo, so it needs `HF_TOKEN` with | |
| **write** access to `WitneyWW/acts-sensoronly-ablation`. | |
| ### Step 1: setup and data (once per machine) | |
| ```bash | |
| bash scripts/setup_env.sh && source env.sh && export HF_TOKEN=hf_... # see section 1 | |
| $PYTHON_BIN scripts/download.py --what data raw ckpts nano wan_vae --tasks pusht # pushT: train + val + test data | |
| $PYTHON_BIN scripts/download.py --what data raw ckpts nano wan_vae --tasks rope # rope: train + val + test data | |
| ``` | |
| ### pushT (experiments 1a, 1b) | |
| ```bash | |
| source env.sh | |
| # train: two separate jobs, 4x H100 each, 20 epochs | |
| sbatch -N1 --ntasks-per-node=4 --gres=gpu:4 scripts/train.sh configs/nano_action_causal_pusht_sensoronly_v1.yaml # 1a | |
| sbatch -N1 --ntasks-per-node=4 --gres=gpu:4 scripts/train.sh configs/nano_action_causal_pusht_sensoronly_noforce_v1.yaml # 1b | |
| # (without Slurm, on a 4x H100 machine: NGPU=4 bash scripts/train.sh configs/<config>.yaml) | |
| # after runs/<run_name>/epoch_019.pt exists: pushT inference script on the pushT test set (30 cases) + upload, 4 GPUs, ~1 h | |
| sbatch -N1 --ntasks=1 --gres=gpu:4 scripts/infer_pusht.sh configs/nano_action_causal_pusht_sensoronly_v1.yaml # 1a | |
| sbatch -N1 --ntasks=1 --gres=gpu:4 scripts/infer_pusht.sh configs/nano_action_causal_pusht_sensoronly_noforce_v1.yaml # 1b | |
| ``` | |
| ### rope (experiments 2a, 2b) | |
| ```bash | |
| source env.sh | |
| # train: two separate jobs, 4x H100 each, 20 epochs | |
| sbatch -N1 --ntasks-per-node=4 --gres=gpu:4 scripts/train.sh configs/nano_action_causal_rope_sensoronly_v1.yaml # 2a | |
| sbatch -N1 --ntasks-per-node=4 --gres=gpu:4 scripts/train.sh configs/nano_action_causal_rope_sensoronly_noforce_v1.yaml # 2b | |
| # after runs/<run_name>/epoch_019.pt exists: rope inference script on the rope test set (34 cases) + upload, 4 GPUs, ~1 h | |
| sbatch -N1 --ntasks=1 --gres=gpu:4 scripts/infer_rope.sh configs/nano_action_causal_rope_sensoronly_v1.yaml # 2a | |
| sbatch -N1 --ntasks=1 --gres=gpu:4 scripts/infer_rope.sh configs/nano_action_causal_rope_sensoronly_noforce_v1.yaml # 2b | |
| ``` | |
| Sanity checks at the start of training (`runs/<run_name>/train.log`): `[data] train episodes=56 windows=55199` (pushT) or | |
| `train episodes=53 windows=77188` (rope), and `trainable=45,301,952`. Progress: `ls runs/<run_name>/epoch_*.pt`. | |
| ### What the inference scripts do | |
| `scripts/infer_pusht.sh` / `scripts/infer_rope.sh` (both call `scripts/todo_eval_upload.py --task <task>`): | |
| 1. refuse a config of the other task, and refuse to run unless `runs/<run_name>/epoch_019.pt` exists (20 epochs finished); | |
| 2. check that the test cases come from the test split of the same split file the model was trained with; | |
| 3. run `eval/run_test_case.py` with `epoch_019.pt` on **all test cases of that task** (each = short 8β8 prediction + 15-window long | |
| rollout, UniPC 35 steps, seed 0), spread over the 4 GPUs. If interrupted, run the same command again: finished cases are skipped; | |
| 4. check every case finished and write `summary.md` (PSNR/SSIM/LPIPS + paired Ξ vs `pusht_v1` / `rope_v1` epoch 019); | |
| 5. upload to this repo: | |
| * `checkpoints/<run_name>/`: `epoch_019.pt`, `config.yaml`, `action_normalizer.json`, `train.log`, `metrics.jsonl`, β¦ | |
| * `results/<run_name>/test_epoch_019/`: every test case (metrics json, predicted latents, GT|PRED PNG strips, `short_gt_pred.mp4`, | |
| `long_gt_pred.mp4`, ~5 GB per experiment) + `summary.md`. | |
| ### Checklist | |
| | # | task | trained 20 epochs (`epoch_019.pt`) | inference on own test set | uploaded `checkpoints/` + `results/` | | |
| |---|---|---|---|---| | |
| | 1a sensor-only | pushT | β | β `infer_pusht.sh` (30 cases) | β | | |
| | 1b sensor-only no-force | pushT | β | β `infer_pusht.sh` (30 cases) | β | | |
| | 2a sensor-only | rope | β | β `infer_rope.sh` (34 cases) | β | | |
| | 2b sensor-only no-force | rope | β | β `infer_rope.sh` (34 cases) | β | | |
| ### What the configs change | |
| * **sensor-only** = the two per-hand view-action streams are not modelled (not tokenised; `view_action2llm9` frozen). The dataset | |
| still reads `view_actions_fps15.pt`, so the window list and the action normaliser are identical to the parent run's. Trainable | |
| params 45.30 M (vs 45.34 M); 1,310 tokens/sample (vs 1,342). | |
| * **with force** = the sensor action is `sensor_actions_with_dforce_fps15.pt`: `dp = R_kα΅Ξp + k_fΒ·ΞfΒ·n`, so the change of normal | |
| force enters as a displacement along the gel normal. | |
| * **no force** = `sensor_actions_fps15_noforce.pt`: the same file with the `k_fΒ·ΞfΒ·n` term removed (pure geometry); it differs from | |
| the force file only in `dp_y`. | |
| * Everything else = parent recipe: lr 5e-5 (100 warm-up steps), wd 0.01, bf16, LoRA r16/Ξ±32 on the GEN tower, Cosmos3-Nano init, seed 0. | |
| ## 1116 TODO: validation set (pushT / rope) at epochs 14 and 15 | |
| Added 2026-11-16. Run the **val** split of pushT and rope on **epoch 14 and epoch 15** (`epoch_014.pt`, `epoch_015.pt`) for | |
| the parent runs and for the four TODO runs, so the sensor-only / no-force ablation can be compared on held-out val episodes | |
| at matched epochs. This does not replace the epoch-19 test inference above; it is an extra check. | |
| ### The validation sets | |
| Each task's val split is the `val` list of its own split file (the same file the run trains with; val β test, whole source | |
| episodes held out). The latents, actions and raw media of these episodes are in this repo under the same `data/` trees as | |
| train/test, so nothing extra has to be prepared. | |
| | task | split file | val episodes (segments) | val windows (stride 1, valid actions) | val cases | file | | |
| |---|---|---|---:|---:|---| | |
| | pushT | `splits/mixed_pusht_wrist_holdout.json` | 0910_ep002, 0911_ep003 (seg00, seg01), 0912_ep000 β 4 segments | 4,439 | **14** (all with a 15-window long rollout) | `eval/cases/pusht_val_cases.json` | | |
| | rope | `splits/mixed_rope_wrist_holdout.json` | 0911_ep001 (2 segs), 0914_ep003 (4 segs), 0915_ep002 (2 segs) β 8 segments | 6,174 | **30** (all with a long rollout) | `eval/cases/rope_val_cases.json` | | |
| Val cases use the test-case protocol: one case per (val segment Γ contact class `none`/`onset`/`release`/`full`, from | |
| `max(F_L, F_R) > 0.5 N` on the initial 16-frame window) where such a window exists, the window drawn with seed 0 among those | |
| with room for the 15-window rollout. Built by `scripts/build_val_cases.py`, which also checks that the parent config and both | |
| TODO configs of the task give the **identical** val window index, so the same case file serves all three runs. (pushT val | |
| has no `none` window in 0911_ep003; rope 0911_ep001_seg01 and 0914_ep003_seg02 have no `none` window.) | |
| **Download** (only the val episodes of one task, + checkpoints incl. the parent `epoch_014.pt` / `epoch_015.pt`): | |
| ```bash | |
| $PYTHON_BIN scripts/download.py --what data raw ckpts nano wan_vae --tasks pusht --splits val | |
| $PYTHON_BIN scripts/download.py --what data raw ckpts nano wan_vae --tasks rope --splits val | |
| ``` | |
| **Load in Python** (the dataset must be built exactly like this, otherwise the case `index` points at a different window): | |
| ```python | |
| import json | |
| from omegaconf import OmegaConf | |
| from vtm_cosmos.data.dataset import VtmWindowDataset | |
| from vtm_cosmos.data.splits import load_split | |
| cfg = OmegaConf.load("configs/nano_action_causal_rope_sensoronly_v1.yaml") # any pushT/rope config; needs `source env.sh` | |
| episodes = load_split(cfg.data.split_file)["val"] # ['rope_0911_episode_001_seg00.pt', ...] | |
| ds = VtmWindowDataset(cfg.data.latent_root, cfg.data.action_root, episodes, | |
| action_file=cfg.data.action_file, window_stride=1, | |
| include_view_actions=True, view_action_file=cfg.data.view_action_file, | |
| require_valid_actions=True, # action_full drops invalid transitions | |
| active_streams=tuple(cfg.model.active_streams) if cfg.model.get("active_streams") else None) | |
| print(len(ds)) # rope 6174, pushT 4439 | |
| item = ds[0] # latents.{video,wrist_left,wrist_right,tactile_left,tactile_right}: (48,4,16,16) = 16 frames @ 15 fps | |
| # actions.action_{left,right}: (15,9); actions.view_action_{left,right}: (16,9); action_valid.*; stream_present | |
| cases = json.load(open("eval/cases/rope_val_cases.json"))["cases"] | |
| c = cases[0]; assert ds.window_meta(c["index"]) == (c["episode"], c["t_start"]) | |
| ``` | |
| Raw media of a val episode: `data/raw/videos/<episode>/{view_middle,tactile_left,tactile_right,wrist_left,wrist_right}.mp4`, | |
| `data/raw/parquet/<episode>.parquet`. During training the val split is only used for the per-epoch flow loss | |
| (`data.val_window_stride: 50`). | |
| ### What to run (4Γ H100 per command, resumable, uploads to this repo) | |
| `scripts/infer_val.sh <config>` = for epoch 14 and 15: all val cases of that task on that epoch's checkpoint | |
| (`todo_eval_upload.py --split val --epoch E`) β `runs/<run_name>/val_epoch_01{4,5}/summary.md` (paired Ξ vs the parent run | |
| at the same epoch, if the parent has already been run) β upload `results/<run_name>/val_epoch_01{4,5}/` and, for the TODO runs, | |
| `checkpoints/<run_name>/epoch_01{4,5}.pt`. It can start as soon as `runs/<run_name>/epoch_014.pt` exists (training can keep | |
| running on another node; epoch 15 needs `epoch_015.pt`). **Run the parent first** so the TODO summaries get the paired column. | |
| ```bash | |
| source env.sh && export HF_TOKEN=hf_... # write access | |
| # pushT (14 val cases; ~30 min per epoch on 4 GPUs) | |
| sbatch -N1 --ntasks=1 --gres=gpu:4 scripts/infer_val.sh configs/nano_action_causal_pusht_v1.yaml # parent, ckpts shipped | |
| sbatch -N1 --ntasks=1 --gres=gpu:4 scripts/infer_val.sh configs/nano_action_causal_pusht_sensoronly_v1.yaml # 1a | |
| sbatch -N1 --ntasks=1 --gres=gpu:4 scripts/infer_val.sh configs/nano_action_causal_pusht_sensoronly_noforce_v1.yaml # 1b | |
| # rope (30 val cases; ~1 h per epoch on 4 GPUs) | |
| sbatch -N1 --ntasks=1 --gres=gpu:4 scripts/infer_val.sh configs/nano_action_causal_rope_v1.yaml # parent, ckpts shipped | |
| sbatch -N1 --ntasks=1 --gres=gpu:4 scripts/infer_val.sh configs/nano_action_causal_rope_sensoronly_v1.yaml # 2a | |
| sbatch -N1 --ntasks=1 --gres=gpu:4 scripts/infer_val.sh configs/nano_action_causal_rope_sensoronly_noforce_v1.yaml # 2b | |
| # one epoch only / no upload: EPOCHS=14 bash scripts/infer_val.sh <config> --no-upload | |
| # re-summarise later (e.g. after the parent finished): | |
| $PYTHON_BIN eval/summarize.py --task rope --split val --root runs/nano_action_causal_rope_sensoronly_v1/val_epoch_014 \ | |
| --ref-root runs/nano_action_causal_rope_v1/val_epoch_014 | |
| ``` | |
| Single case by hand: `$PYTHON_BIN eval/run_test_case.py --task pusht --split val --case 3 --ckpt <epoch_014.pt>` | |
| (output `runs/<run dir>/val_epoch_014/03_.../`). | |
| ### 1116 checklist | |
| | run | task | epoch 14 val | epoch 15 val | uploaded `results/<run>/val_epoch_01{4,5}/` | | |
| |---|---|---|---|---| | |
| | `nano_action_causal_pusht_v1` (parent) | pushT | β 14 cases | β 14 cases | β | | |
| | 1a `pusht_sensoronly_v1` | pushT | β | β | β (+ ckpts) | | |
| | 1b `pusht_sensoronly_noforce_v1` | pushT | β | β | β (+ ckpts) | | |
| | `nano_action_causal_rope_v1` (parent) | rope | β 30 cases | β 30 cases | β | | |
| | 2a `rope_sensoronly_v1` | rope | β | β | β (+ ckpts) | | |
| | 2b `rope_sensoronly_noforce_v1` | rope | β | β | β (+ ckpts) | | |
| ## 1. Setup on a new machine | |
| Tested on Linux, NVIDIA H100 80 GB, a driver that supports CUDA 12.8, and conda. | |
| ```bash | |
| export HF_TOKEN=hf_... # this repo is private | |
| # code only (skip the ~150 GB of data/ and checkpoints/ for now) | |
| hf download WitneyWW/acts-sensoronly-ablation --exclude "data/*" "checkpoints/*" --local-dir acts-sensoronly-ablation | |
| cd acts-sensoronly-ablation | |
| bash scripts/setup_env.sh # conda env `acts_wm` (override ENV_NAME=..., CONDA_ROOT=...): torch 2.10.0+cu128, | |
| # pinned deps, and NVIDIA/cosmos-framework @ 3f94d25 cloned into third_party/ | |
| source env.sh # paths + PYTHONPATH; PYTHON_BIN defaults to $CONDA_ROOT/envs/acts_wm/bin/python | |
| $PYTHON_BIN scripts/download.py --what all # everything (~175 GB incl. Cosmos3-Nano 29 GB) | |
| # or only what you need, e.g. evaluating pushT on its test split: | |
| $PYTHON_BIN scripts/download.py --what data raw ckpts nano wan_vae --tasks pusht --splits test | |
| ``` | |
| `download.py` places files in `./data` and `./checkpoints`, which are the `env.sh` defaults of `ACTS_DATA_ROOT` / `ACTS_CKPT_ROOT`. | |
| `--tasks` β `mb` (motherboard), `pusht`, `rope`. Training needs the train and val splits (the default fetches all three). | |
| If the env is not at `~/miniconda3/envs/acts_wm`, export `PYTHON_BIN=/path/to/env/bin/python` before `source env.sh`. | |
| | variable (env.sh) | default | content | | |
| |---|---|---| | |
| | `ACTS_DATA_ROOT` | `./data` | `mixed_*_latents`, `mixed_*_actions`, `raw/` | | |
| | `ACTS_CKPT_ROOT` | `./checkpoints` | `Cosmos3-Nano/`, `Wan2.2_VAE.pth`, `vae/lora_last.pt`, `nano_action_causal_*/epoch_XXX.pt` | | |
| | `ACTS_RUNS_ROOT` | `./runs` | new training runs and eval outputs | | |
| | `VTM_SOURCE_VIDEO_ROOT` | `$ACTS_DATA_ROOT/raw/videos` | mp4s used to rebuild the VAE anchor frame at decode time | | |
| ## 2. What is in this repo | |
| ``` | |
| configs/ exact configs of the 3 runs above + umi_sensoronly_v1 (force arm of 884599) + the 4 TODO configs | |
| splits/ train/val/test episode lists (mixed_{mb,pusht,rope}_wrist_holdout.json) | |
| vtm_cosmos/ model / data / train / infer package (Cosmos3 Generator adapter, causal mask, dataset, rollout) | |
| eval/ run_test_case.py, summarize.py, cases/<task>_test_cases.json (+ reference metrics), | |
| cases/{pusht,rope}_val_cases.json (1116 TODO) | |
| scripts/ setup_env.sh, download.py, train.sh, infer_pusht.sh / infer_rope.sh (TODO: per-task test inference + upload, | |
| via todo_eval_upload.py), infer_val.sh (1116 TODO: val at epochs 14/15), build_val_cases.py, | |
| select_contact_windows.py | |
| results/ (filled by the TODO runs) <run_name>/test_epoch_019/: per-case test outputs + summary.md; | |
| <run_name>/val_epoch_01{4,5}/ (1116 TODO) | |
| preprocessing/ raw mp4 + parquet -> latents + actions (see section 5) | |
| third_party/wan22_vae/ standalone Wan2.2 VAE + tactile LoRA + encoder + pad_latent.py | |
| data/ | |
| mixed_{mb,pushT,rope}_latents/{tac,visual,wrist}/<episode>.pt Wan2.2 VAE latents, (N_windows,48,4,16,16) fp16 per stream | |
| mixed_{mb,pushT,rope}_actions/<episode>/ | |
| sensor_actions_with_dforce_fps15.pt per-hand 9-D sensor action, force as k_fΒ·ΞfΒ·n (pusht_v1, rope_v1, TODO 1a/2a) | |
| sensor_actions_fps15_noforce.pt same without the force term (884599, TODO 1b/2b) | |
| view_actions_fps15.pt per-hand view action (read by every run; modelled only by pusht_v1 / rope_v1) | |
| raw/videos/<episode>/{view_middle,tactile_left,tactile_right,wrist_left,wrist_right}.mp4 30 fps source | |
| raw/parquet/<episode>.parquet, raw/episodes.json poses + normal force | |
| checkpoints/ | |
| vae/lora_last.pt tactile (GelSight) LoRA of the Wan2.2 VAE | |
| nano_action_causal_umi_sensoronly_noforce_v1/epoch_013.pt (+ config.yaml, action_normalizer.json, train.log, metrics.jsonl) | |
| nano_action_causal_pusht_v1/epoch_{014,015,019}.pt (+ same files) | |
| nano_action_causal_rope_v1/epoch_{014,015,019}.pt (+ same files) | |
| ``` | |
| Data = exactly the episodes of the three split files (motherboard 21/2/5, pushT 56/4/8, rope 53/8/9 train/val/test = 166 episodes; | |
| latents + actions 117 GB, raw 16 GB). Training windows (stride 1, after dropping windows with an invalid action transition): | |
| motherboard 71,386, pushT 55,199, rope 77,188. | |
| ## 3. Inference / evaluation | |
| Test protocol: per task, a fixed set of contact-stratified cases (`none` / `onset` / `release` / `full`, from the GelSight normal force, | |
| `max(F_L, F_R) > 0.5 N`). Each case = one short prediction (8 observed β 8 predicted frames) plus one autoregressive rollout of 15 | |
| windows (120 frames) from the same start, conditioned on the recorded actions; seed 0, UniPC 35 steps per latent block. | |
| ```bash | |
| source env.sh | |
| $PYTHON_BIN eval/run_test_case.py --task rope --case 0 # one case (~7 min on H100; ~40 s with --short-only) | |
| $PYTHON_BIN eval/run_test_case.py --task rope --case all # 34 cases, sequential, resumable | |
| $PYTHON_BIN eval/summarize.py --task rope --reference # table + paired Ξ vs the shipped reference | |
| $PYTHON_BIN eval/run_test_case.py --task umi_sensoronly_noforce --case all # job 884599's checkpoint, 20 motherboard cases | |
| ``` | |
| | `--task` | cases | default checkpoint | | |
| |---|---:|---| | |
| | `pusht` | 30 (28 with a long rollout) | `nano_action_causal_pusht_v1/epoch_019.pt` | | |
| | `rope` | 34 | `nano_action_causal_rope_v1/epoch_019.pt` | | |
| | `umi_sensoronly_noforce` | 20 | `nano_action_causal_umi_sensoronly_noforce_v1/epoch_013.pt` | | |
| Outputs: `$ACTS_RUNS_ROOT/<ckpt run dir>/test_<ckpt stem>/<NN_episode_class>/` with `window_eval.json` (short metrics), | |
| `reference_rollout.json` (per-window long metrics), `*_pred.pt`, GT/PRED strip PNGs, `short_gt_pred.mp4`, `long_gt_pred.mp4`. | |
| Cases are independent, so on a cluster run them as an array job, e.g. | |
| `sbatch --array=0-33 --gres=gpu:1 --wrap "source env.sh; \$PYTHON_BIN eval/run_test_case.py --task rope --case \$SLURM_ARRAY_TASK_ID"`. | |
| Lower-level CLI for arbitrary windows (the checkpoint stores the original absolute paths, hence the overrides): | |
| ```bash | |
| $PYTHON_BIN -m vtm_cosmos.infer.rollout --ckpt $ACTS_CKPT_ROOT/nano_action_causal_rope_v1/epoch_019.pt \ | |
| --split test --window-stride 1 --windows 0 500 --rollout-windows 15 --steps 35 --solver unipc --seed 0 \ | |
| --out-dir runs/my_eval --override data.latent_root=$ACTS_DATA_ROOT/mixed_rope_latents \ | |
| data.action_root=$ACTS_DATA_ROOT/mixed_rope_actions data.split_file=splits/mixed_rope_wrist_holdout.json \ | |
| model.nano_checkpoint=$ACTS_CKPT_ROOT/Cosmos3-Nano | |
| ``` | |
| ### Reference results (test split, PSNR dB) | |
| | run | n | horizon | video | wrist_left | wrist_right | tactile_left | tactile_right | | |
| |---|---:|---|---:|---:|---:|---:|---:| | |
| | rope_v1 e019 | 34 | short | 28.92 | 22.30 | 23.70 | 46.08 | 46.51 | | |
| | rope_v1 e019 | 34 | long | 23.68 | 16.40 | 18.22 | 38.26 | 40.09 | | |
| | pusht_v1 e019 | 30 | short | 29.68 | 22.86 | 22.76 | 46.43 | 48.37 | | |
| | pusht_v1 e019 | 28 | long | 23.28 | 14.77 | 15.50 | 39.21 | 39.65 | | |
| | umi_sensoronly_noforce e013 | 20 | short | 27.54 | 17.56 | 20.49 | 41.56 | 45.94 | | |
| | umi_sensoronly_noforce e013 | 20 | long | 21.02 | 13.53 | 14.35 | 34.53 | 35.51 | | |
| SSIM / LPIPS and per-case numbers: `eval/cases/<task>_reference_metrics.json`. | |
| ## 4. Training | |
| ```bash | |
| source env.sh | |
| bash scripts/train.sh configs/nano_action_causal_rope_v1.yaml # all local GPUs (torchrun) | |
| NGPU=1 bash scripts/train.sh configs/nano_action_causal_rope_sensoronly_v1.yaml \ | |
| run_name=smoke training.max_steps=20 # quick smoke test (~31 GiB) | |
| sbatch -N1 --ntasks-per-node=4 --gres=gpu:4 scripts/train.sh configs/<config>.yaml # Slurm | |
| ``` | |
| Any `key=value` after the config is an OmegaConf dot-list override. The run directory gets `config.yaml`, `train.log`, `metrics.jsonl`, | |
| `action_normalizer.json` (quantile normaliser fitted on the train split), `epoch_XXX.pt`, `last.pt`, `best.pt` (lowest val flow loss; | |
| only a monitor: report the last or a matched epoch). Re-launching with the same `run_name` resumes from `last.pt`. | |
| ## 5. Preprocessing (raw β latents + actions) | |
| Only needed to rebuild or extend the data; `data/` already holds its output. | |
| ```bash | |
| bash preprocessing/run_preprocess.sh rope # -> $ACTS_DATA_ROOT/rebuilt/mixed_rope_{latents,actions} | |
| ``` | |
| (`run_preprocess.sh` task names: `umi_all` = motherboard, `rope`, `pusht`.) Latents: 30 β 15 fps, 256Β² crop, sliding 16-frame windows, | |
| Wan2.2 VAE (tactile with the GelSight LoRA), anchor latent dropped. Sensor actions: `build_force_actions.py --force-term delta` | |
| (β `..._with_dforce_fps15.pt`). View actions: `build_view_actions.py --stride 2`. | |
| ## Notes and pitfalls | |
| * Always `source env.sh` first. It sets `PYTHONNOUSERSITE=1` and puts `third_party/cosmos-framework` + `third_party/wan22_vae` on `PYTHONPATH`. | |
| * `setup_env.sh` pins torch 2.10.0+cu128 and removes `nvidia-*-cu13` libraries that some PyPI deps pull in, because they break cuDNN | |
| convolutions (`CUDNN_STATUS_NOT_INITIALIZED`). If you install extra packages, re-run its last block. | |
| * `imageio-ffmpeg` is pinned to 0.6.0 (ffmpeg 7.0.2). Bit-exact decoding of the anchor frame depends on the ffmpeg build. | |
| * The first training / inference step compiles the FlexAttention Triton kernels, which takes a few minutes. | |
| * Code relative to the 11-08 release `WitneyWW/acts-world-model`: `streams.py`, `model/layout.py` and `model/generator_adapter.py` | |
| allow `action_full` runs without the view-action pair (sensor-only), and the eval scripts take any checkpoint. | |
| Checks done before upload: `REPRO_CHECKS.md`. | |