augment/vace/SETUP.md

# First hour on a new machine

Do these in order. Everything else — how the pipeline works, why the sources are picked the
way they are, runtime numbers — is in `README.md`; this file is only the setup path.

Assumes you have this repo and nothing else.

---

## 0. What you must bring yourself

| | size | why it is not here |
|---|---|---|
| **WAN weights** — `Wan2.1-VACE-1.3B`, `VACE-Annotators` | 31 GB | too large to ship; get them from the VACE release or copy from the original machine |
| **GPU** with ≥ 27 GB free | | `--offload_model False` keeps the DiT resident |

Everything else is in this repo.

---

## 1. Unpack the data (~3 GB, 2 min)

```bash
cd <repo>/                       # the directory holding aug32/, VACE/, datasets_pack/
mkdir -p data && cd data
tar -xf  ../datasets_pack/source_episodes.tar       # 2.7 GB -> pickplace_target_human/
tar -xzf ../datasets_pack/aug_materials_ext.tar.gz  # 121 MB -> aug_materials_ext/
tar -xzf ../datasets_pack/target_meshes.tar.gz      #  14 MB -> objaverse/ lightwheel/
cd ..
```

Check:

```
data/pickplace_target_human/<5 tasks>/{videos,extras}
data/aug_materials_ext/<4 tasks>/ep<N>/mask_gt_target_object_<camera>.mp4
data/{objaverse,lightwheel}/<category>/<model>/model.xml
```

---

## 2. Link the WAN weights

```bash
ln -s /path/to/VACE-home-models VACE/models
ls VACE/models        # must show Wan2.1-VACE-1.3B and VACE-Annotators
```

---

## 3. Build two Python environments

**A — batch driver.** Needs `robocasa`, `robosuite`, `mujoco`, `trimesh`, `pyrender`,
`opencv-python`, `numpy`. This one also renders the target meshes' reference views, so
`trimesh` + `pyrender` are not optional. Versions that work: mujoco 3.3.1, robosuite 1.5.2,
robocasa 1.0.0, trimesh 4.12.1, pyrender 0.1.45, numpy 1.26.4, Python 3.10.

**B — WAN inference.** Python 3.12, `torch 2.5.1+cu124`, `flash_attn 2.8.3`, plus
`VACE/requirements.txt`. `wan` must be importable (it is a site-package in the original
environment, installed from the VACE repo).

```bash
export VACE_PY_ROBOCASA=/path/to/env-A/bin/python
export VACE_PY_WAN=/path/to/env-B/bin/python
```

---

## 4. Export the data paths

```bash
cd <repo>
export VACE_ROBOCASA_ROOT=$PWD/data/pickplace_target_human
export VACE_AUG_MATERIALS=$PWD/data/aug_materials_ext
export VACE_ASSET_ROOTS=$PWD/data/lightwheel:$PWD/data/objaverse
```

Leave `VACE_RENDER_SCRIPT_DIR` and `VACE_RENDER_PYTHON` alone — they default to
`aug32/vendor/` and to `VACE_PY_ROBOCASA`, both already correct.

Put all five exports in a file you can `source`; every later step needs them.

---

## 5. Preflight

```bash
./aug32/preflight.sh
```

Every line must read `OK`. It checks both interpreters, the vendored renderer and its
imports, the VACE fork and weights, the three data roots, per-task GT-mask coverage, and
free GPU memory. A `MISS` names the exact path that is wrong.

---

## 6. Smoke test before the real batch

One assignment end to end, ~10 min on one GPU, to catch a broken path before a multi-hour
run:

```bash
mkdir -p /tmp/vace_smoke
$VACE_PY_ROBOCASA aug32/build_aug32_assignments.py \
    --out-root /tmp/vace_smoke --per-mesh 1 --targets teapot/teapot_6
head -1 /tmp/vace_smoke/assignments.jsonl > /tmp/vace_smoke/one.jsonl
mv /tmp/vace_smoke/one.jsonl /tmp/vace_smoke/assignments.jsonl
./aug32/run_aug32.sh /tmp/vace_smoke /tmp/vace_smoke_out 0
```

Expect `/tmp/vace_smoke_out/<task>-ep<N>_<src>_teapot_6/` with three camera folders, each
holding one mp4.

---

## 7. The real run

```bash
./aug32/run_aug32.sh /scratch/vace_aug32 /scratch/output_aug_256 0
```

8 targets × 32 episodes × 3 cameras ≈ 3,084 WAN jobs, **~44 h on one GPU** at the measured
51.4 s/job. Split it: `run_aug32.sh ... <gpu> 4` writes four worker manifests, and the four
WAN processes can be moved onto four GPUs (~11 h). Every stage is idempotent, so an
interrupted run resumes where it stopped.

---

## Common failures

| symptom | cause |
|---|---|
| `FileNotFoundError: Missing GT target-object mask video` | `VACE_AUG_MATERIALS` not exported, or pointing above `aug_materials_ext/` |
| `ModuleNotFoundError: No module named 'trimesh'` during `prep-worker` | env A lacks trimesh/pyrender — install them, or point `VACE_RENDER_PYTHON` at an env that has them |
| `unrecognized arguments: --batch_manifest` | `VACE/` was replaced with a fresh upstream clone; apply `aug32/vace_wan_inference_batch.patch` |
| WAN job ~82 s instead of ~51 s | `--offload_model False` got dropped; it is in `run_aug32.sh`, do not remove it |
| exo cameras come out with the object *erased* rather than swapped | expected at 256×256 — the target mask is ~0.2 % of an exo frame. Only `robot0_eye_in_hand` (~9 %) reliably swaps |
| assignments have fewer same-category sources than expected | the dataset only holds 5 teapot / 3 donut / 2 jar / 2 syrup_bottle episodes. Only MeasuringCup009 can fill 32 from its own category |