# Turning VACE output into a trainable dataset — everything needed After `run_aug32.sh` finishes you have per-episode videos. Training needs a LeRobot v2.1 "gr00t_views" dataset. This is the full inventory for doing that conversion on a machine that has nothing but this repo. Written 2026-09-19. The MimicGen side of the same job is documented in `baseline/mimicgen/TRAINING_DATASET_PREP.md`; **the two are not the same procedure** — see "Why VACE is the easy case" below. --- ## Why VACE is the easy case VACE repaints pixels; it never changes the trajectory. So the augmented episode's state/action rows are **the source episode's own rows, unchanged**. The builder copies that parquet, rewrites indices and task ids, and links the new videos in. No simulator, no re-render, no MuJoCo. MimicGen is the opposite: every episode is a new rollout, so its parquet has to be rebuilt from the generation HDF5 by replaying it in the simulator (~2 h for 256 episodes). | | VACE | MimicGen | |---|---|---| | builder | `build_vace_objwise_gr00t_dataset.py` | `render_to_gr00t.py` | | parquet | copied from the source episode | rebuilt from HDF5 | | video | symlinked from the VACE output | re-rendered in MuJoCo | | needs robosuite/robocasa/mujoco | **no** | yes | | runtime for 256 eps | minutes | ~2 h | --- ## What to bring ### 1. The builder and its checker ``` train_robocasa/scripts/dataset_build/build_vace_objwise_gr00t_dataset.py train_robocasa/scripts/dataset_build/repair_mimicgen_task_ids.py # also needed here train_robocasa/scripts/dataset_build/verify_geom_objwise_gr00t_dataset.py train_robocasa/scripts/dataset_build/merge_gr00t_view_datasets.py # only if merging with the 502 base train_robocasa/scripts/dataset_build/verify_merged_gr00t_dataset.py ``` Only `numpy` and `pandas` beyond the standard library, plus `ffmpeg` on `PATH` for the optional three-view preview (`--skip-three-view` avoids even that). ### 2. Inputs | `--flag` | what | where it comes from | |---|---|---| | `--vace-root` | the VACE output tree, one directory per assignment holding `work_//out_video.mp4` | `run_aug32.sh`'s work dir | | `--manifest` | `assignments.jsonl` (repeatable) | `aug32/build_aug32_assignments.py` | | `--source-root` | the original RoboCasa dataset — **the parquet is taken from here** | `datasets_pack/source_episodes.tar` | | `--canonical-meta-source` | a dataset whose `info.json` / `modality.json` / `view_manifest.json` contract is copied | **`dataset_build/canonical_meta/`**, bundled here | | `--output` | where the dataset is written | your choice | The defaults all point at the original machine, so **pass all five explicitly**. ### The 502-episode gr00t_views set is NOT needed Neither for training (we do not merge) nor for conversion: - its `meta/` was the only thing `--canonical-meta-source` reads, and those six files are bundled at `dataset_build/canonical_meta/` (96 KB); - the state/action rows come from `--source-root`, i.e. `pickplace_target_human`, whose parquet already carries **state 16-d and action 12-d** — the same schema as gr00t_views. `--canonical-meta-source` expects a *dataset directory* and reads `/meta/`, so `dataset_build/canonical_dataset/` is set up for exactly that — its `meta` is a symlink to `canonical_meta`. Pass `--canonical-meta-source dataset_build/canonical_dataset`. ### 3. Manifest fields the builder reads `case_dir`, `episode_index`, `provenance`, and the whole `source` block (episode, task, `lang`, `source_mesh`, `source_category`, per-camera `videos`). `build_aug32_assignments.py` writes all of them. `provenance` decides what gets in: ```python row.setdefault("provenance", "existing") if row["provenance"] == "random_fill" and not include_random_fill: continue # dropped ``` Our builder marks same-category assignments `existing` and the random remainder `random_fill`. With 32 per target that is **59 same-category and 197 random**, so the default keeps only 59. **Pass `--include-random-fill` to get all 256.** --- ## Run it ```bash python build_vace_objwise_gr00t_dataset.py \ --vace-root /path/to/vace_aug32_work \ --manifest /path/to/vace_aug32_work/assignments.jsonl \ --source-root $PWD/data/pickplace_target_human \ --canonical-meta-source $PWD/dataset_build/canonical_dataset \ --output /path/to/gr00t_views/vace_per_target_32_256eps \ --include-random-fill \ --skip-three-view python repair_mimicgen_task_ids.py --dataset /path/to/gr00t_views/vace_per_target_32_256eps python verify_geom_objwise_gr00t_dataset.py --dataset /path/to/gr00t_views/vace_per_target_32_256eps ``` `--copy-videos` materialises the videos instead of symlinking — needed if the dataset will be moved away from the VACE output tree. ### `repair_mimicgen_task_ids.py` is not optional Its docstring: the writer "stores all parquet task columns as 0 while episodes.jsonl stores the intended language per episode", and training reads the task id from the parquet — so without this pass **every augmented episode trains as task 0** and the language conditioning silently collapses. It also renames the modality keys to the canonical `left_view` / `right_view` / `wrist_view` that `single_panda_gripper` expects. --- ## What the result must look like ``` / meta/{info,modality,episodes,tasks,stats}.json all five, asserted by the loader data/chunk-000/episode_NNNNNN.parquet videos/chunk-000/observation.images.robot0_agentview_left/episode_NNNNNN.mp4 .../robot0_agentview_right/... .../robot0_eye_in_hand/... extras/episode_NNNNNN/... eval only, training never reads it ``` parquet columns (11): `observation.state` (list, **16**), `action` (list, **12**), `next.reward`, `next.done`, `timestamp`, `frame_index`, `episode_index`, `index`, `task_index`, `annotation.human.task_description` (**int index into tasks.jsonl**, not the string), `annotation.human.task_name`. `modality.json` slicing: ``` state base_position 0:3 | base_rotation 3:7 (quat) | EE_pos_rel 7:10 | EE_rot_rel 10:14 (quat) | gripper_qpos 14:16 action base_motion 0:4 | control_mode 4:5 | EE_pos 5:8 | EE_rot 8:11 | gripper_close 11:12 video left_view / right_view / wrist_view -> observation.images.robot0_agentview_{left,right} and observation.images.robot0_eye_in_hand ``` --- ## Then training `train_robocasa/` is a separate self-contained bundle (49 MB + the `mygr00t` conda env and a 7.1 GB base checkpoint, both referenced by absolute path). Point it at the dataset by absolute path rather than by name: ```bash cd train_robocasa DATASET_PATH=/path/to/gr00t_views/vace_per_target_32_256eps \ GPUS=0,1,2,3 MAX_STEPS=30000 SAVE_STEPS=5000 PER_GPU_BATCH=16 \ bash scripts/train_groot15_single_dataset.sh ``` `DATASETNAME` (the alternative) resolves against `/lp-dev/jonghoon/robocasa_full/gr00t_views/`, which will not exist elsewhere. Check first with `bash scripts/smoke_test.sh ` — imports, dataset load, and a real 20-step fine-tune. --- ## Checklist for the other machine - [ ] `run_aug32.sh` finished; every assignment has three `out_video.mp4` - [ ] `assignments.jsonl` carries `provenance` (re-run `build_aug32_assignments.py` if it predates 2026-09-19) - [ ] `source_episodes.tar` unpacked — **the parquet (`data/`) is read from it**; the archive was repacked 2026-09-19 to include it, an earlier copy has only `videos/` and `extras/` - [ ] `dataset_build/canonical_dataset/` present (its `meta` symlink must resolve; use `cp -rL` when moving) - [ ] `build_vace_objwise_gr00t_dataset.py` + `repair_mimicgen_task_ids.py` + a verifier copied over - [ ] python with `numpy`, `pandas`; `ffmpeg` unless `--skip-three-view` - [ ] decided: `--include-random-fill` (256) or not (59) - [ ] decided: merge with the 502 base or train on the augmented set alone