| # Turning VACE output into a trainable dataset — everything needed |
|
|
| After `run_aug32.sh` finishes you have per-episode videos. Training needs a LeRobot v2.1 |
| "gr00t_views" dataset. This is the full inventory for doing that conversion on a machine |
| that has nothing but this repo. |
| |
| Written 2026-09-19. The MimicGen side of the same job is documented in |
| `baseline/mimicgen/TRAINING_DATASET_PREP.md`; **the two are not the same procedure** — see |
| "Why VACE is the easy case" below. |
| |
| --- |
| |
| ## Why VACE is the easy case |
| |
| VACE repaints pixels; it never changes the trajectory. So the augmented episode's |
| state/action rows are **the source episode's own rows, unchanged**. The builder copies that |
| parquet, rewrites indices and task ids, and links the new videos in. No simulator, no |
| re-render, no MuJoCo. |
| |
| MimicGen is the opposite: every episode is a new rollout, so its parquet has to be rebuilt |
| from the generation HDF5 by replaying it in the simulator (~2 h for 256 episodes). |
| |
| | | VACE | MimicGen | |
| |---|---|---| |
| | builder | `build_vace_objwise_gr00t_dataset.py` | `render_to_gr00t.py` | |
| | parquet | copied from the source episode | rebuilt from HDF5 | |
| | video | symlinked from the VACE output | re-rendered in MuJoCo | |
| | needs robosuite/robocasa/mujoco | **no** | yes | |
| | runtime for 256 eps | minutes | ~2 h | |
| |
| --- |
| |
| ## What to bring |
| |
| ### 1. The builder and its checker |
| |
| ``` |
| train_robocasa/scripts/dataset_build/build_vace_objwise_gr00t_dataset.py |
| train_robocasa/scripts/dataset_build/repair_mimicgen_task_ids.py # also needed here |
| train_robocasa/scripts/dataset_build/verify_geom_objwise_gr00t_dataset.py |
| train_robocasa/scripts/dataset_build/merge_gr00t_view_datasets.py # only if merging with the 502 base |
| train_robocasa/scripts/dataset_build/verify_merged_gr00t_dataset.py |
| ``` |
| |
| Only `numpy` and `pandas` beyond the standard library, plus `ffmpeg` on `PATH` for the |
| optional three-view preview (`--skip-three-view` avoids even that). |
| |
| ### 2. Inputs |
| |
| | `--flag` | what | where it comes from | |
| |---|---|---| |
| | `--vace-root` | the VACE output tree, one directory per assignment holding `work_<view>/<inference_dir>/out_video.mp4` | `run_aug32.sh`'s work dir | |
| | `--manifest` | `assignments.jsonl` (repeatable) | `aug32/build_aug32_assignments.py` | |
| | `--source-root` | the original RoboCasa dataset — **the parquet is taken from here** | `datasets_pack/source_episodes.tar` | |
| | `--canonical-meta-source` | a dataset whose `info.json` / `modality.json` / `view_manifest.json` contract is copied | **`dataset_build/canonical_meta/`**, bundled here | |
| | `--output` | where the dataset is written | your choice | |
| |
| The defaults all point at the original machine, so **pass all five explicitly**. |
| |
| ### The 502-episode gr00t_views set is NOT needed |
| |
| Neither for training (we do not merge) nor for conversion: |
| |
| - its `meta/` was the only thing `--canonical-meta-source` reads, and those six files are |
| bundled at `dataset_build/canonical_meta/` (96 KB); |
| - the state/action rows come from `--source-root`, i.e. `pickplace_target_human`, whose |
| parquet already carries **state 16-d and action 12-d** — the same schema as gr00t_views. |
| |
| `--canonical-meta-source` expects a *dataset directory* and reads `<dir>/meta/`, so |
| `dataset_build/canonical_dataset/` is set up for exactly that — its `meta` is a symlink to |
| `canonical_meta`. Pass `--canonical-meta-source dataset_build/canonical_dataset`. |
| |
| ### 3. Manifest fields the builder reads |
| |
| `case_dir`, `episode_index`, `provenance`, and the whole `source` block (episode, task, |
| `lang`, `source_mesh`, `source_category`, per-camera `videos`). `build_aug32_assignments.py` |
| writes all of them. |
| |
| `provenance` decides what gets in: |
| |
| ```python |
| row.setdefault("provenance", "existing") |
| if row["provenance"] == "random_fill" and not include_random_fill: |
| continue # dropped |
| ``` |
| |
| Our builder marks same-category assignments `existing` and the random remainder |
| `random_fill`. With 32 per target that is **59 same-category and 197 random**, so the |
| default keeps only 59. **Pass `--include-random-fill` to get all 256.** |
|
|
| --- |
|
|
| ## Run it |
|
|
| ```bash |
| python build_vace_objwise_gr00t_dataset.py \ |
| --vace-root /path/to/vace_aug32_work \ |
| --manifest /path/to/vace_aug32_work/assignments.jsonl \ |
| --source-root $PWD/data/pickplace_target_human \ |
| --canonical-meta-source $PWD/dataset_build/canonical_dataset \ |
| --output /path/to/gr00t_views/vace_per_target_32_256eps \ |
| --include-random-fill \ |
| --skip-three-view |
| |
| python repair_mimicgen_task_ids.py --dataset /path/to/gr00t_views/vace_per_target_32_256eps |
| python verify_geom_objwise_gr00t_dataset.py --dataset /path/to/gr00t_views/vace_per_target_32_256eps |
| ``` |
|
|
| `--copy-videos` materialises the videos instead of symlinking — needed if the dataset will |
| be moved away from the VACE output tree. |
|
|
| ### `repair_mimicgen_task_ids.py` is not optional |
| |
| Its docstring: the writer "stores all parquet task columns as 0 while episodes.jsonl stores |
| the intended language per episode", and training reads the task id from the parquet — so |
| without this pass **every augmented episode trains as task 0** and the language conditioning |
| silently collapses. It also renames the modality keys to the canonical |
| `left_view` / `right_view` / `wrist_view` that `single_panda_gripper` expects. |
|
|
| --- |
|
|
| ## What the result must look like |
|
|
| ``` |
| <output>/ |
| meta/{info,modality,episodes,tasks,stats}.json all five, asserted by the loader |
| data/chunk-000/episode_NNNNNN.parquet |
| videos/chunk-000/observation.images.robot0_agentview_left/episode_NNNNNN.mp4 |
| .../robot0_agentview_right/... .../robot0_eye_in_hand/... |
| extras/episode_NNNNNN/... eval only, training never reads it |
| ``` |
|
|
| parquet columns (11): `observation.state` (list, **16**), `action` (list, **12**), |
| `next.reward`, `next.done`, `timestamp`, `frame_index`, `episode_index`, `index`, |
| `task_index`, `annotation.human.task_description` (**int index into tasks.jsonl**, not the |
| string), `annotation.human.task_name`. |
|
|
| `modality.json` slicing: |
|
|
| ``` |
| state base_position 0:3 | base_rotation 3:7 (quat) | EE_pos_rel 7:10 |
| | EE_rot_rel 10:14 (quat) | gripper_qpos 14:16 |
| action base_motion 0:4 | control_mode 4:5 | EE_pos 5:8 | EE_rot 8:11 | gripper_close 11:12 |
| video left_view / right_view / wrist_view -> observation.images.robot0_agentview_{left,right} |
| and observation.images.robot0_eye_in_hand |
| ``` |
|
|
| --- |
|
|
| ## Then training |
|
|
| `train_robocasa/` is a separate self-contained bundle (49 MB + the `mygr00t` conda env and a |
| 7.1 GB base checkpoint, both referenced by absolute path). Point it at the dataset by |
| absolute path rather than by name: |
|
|
| ```bash |
| cd train_robocasa |
| DATASET_PATH=/path/to/gr00t_views/vace_per_target_32_256eps \ |
| GPUS=0,1,2,3 MAX_STEPS=30000 SAVE_STEPS=5000 PER_GPU_BATCH=16 \ |
| bash scripts/train_groot15_single_dataset.sh |
| ``` |
|
|
| `DATASETNAME` (the alternative) resolves against |
| `/lp-dev/jonghoon/robocasa_full/gr00t_views/`, which will not exist elsewhere. |
|
|
| Check first with `bash scripts/smoke_test.sh <gpu>` — imports, dataset load, and a real |
| 20-step fine-tune. |
|
|
| --- |
|
|
| ## Checklist for the other machine |
|
|
| - [ ] `run_aug32.sh` finished; every assignment has three `out_video.mp4` |
| - [ ] `assignments.jsonl` carries `provenance` (re-run `build_aug32_assignments.py` if it predates 2026-09-19) |
| - [ ] `source_episodes.tar` unpacked — **the parquet (`data/`) is read from it**; the archive |
| was repacked 2026-09-19 to include it, an earlier copy has only `videos/` and `extras/` |
| - [ ] `dataset_build/canonical_dataset/` present (its `meta` symlink must resolve; use |
| `cp -rL` when moving) |
| - [ ] `build_vace_objwise_gr00t_dataset.py` + `repair_mimicgen_task_ids.py` + a verifier copied over |
| - [ ] python with `numpy`, `pandas`; `ffmpeg` unless `--skip-three-view` |
| - [ ] decided: `--include-random-fill` (256) or not (59) |
| - [ ] decided: merge with the 502 base or train on the augmented set alone |
| |
