mlnha commited on
Commit
5d08972
·
verified ·
1 Parent(s): d46ea65

pose6daug augmentation / dataset / training / eval scripts

Browse files
README.md ADDED
@@ -0,0 +1,163 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # pose6daug — augmentation, dataset build, training and evaluation scripts
2
+
3
+ The scripts behind the MimicGen and VACE augmentation baselines for RoboCasa
4
+ PickPlaceCounterToCabinet, and the GR00T 1.5 fine-tuning and exact-replay evaluation run on
5
+ top of them. Archived as-run: paths are absolute and point at one particular machine, so treat
6
+ this as a record of the procedure rather than a turnkey package. See **Paths to change** below.
7
+
8
+ ```
9
+ augment/mimicgen/ generate episodes with MimicGen, render preview videos
10
+ augment/vace/ VACE object-swap augmentation (assignment builder, runner, GT masks)
11
+ dataset/ convert generated episodes into a gr00t_views (LeRobot v2.1) dataset
12
+ train/ GR00T 1.5 fine-tuning launcher
13
+ eval/ exact-state replay evaluation
14
+ tools/ dataset sanity checks
15
+ ```
16
+
17
+ ## Pipeline
18
+
19
+ ```
20
+ MimicGen generation ──► demo.hdf5 (MuJoCo states + actions, no pixels)
21
+ │
22
+ ┌─────────┴──────────┐
23
+ ▼ ▼
24
+ preview mp4 (3 views) gr00t_views dataset ──► fine-tune ──► exact-replay eval
25
+ ```
26
+
27
+ VACE skips the simulator: it repaints pixels and keeps the source trajectory, so its builder
28
+ copies the source parquet instead of replaying.
29
+
30
+ ### 1. Augment
31
+
32
+ `augment/mimicgen/run_generation.sh` launches one MimicGen process per worker from a JSON
33
+ config per (object, worker); `config_template.json` is the template. Two knobs matter:
34
+
35
+ - `guarantee=false` makes `num_trials` mean **attempts**, not successes. Giving every object
36
+ the same attempt budget is what produces the generator's natural yield instead of a
37
+ per-object quota.
38
+ - `obj_registries` must include `aigen` for objects under `aigen_objs/` (e.g. `wine_5`), or
39
+ `sample_kitchen_object_helper` raises a bare `ValueError`.
40
+
41
+ `snapshot_episode_times.sh` records the per-episode temporary filenames while they exist:
42
+ MimicGen writes each success to `tmp/date_..._time_HH_MM_SS.hdf5`, then `merge_all_hdf5`
43
+ sorts by timestamp and deletes the folder. That snapshot is the only record of when each
44
+ episode was produced, and `select_by_generation_time.py` joins it back to order episodes
45
+ globally across workers. The poller can only ever miss a worker's **last** file, which the
46
+ selector pads with the merged file's mtime.
47
+
48
+ ### 2. Build the dataset
49
+
50
+ `dataset/render_to_gr00t.py` replays each episode in MuJoCo, renders three cameras and writes
51
+ parquet + videos + meta. `dataset/run_convert.sh` shards it (each shard needs its own
52
+ `GEN_DIR` and `DATASET_OUT`, or the glob picks up the others), then
53
+ `merge_gr00t_view_datasets.py` merges and `repair_mimicgen_task_ids.py` fixes task ids.
54
+
55
+ `repair_mimicgen_task_ids.py` is not optional: the writer stores every parquet task column as
56
+ 0 while `episodes.jsonl` holds the intended language, so without it every episode trains as
57
+ task 0 and the language conditioning silently collapses.
58
+
59
+ ### 3. Train
60
+
61
+ ```bash
62
+ DATASET_PATH=<gr00t_views dataset> DATASETNAME=<name> \
63
+ GPUS=2,3 PER_GPU_BATCH=32 MAX_STEPS=30000 SAVE_STEPS=5000 \
64
+ bash train/train_groot15_single_dataset.sh
65
+ ```
66
+
67
+ Fine-tunes the action-head projector and diffusion head from a base checkpoint; the backbone
68
+ stays frozen. `RESUME=1` picks up the newest checkpoint in the output directory.
69
+
70
+ ### 4. Evaluate
71
+
72
+ ```bash
73
+ MODEL_PATH=<checkpoint> MYGROOT_ROOT=<myGR00T tree> \
74
+ EVAL_CLIENT=<eval/eval_robocasa_replay_state_grasp.py> \
75
+ REPLAY_STATE_ROOT=<replay set> N_EPISODES=160 GPUS=4,5,6,7 POLICY_SEED=12345 SEED_BASE=42 \
76
+ bash eval/eval_groot15_exact_replay.sh
77
+ ```
78
+
79
+ Each episode restores a saved scene XML and flattened MuJoCo state, so runs are comparable
80
+ across checkpoints. Two independent seeds:
81
+
82
+ - `POLICY_SEED` — the policy server's global RNG **and** the per-step action seed
83
+ (`policy_seed + episode_index × stride + step`). This is the one that changes the rollout.
84
+ - `SEED_BASE` — the env seed, `SEED_BASE + worker_id` per worker. Exact replay overwrites the
85
+ scene immediately after, so it should not affect the initial state.
86
+
87
+ Worker count follows the GPU list, and episodes are split evenly across workers — so the same
88
+ `SEED_BASE` with a different GPU count gives each episode a different worker seed.
89
+
90
+ Client variants:
91
+
92
+ | file | adds |
93
+ |---|---|
94
+ | `eval_robocasa_replay_state_3view.py` | saves the wrist (ego) view; `RecordVideo` only captures `robot0_agentview_center` |
95
+ | `eval_robocasa_replay_state_grasp.py` | the above, plus `--episode_indices` for an arbitrary subset, and per-episode `grasped` / `lifted` / `in_cab` stage flags |
96
+
97
+ Stock RoboCasa success is `obj_inside_of(cab) and gripper_obj_far` — a single boolean, which
98
+ says nothing about where a failed episode broke down. The stage flags come from
99
+ `_check_grasp`, a 3 cm rise in the object's body z, and `OU.obj_inside_of`.
100
+
101
+ `organize_videos.py` regroups the output into one folder per global episode
102
+ (`rollouts/episode_NNNNNN/{center.mp4, wrist.mp4, info.txt}`); `RecordVideo` names files by
103
+ worker-local index, which does not match the global index.
104
+
105
+ ## The action-order bug — check any dataset before training on it
106
+
107
+ RoboCasa's simulator exports actions **arm-first**:
108
+
109
+ ```
110
+ [eef_pos(3), eef_rot(3), gripper, base(3), torso, base_mode]
111
+ ```
112
+
113
+ A gr00t_views dataset declares them **base-first**:
114
+
115
+ ```
116
+ base_motion[0:4] control_mode[4:5] eef_pos[5:8] eef_rot[8:11] gripper_close[11:12]
117
+ ```
118
+
119
+ A builder that re-derives actions from a simulator rollout has to reorder; one that copies
120
+ rows from an existing LeRobot dataset does not. Copying them through unchanged puts arm motion
121
+ on the base-motion channel and `base_mode` on the gripper. Nothing errors, the loss converges
122
+ to a small value, and the policy drives the base away from the counter and never closes the
123
+ gripper — **0/160** on an exact-replay eval whose base checkpoint scored 11/160.
124
+
125
+ ```python
126
+ actions = raw_actions[:, [7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6]]
127
+ ```
128
+
129
+ `tools/check_action_layout.py` tells the two layouts apart from the data alone (the mobile
130
+ base never moves in this task, so the constant dimensions give it away), and
131
+ `dataset/repair_action_order.py` fixes an already-built dataset in place — parquet plus the
132
+ action entry of `meta/stats.json`, no re-render. It refuses to run on a dataset that does not
133
+ look like simulator order, so it cannot be applied twice. Read `tools/ACTION_LAYOUT.md` first.
134
+
135
+ ## Paths to change
136
+
137
+ Every script hard-codes absolute paths from the machine this was run on. At minimum:
138
+
139
+ | what | appears as |
140
+ |---|---|
141
+ | RoboCasa / robosuite checkouts | `/lp-dev/jonghoon/robocasa_calib/repos/...` |
142
+ | MimicGen env + augmentation code | `/lp-dev/jonghoon/mimicgen_augment/...` |
143
+ | myGR00T tree and conda envs | `/data/minha/pose6daug/train_robocasa/myGR00T`, `/data/nvidia/gripper_augmentator/conda-envs/...` |
144
+ | base checkpoint | `/lp-dev/jonghoon/myGR00T_outputs/pnpcountertocab_all502_.../checkpoint-60000` |
145
+ | replay sets and eval output | `/lp-dev/jonghoon/isaac-gr00t/eval_results/...` |
146
+ | generation scratch | `/tmp/claude-.../scratchpad/...` |
147
+
148
+ `eval/eval_groot15_exact_replay.sh` defaults `MYGROOT_ROOT` to a path that no longer exists;
149
+ pass it explicitly. No credentials are embedded — the training launcher reads
150
+ `WANDB_API_KEY` from an env var or a file path you supply.
151
+
152
+ ## Attribution
153
+
154
+ `eval/eval_groot15_exact_replay.sh` and the upstream of `dataset/render_to_gr00t.py` and the
155
+ eval clients come from the pose6daug project's shared tree; the copies here carry the fixes
156
+ described above (action reorder, mesh-swap bbox fix, target sharding, wrist-view capture,
157
+ stage logging).
158
+
159
+ ## Related artifacts
160
+
161
+ - `mlnha/mimicgen-batch64-30k-ckpts` — checkpoints from the fixed-action-order run
162
+ - `mlnha/vace-batch64-30k-ckpts` — VACE run checkpoints
163
+ - `mlnha/mimicgen-pi05-aug256` — MimicGen augmentation for the pi0.5 hard-object set
augment/mimicgen/config_template.json ADDED
@@ -0,0 +1,90 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "name": "pick_place_counter_to_cabinet",
3
+ "type": "robocasa",
4
+ "experiment": {
5
+ "name": "nat_MeasuringCup009_w0",
6
+ "source": {
7
+ "dataset_path": "/tmp/claude-1001/-home-nvidia/68c27ffb-706b-4cb3-8bf2-4618a200d378/scratchpad/big/src150_fix.hdf5",
8
+ "filter_key": "valid",
9
+ "n": null,
10
+ "start": null
11
+ },
12
+ "generation": {
13
+ "path": "/tmp/claude-1001/-home-nvidia/68c27ffb-706b-4cb3-8bf2-4618a200d378/scratchpad/natural256/nat_MeasuringCup009_w0",
14
+ "guarantee": false,
15
+ "keep_failed": false,
16
+ "num_trials": 49,
17
+ "target_initial_state": {
18
+ "enabled": false,
19
+ "mode": null,
20
+ "demo_key": null,
21
+ "demo_key_policy": "fixed",
22
+ "object_name": "obj",
23
+ "asset_path": null,
24
+ "model_xml_mode": "regenerate"
25
+ },
26
+ "select_src_per_subtask": false,
27
+ "transform_first_robot_pose": false,
28
+ "interpolate_from_last_target_pose": true
29
+ },
30
+ "task": {
31
+ "name": "PickPlaceCounterToCabinet",
32
+ "robot": null,
33
+ "gripper": null,
34
+ "env_meta_update_kwargs": {
35
+ "env_kwargs": {
36
+ "obj_instance_split": null,
37
+ "obj_groups": "/lp-dev/jonghoon/robocasa_calib/repos/robocasa/robocasa/models/assets/objects/lightwheel/measuring_cup/MeasuringCup009/model.xml"
38
+ }
39
+ },
40
+ "interface": "MG_PickPlaceCounterToCabinet",
41
+ "interface_type": "robocasa"
42
+ },
43
+ "max_num_failures": 999999,
44
+ "max_num_problematic": null,
45
+ "render_video": false,
46
+ "num_demo_to_render": 50,
47
+ "num_fail_demo_to_render": 50,
48
+ "log_every_n_attempts": 10,
49
+ "seed": 101
50
+ },
51
+ "obs": {
52
+ "collect_obs": false,
53
+ "camera_names": [],
54
+ "camera_height": 84,
55
+ "camera_width": 84
56
+ },
57
+ "task": {
58
+ "task_spec": {
59
+ "subtask_1": {
60
+ "object_ref": "obj",
61
+ "subtask_term_signal": "stage_contact_obj",
62
+ "subtask_term_offset_range": [
63
+ 10,
64
+ 20
65
+ ],
66
+ "selection_strategy": "nearest_neighbor_object",
67
+ "selection_strategy_kwargs": {
68
+ "nn_k": 3
69
+ },
70
+ "action_noise": 0.0,
71
+ "num_interpolation_steps": 5,
72
+ "num_fixed_steps": 0,
73
+ "apply_noise_during_interpolation": false
74
+ },
75
+ "subtask_2": {
76
+ "object_ref": "cab",
77
+ "subtask_term_signal": null,
78
+ "subtask_term_offset_range": null,
79
+ "selection_strategy": "nearest_neighbor_object",
80
+ "selection_strategy_kwargs": {
81
+ "nn_k": 3
82
+ },
83
+ "action_noise": 0.0,
84
+ "num_interpolation_steps": 5,
85
+ "num_fixed_steps": 0,
86
+ "apply_noise_during_interpolation": false
87
+ }
88
+ }
89
+ }
90
+ }
augment/mimicgen/render_from_plan.py ADDED
@@ -0,0 +1,132 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """Render episodes listed in a render plan, one mp4 per camera.
3
+
4
+ The plan fixes every folder name up front, in one global pass over all runs, so a name is
5
+ unique across the whole output rather than only within the run that produced it. Assigning
6
+ names per run is what silently dropped episodes: a later run's success collided with an
7
+ earlier one's folder and was skipped as already done.
8
+
9
+ Plan entries are `{run_dir, demo, target, name}`. Shard with `--shard i --num-shards n` to
10
+ render in parallel; each episode writes a `.done` marker, so re-running skips finished work.
11
+
12
+ MUJOCO_GL=egl python render_from_plan.py --plan render_plan.json \
13
+ --source src150_fix.hdf5 --filter-key valid --out output_aug_256 --shard 0 --num-shards 6
14
+ """
15
+ from __future__ import annotations
16
+
17
+ import argparse
18
+ import json
19
+ import os
20
+ import time
21
+
22
+ os.environ.setdefault("MUJOCO_GL", "egl")
23
+ os.environ.setdefault("PYOPENGL_PLATFORM", "egl")
24
+
25
+ import h5py
26
+ import imageio
27
+ import mujoco
28
+ import numpy as np
29
+ import robocasa # noqa: F401
30
+ import robosuite
31
+ from robosuite.controllers import load_composite_controller_config
32
+
33
+ CAMERAS = ["robot0_agentview_left", "robot0_agentview_right", "robot0_eye_in_hand"]
34
+
35
+
36
+ def source_index(path, filter_key):
37
+ out = {}
38
+ with h5py.File(path, "r") as f:
39
+ keys = ([k.decode() for k in f[f"mask/{filter_key}"][:]] if filter_key
40
+ else sorted(f["data"].keys(), key=lambda x: int(x.split("_")[1])))
41
+ for i, k in enumerate(keys):
42
+ meta = json.loads(f[f"data/{k}"].attrs["ep_meta"])
43
+ cat = next((c["info"]["cat"] for c in meta.get("object_cfgs", [])
44
+ if c.get("name") == "obj"), "unknown")
45
+ out[i] = (k, cat)
46
+ return out
47
+
48
+
49
+ def build_env(env_args, res):
50
+ kw = dict(env_args["env_kwargs"])
51
+ kw.pop("env_name", None)
52
+ kw.update(has_renderer=False, has_offscreen_renderer=True, use_camera_obs=False,
53
+ use_object_obs=True, camera_depths=False, camera_names=CAMERAS,
54
+ camera_widths=res, camera_heights=res, ignore_done=True)
55
+ kw.setdefault("controller_configs",
56
+ load_composite_controller_config(controller=None, robot="PandaOmron"))
57
+ return robosuite.make(env_name=env_args["env_name"], **kw)
58
+
59
+
60
+ def main():
61
+ ap = argparse.ArgumentParser()
62
+ ap.add_argument("--plan", required=True)
63
+ ap.add_argument("--source", required=True)
64
+ ap.add_argument("--filter-key", default=None)
65
+ ap.add_argument("--out", required=True)
66
+ ap.add_argument("--shard", type=int, default=0)
67
+ ap.add_argument("--num-shards", type=int, default=1)
68
+ ap.add_argument("--res", type=int, default=256)
69
+ ap.add_argument("--fps", type=int, default=20)
70
+ a = ap.parse_args()
71
+
72
+ plan = [e for i, e in enumerate(json.load(open(a.plan))) if i % a.num_shards == a.shard]
73
+ src = source_index(a.source, a.filter_key)
74
+ os.makedirs(a.out, exist_ok=True)
75
+
76
+ env, opened, done = None, None, 0
77
+ for e in plan:
78
+ ep_dir = os.path.join(a.out, e["name"])
79
+ if os.path.exists(os.path.join(ep_dir, ".done")):
80
+ done += 1
81
+ continue
82
+ t0 = time.time()
83
+ with h5py.File(os.path.join(e["run_dir"], "demo.hdf5"), "r") as f:
84
+ g = f[f"data/{e['demo']}"]
85
+ states = g["states"][:]
86
+ meta = json.loads(g.attrs["ep_meta"])
87
+ model_xml = g.attrs["model_file"]
88
+ si = int(g["src_demo_inds"][0]) if "src_demo_inds" in g else -1
89
+ env_args = json.loads(f["data"].attrs["env_args"])
90
+ sk, scat = src.get(si, (f"idx{si}", "unknown"))
91
+
92
+ if env is None:
93
+ env = build_env(env_args, a.res)
94
+ env.set_ep_meta(meta)
95
+ env.reset()
96
+ env.reset_from_xml_string(model_xml)
97
+ env.sim.reset()
98
+
99
+ model, data = env.sim.model._model, env.sim.data._data
100
+ opt = mujoco.MjvOption()
101
+ opt.geomgroup[:] = 0
102
+ opt.geomgroup[1] = 1 # visual geoms, as the dataset's own videos are drawn
103
+ opt.sitegroup[:] = 0
104
+ r = mujoco.Renderer(model, height=a.res, width=a.res)
105
+ cam_ids = {c: mujoco.mj_name2id(model, mujoco.mjtObj.mjOBJ_CAMERA, c) for c in CAMERAS}
106
+ frames = {c: [] for c in CAMERAS}
107
+ for t in range(len(states)):
108
+ env.sim.set_state_from_flattened(states[t])
109
+ env.sim.forward()
110
+ for c in CAMERAS:
111
+ r.update_scene(data, camera=cam_ids[c], scene_option=opt)
112
+ frames[c].append(r.render().copy())
113
+ r.close()
114
+
115
+ for c in CAMERAS:
116
+ d = os.path.join(ep_dir, c)
117
+ os.makedirs(d, exist_ok=True)
118
+ imageio.mimwrite(os.path.join(d, f"{c}.mp4"), frames[c], fps=a.fps,
119
+ codec="libx264", macro_block_size=1,
120
+ ffmpeg_params=["-pix_fmt", "yuv420p"])
121
+ with open(os.path.join(ep_dir, ".done"), "w") as fh:
122
+ json.dump(dict(generated_demo=e["demo"], run_dir=e["run_dir"],
123
+ source_demo=sk, source_object=scat, target_object=e["target"],
124
+ frames=len(states), layout=meta.get("layout_id"),
125
+ style=meta.get("style_id"), lang=meta.get("lang")), fh, indent=1)
126
+ done += 1
127
+ print(f" [{a.shard}] {e['name']}: {len(states)}f {time.time()-t0:.0f}s", flush=True)
128
+ print(f"shard {a.shard}: {done}/{len(plan)}", flush=True)
129
+
130
+
131
+ if __name__ == "__main__":
132
+ raise SystemExit(main())
augment/mimicgen/run_generation.sh ADDED
@@ -0,0 +1,13 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env bash
2
+ # Launch all 32 equal-attempt generation jobs at once (one core each, no GPU).
3
+ set -u
4
+ MG=/lp-dev/jonghoon/mimicgen_augment/envs/mimicgen/bin/python
5
+ cd /home/nvidia/minha/pose6daug/baseline/mimicgen/mimicgen_mh || exit 1
6
+ OUT=/tmp/claude-1001/-home-nvidia/68c27ffb-706b-4cb3-8bf2-4618a200d378/scratchpad/natural256
7
+ date -u +"[nat] start %FT%TZ" | tee "$OUT/chain.log"
8
+ while IFS=$'\t' read -r name cfg log; do
9
+ MUJOCO_GL=egl nohup "$MG" mimicgen/scripts/generate_dataset.py \
10
+ --config "$cfg" --auto-remove-exp >"$log" 2>&1 &
11
+ done < "$OUT/jobs.txt"
12
+ wait
13
+ date -u +"[nat] generation done %FT%TZ" | tee -a "$OUT/chain.log"
augment/mimicgen/run_render.sh ADDED
@@ -0,0 +1,23 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env bash
2
+ # Preview mp4s, original RoboCasa asset, 3 cameras -- same layout as output_aug_256.
3
+ # taskset escapes this shell's 64-CPU affinity mask; EGL device round-robins over the
4
+ # three free GPUs (4-7 belong to another user's training).
5
+ set -u
6
+ N=/tmp/claude-1001/-home-nvidia/68c27ffb-706b-4cb3-8bf2-4618a200d378/scratchpad/natural256
7
+ S=/tmp/claude-1001/-home-nvidia/68c27ffb-706b-4cb3-8bf2-4618a200d378/scratchpad
8
+ MG=/lp-dev/jonghoon/mimicgen_augment/envs/mimicgen/bin/python
9
+ OUT=/home/nvidia/minha/pose6daug/baseline/mimicgen/output_natural_256
10
+ SHARDS=${SHARDS:-16}
11
+ GPUS=(0 2 3)
12
+ mkdir -p "$OUT" "$N/logs"
13
+ cd /home/nvidia/minha/pose6daug/baseline/mimicgen/mimicgen_augment || exit 1
14
+ for i in $(seq 0 $((SHARDS-1))); do
15
+ g=${GPUS[$((i % ${#GPUS[@]}))]}
16
+ MUJOCO_GL=egl PYOPENGL_PLATFORM=egl MUJOCO_EGL_DEVICE_ID=$g \
17
+ taskset -c 0-254 nohup "$MG" -u render_from_plan.py \
18
+ --plan "$N/render_plan.json" --source "$S/big/src150_fix.hdf5" \
19
+ --filter-key valid --out "$OUT" --shard "$i" --num-shards "$SHARDS" \
20
+ > "$N/logs/render_$i.log" 2>&1 &
21
+ done
22
+ wait
23
+ echo "[nat] render done $(date -u +%FT%TZ)"
augment/mimicgen/select_by_generation_time.py ADDED
@@ -0,0 +1,143 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """Pick the first 256 episodes in true generation order and lay out the work for both
3
+ downstream stages.
4
+
5
+ Every worker writes each success to tmp/date_..._time_HH_MM_SS.hdf5 before merge_all_hdf5
6
+ sorts those files by timestamp into demo_0..demo_N and deletes the folder. snapshot_times.sh
7
+ captured the filenames while they existed, so joining them back gives each merged demo its
8
+ wall-clock time -- and since all 32 workers share one machine clock, a global ordering.
9
+
10
+ Writes:
11
+ selection.json the ordered 256 with their timestamps
12
+ render_plan.json for render_from_plan.py (preview mp4s, original asset)
13
+ gen/shard_K/... per-shard gen trees for render_to_gr00t.py (dataset, swapped mesh)
14
+ """
15
+ import json, os, re, glob, sys, collections, datetime
16
+ import h5py
17
+
18
+ S = "/tmp/claude-1001/-home-nvidia/68c27ffb-706b-4cb3-8bf2-4618a200d378/scratchpad"
19
+ NAT = f"{S}/natural256"
20
+ SRC = f"{S}/big/src150_fix.hdf5"
21
+ N_KEEP = int(os.environ.get("N_KEEP", 256))
22
+ N_CONV_SHARDS = int(os.environ.get("N_CONV_SHARDS", 24))
23
+
24
+ CATEGORY = {
25
+ "donut_5": "donut", "Jar023": "jar", "MeasuringCup009": "measuring_cup",
26
+ "SoapDispenser010": "soap_dispenser", "steak_8": "steak",
27
+ "SyrupBottle005": "syrup_bottle", "teapot_6": "teapot", "teapot_7": "teapot",
28
+ }
29
+
30
+ # ---- source demo index -> (demo key, the object that source episode had) ----
31
+ src = {}
32
+ with h5py.File(SRC, "r") as f:
33
+ keys = [k.decode() for k in f["mask/valid"][:]]
34
+ for i, k in enumerate(keys):
35
+ meta = json.loads(f[f"data/{k}"].attrs["ep_meta"])
36
+ cat = next((c["info"]["cat"] for c in meta.get("object_cfgs", [])
37
+ if c.get("name") == "obj"), "unknown")
38
+ src[i] = (k, cat)
39
+
40
+ # ---- timestamps captured before the tmp folders were deleted ----
41
+ # One line per poll, so the same file appears many times and its mtime drifts while it is
42
+ # still being written -- dedupe on the name and order by the timestamp the name itself
43
+ # carries, which is fixed at creation.
44
+ seen = collections.defaultdict(dict)
45
+ with open(f"{NAT}/episode_times.tsv") as f:
46
+ for line in f:
47
+ parts = line.rstrip("\n").split("\t")
48
+ if len(parts) != 3:
49
+ continue
50
+ tmpdir, fname, _ = parts
51
+ m = re.match(r"date_(\d+)_(\d+)_(\d+)_time_(\d+)_(\d+)_(\d+)\.hdf5", fname)
52
+ if not m:
53
+ continue
54
+ worker = os.path.basename(os.path.dirname(os.path.dirname(tmpdir)))
55
+ mo, dy, yr, hh, mm, ss = (int(x) for x in m.groups())
56
+ seen[worker][fname] = datetime.datetime(
57
+ yr, mo, dy, hh, mm, ss, tzinfo=datetime.timezone.utc).timestamp()
58
+ times = {w: sorted((v, k) for k, v in d.items()) for w, d in seen.items()}
59
+
60
+ rows, missing = [], []
61
+ for wdir in sorted(glob.glob(f"{NAT}/nat_*")) + sorted(glob.glob(f"{NAT}/r2_*")):
62
+ worker = os.path.basename(wdir)
63
+ model = re.sub(r"^(nat|r2)_|_w\d+$", "", worker)
64
+ hp = f"{wdir}/{worker}/demo.hdf5"
65
+ if not os.path.isdir(wdir) or not os.path.exists(hp):
66
+ continue
67
+ with h5py.File(hp, "r") as f:
68
+ demos = sorted(f["data"].keys(), key=lambda x: int(x.split("_")[1]))
69
+ stamps = list(times.get(worker, []))
70
+ # The poller can miss the very last success: the worker merges and deletes tmp/
71
+ # immediately after writing it. Only the tail can ever be missing (earlier files sat
72
+ # there for the whole run), so pad it with the merged file's own mtime -- the moment
73
+ # the worker finished, which bounds that episode from above.
74
+ if len(stamps) < len(demos):
75
+ end = os.path.getmtime(hp)
76
+ pad = len(demos) - len(stamps)
77
+ missing.append(f"{worker}: padded {pad} with run end")
78
+ stamps += [(end, "<run end>")] * pad
79
+ for i, dk in enumerate(demos):
80
+ g = f[f"data/{dk}"]
81
+ si = int(g["src_demo_inds"][0]) if "src_demo_inds" in g else -1
82
+ sk, scat = src.get(si, (f"idx{si}", "unknown"))
83
+ t = stamps[i][0] if i < len(stamps) else float("inf")
84
+ rows.append(dict(t=t, worker=worker, run_dir=f"{wdir}/{worker}", demo=dk,
85
+ target=model, source_demo=sk, source_object=scat))
86
+
87
+ rows.sort(key=lambda r: (r["t"], r["worker"], r["demo"]))
88
+ print(f"총 성공 {len(rows)}개" + (f" (경고: {missing})" if missing else ""))
89
+ for i, r in enumerate(rows):
90
+ r["order"] = i
91
+ kept = rows[:N_KEEP]
92
+
93
+ taken = collections.Counter()
94
+ for r in kept:
95
+ base = f"{r['source_demo'].replace('demo_', 'ep')}_{r['source_object']}_{r['target']}"
96
+ taken[base] += 1
97
+ r["name"] = base if taken[base] == 1 else f"{base}_{taken[base]}"
98
+
99
+ json.dump(kept, open(f"{NAT}/selection.json", "w"), indent=1)
100
+ json.dump([{k: r[k] for k in ("run_dir", "demo", "target", "name")} for r in kept],
101
+ open(f"{NAT}/render_plan.json", "w"), indent=1)
102
+
103
+ # ---- conversion shards: contiguous chunks, each its own GEN_DIR tree ----
104
+ by_target = collections.defaultdict(list)
105
+ for r in kept:
106
+ by_target[r["target"]].append(r)
107
+ chunks = []
108
+ per = max(1, -(-len(kept) // N_CONV_SHARDS))
109
+ for model, rs in by_target.items():
110
+ for i in range(0, len(rs), per):
111
+ chunks.append((model, rs[i:i + per]))
112
+
113
+ for k, (model, rs) in enumerate(chunks):
114
+ name = f"{CATEGORY[model]}_{model}"
115
+ d = f"{NAT}/gen/shard_{k:02d}/{name}/{name}_gen"
116
+ os.makedirs(d, exist_ok=True)
117
+ out = f"{d}/demo.hdf5"
118
+ if os.path.exists(out):
119
+ os.remove(out)
120
+ with h5py.File(out, "w") as fo:
121
+ grp = fo.create_group("data")
122
+ total, env_args = 0, None
123
+ by_run = collections.defaultdict(list)
124
+ for r in rs:
125
+ by_run[r["run_dir"]].append(r)
126
+ j = 0
127
+ for run_dir, items in by_run.items():
128
+ with h5py.File(f"{run_dir}/demo.hdf5", "r") as fi:
129
+ if env_args is None:
130
+ env_args = fi["data"].attrs["env_args"]
131
+ for r in items:
132
+ fi.copy(f"data/{r['demo']}", grp, name=f"demo_{j}")
133
+ total += int(fi[f"data/{r['demo']}"].attrs["num_samples"])
134
+ j += 1
135
+ grp.attrs["total"] = total
136
+ grp.attrs["env_args"] = env_args
137
+
138
+ print(f"선택 {len(kept)}개 -> render_plan.json, 변환 샤드 {len(chunks)}개 "
139
+ f"(샤드당 최대 {per})")
140
+ c = collections.Counter(r["target"] for r in kept)
141
+ print("\n물체별 분포:")
142
+ for k, v in c.most_common():
143
+ print(f" {k:18s} {v}")
augment/mimicgen/select_by_generation_time_pi05.py ADDED
@@ -0,0 +1,80 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """Order every success by real generation time and keep the first 256.
3
+
4
+ Each worker writes a success to tmp/date_..._time_HH_MM_SS.hdf5 before merge_all_hdf5 sorts
5
+ those by timestamp into demo_0..demo_N and deletes the folder; snapshot.sh captured the names
6
+ while they existed. The poller can only ever miss a worker's LAST file (written immediately
7
+ before the merge), so a short tail is padded with the merged file's mtime.
8
+ """
9
+ import json, os, re, glob, collections, datetime
10
+ import h5py
11
+
12
+ S = "/tmp/claude-1001/-home-nvidia/68c27ffb-706b-4cb3-8bf2-4618a200d378/scratchpad"
13
+ Q = f"{S}/pi05_256"
14
+ SRC = f"{S}/big/src150_fix.hdf5"
15
+ N_KEEP = int(os.environ.get("N_KEEP", 256))
16
+
17
+ src = {}
18
+ with h5py.File(SRC, "r") as f:
19
+ for i, k in enumerate([k.decode() for k in f["mask/valid"][:]]):
20
+ meta = json.loads(f[f"data/{k}"].attrs["ep_meta"])
21
+ src[i] = (k, next((c["info"]["cat"] for c in meta.get("object_cfgs", [])
22
+ if c.get("name") == "obj"), "unknown"))
23
+
24
+ seen = collections.defaultdict(dict)
25
+ with open(f"{Q}/episode_times.tsv") as f:
26
+ for line in f:
27
+ p = line.rstrip("\n").split("\t")
28
+ if len(p) != 3:
29
+ continue
30
+ m = re.match(r"date_(\d+)_(\d+)_(\d+)_time_(\d+)_(\d+)_(\d+)\.hdf5", p[1])
31
+ if not m:
32
+ continue
33
+ w = os.path.basename(os.path.dirname(os.path.dirname(p[0])))
34
+ mo, dy, yr, hh, mm, ss = (int(x) for x in m.groups())
35
+ seen[w][p[1]] = datetime.datetime(yr, mo, dy, hh, mm, ss,
36
+ tzinfo=datetime.timezone.utc).timestamp()
37
+ times = {w: sorted(v for v in d.values()) for w, d in seen.items()}
38
+
39
+ rows, notes = [], []
40
+ for wdir in sorted(glob.glob(f"{Q}/q_*")):
41
+ if not os.path.isdir(wdir):
42
+ continue
43
+ worker = os.path.basename(wdir)
44
+ model = re.sub(r"^q_|_w\d+$", "", worker)
45
+ hp = f"{wdir}/{worker}/demo.hdf5"
46
+ if not os.path.exists(hp):
47
+ continue
48
+ with h5py.File(hp, "r") as f:
49
+ demos = sorted(f["data"].keys(), key=lambda x: int(x.split("_")[1]))
50
+ st = list(times.get(worker, []))
51
+ if len(st) < len(demos):
52
+ st += [os.path.getmtime(hp)] * (len(demos) - len(st))
53
+ notes.append(f"{worker}+{len(demos)-len(times.get(worker, []))}")
54
+ for i, dk in enumerate(demos):
55
+ g = f[f"data/{dk}"]
56
+ si = int(g["src_demo_inds"][0]) if "src_demo_inds" in g else -1
57
+ sk, scat = src.get(si, (f"idx{si}", "unknown"))
58
+ rows.append(dict(t=st[i], worker=worker, run_dir=f"{wdir}/{worker}",
59
+ demo=dk, target=model, source_demo=sk, source_object=scat))
60
+
61
+ rows.sort(key=lambda r: (r["t"], r["worker"], r["demo"]))
62
+ for i, r in enumerate(rows):
63
+ r["order"] = i
64
+ kept = rows[:N_KEEP]
65
+ taken = collections.Counter()
66
+ for r in kept:
67
+ base = f"{r['source_demo'].replace('demo_', 'ep')}_{r['source_object']}_{r['target']}"
68
+ taken[base] += 1
69
+ r["name"] = base if taken[base] == 1 else f"{base}_{taken[base]}"
70
+
71
+ json.dump(kept, open(f"{Q}/selection.json", "w"), indent=1)
72
+ json.dump([{k: r[k] for k in ("run_dir", "demo", "target", "name")} for r in kept],
73
+ open(f"{Q}/render_plan.json", "w"), indent=1)
74
+
75
+ print(f"총 성공 {len(rows)}개 -> 선택 {len(kept)}개" + (f" (tail 패딩: {len(notes)}워커)" if notes else ""))
76
+ tot = collections.Counter(r["target"] for r in rows)
77
+ sel = collections.Counter(r["target"] for r in kept)
78
+ print(f"\n{'물체':20s} {'전체':>6s} {'선택256':>8s}")
79
+ for m, _ in tot.most_common():
80
+ print(f" {m:20s} {tot[m]:6d} {sel[m]:8d}")
augment/mimicgen/snapshot_episode_times.sh ADDED
@@ -0,0 +1,16 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env bash
2
+ # Per-episode hdf5s are named with their completion timestamp, but merge_all_hdf5 deletes
3
+ # the tmp folder once a worker finishes. Snapshot the names while they still exist -- this
4
+ # is the only record of when each episode was actually produced.
5
+ S=/tmp/claude-1001/-home-nvidia/68c27ffb-706b-4cb3-8bf2-4618a200d378/scratchpad/natural256
6
+ M=$S/episode_times.tsv
7
+ touch "$M"
8
+ while pgrep -f "generate_dataset.py --config $S" >/dev/null; do
9
+ find "$S" -path "*/tmp/*.hdf5" -printf '%h\t%f\t%T@\n' 2>/dev/null >> "$M.new"
10
+ sort -u "$M" "$M.new" 2>/dev/null > "$M.tmp" && mv "$M.tmp" "$M"
11
+ rm -f "$M.new"
12
+ sleep 15
13
+ done
14
+ find "$S" -path "*/tmp/*.hdf5" -printf '%h\t%f\t%T@\n' 2>/dev/null >> "$M"
15
+ sort -u "$M" -o "$M"
16
+ echo "snapshot done: $(wc -l < "$M") episodes"
augment/vace/DATASET_CONVERSION.md ADDED
@@ -0,0 +1,182 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Turning VACE output into a trainable dataset — everything needed
2
+
3
+ After `run_aug32.sh` finishes you have per-episode videos. Training needs a LeRobot v2.1
4
+ "gr00t_views" dataset. This is the full inventory for doing that conversion on a machine
5
+ that has nothing but this repo.
6
+
7
+ Written 2026-09-19. The MimicGen side of the same job is documented in
8
+ `baseline/mimicgen/TRAINING_DATASET_PREP.md`; **the two are not the same procedure** — see
9
+ "Why VACE is the easy case" below.
10
+
11
+ ---
12
+
13
+ ## Why VACE is the easy case
14
+
15
+ VACE repaints pixels; it never changes the trajectory. So the augmented episode's
16
+ state/action rows are **the source episode's own rows, unchanged**. The builder copies that
17
+ parquet, rewrites indices and task ids, and links the new videos in. No simulator, no
18
+ re-render, no MuJoCo.
19
+
20
+ MimicGen is the opposite: every episode is a new rollout, so its parquet has to be rebuilt
21
+ from the generation HDF5 by replaying it in the simulator (~2 h for 256 episodes).
22
+
23
+ | | VACE | MimicGen |
24
+ |---|---|---|
25
+ | builder | `build_vace_objwise_gr00t_dataset.py` | `render_to_gr00t.py` |
26
+ | parquet | copied from the source episode | rebuilt from HDF5 |
27
+ | video | symlinked from the VACE output | re-rendered in MuJoCo |
28
+ | needs robosuite/robocasa/mujoco | **no** | yes |
29
+ | runtime for 256 eps | minutes | ~2 h |
30
+
31
+ ---
32
+
33
+ ## What to bring
34
+
35
+ ### 1. The builder and its checker
36
+
37
+ ```
38
+ train_robocasa/scripts/dataset_build/build_vace_objwise_gr00t_dataset.py
39
+ train_robocasa/scripts/dataset_build/repair_mimicgen_task_ids.py # also needed here
40
+ train_robocasa/scripts/dataset_build/verify_geom_objwise_gr00t_dataset.py
41
+ train_robocasa/scripts/dataset_build/merge_gr00t_view_datasets.py # only if merging with the 502 base
42
+ train_robocasa/scripts/dataset_build/verify_merged_gr00t_dataset.py
43
+ ```
44
+
45
+ Only `numpy` and `pandas` beyond the standard library, plus `ffmpeg` on `PATH` for the
46
+ optional three-view preview (`--skip-three-view` avoids even that).
47
+
48
+ ### 2. Inputs
49
+
50
+ | `--flag` | what | where it comes from |
51
+ |---|---|---|
52
+ | `--vace-root` | the VACE output tree, one directory per assignment holding `work_<view>/<inference_dir>/out_video.mp4` | `run_aug32.sh`'s work dir |
53
+ | `--manifest` | `assignments.jsonl` (repeatable) | `aug32/build_aug32_assignments.py` |
54
+ | `--source-root` | the original RoboCasa dataset — **the parquet is taken from here** | `datasets_pack/source_episodes.tar` |
55
+ | `--canonical-meta-source` | a dataset whose `info.json` / `modality.json` / `view_manifest.json` contract is copied | **`dataset_build/canonical_meta/`**, bundled here |
56
+ | `--output` | where the dataset is written | your choice |
57
+
58
+ The defaults all point at the original machine, so **pass all five explicitly**.
59
+
60
+ ### The 502-episode gr00t_views set is NOT needed
61
+
62
+ Neither for training (we do not merge) nor for conversion:
63
+
64
+ - its `meta/` was the only thing `--canonical-meta-source` reads, and those six files are
65
+ bundled at `dataset_build/canonical_meta/` (96 KB);
66
+ - the state/action rows come from `--source-root`, i.e. `pickplace_target_human`, whose
67
+ parquet already carries **state 16-d and action 12-d** — the same schema as gr00t_views.
68
+
69
+ `--canonical-meta-source` expects a *dataset directory* and reads `<dir>/meta/`, so
70
+ `dataset_build/canonical_dataset/` is set up for exactly that — its `meta` is a symlink to
71
+ `canonical_meta`. Pass `--canonical-meta-source dataset_build/canonical_dataset`.
72
+
73
+ ### 3. Manifest fields the builder reads
74
+
75
+ `case_dir`, `episode_index`, `provenance`, and the whole `source` block (episode, task,
76
+ `lang`, `source_mesh`, `source_category`, per-camera `videos`). `build_aug32_assignments.py`
77
+ writes all of them.
78
+
79
+ `provenance` decides what gets in:
80
+
81
+ ```python
82
+ row.setdefault("provenance", "existing")
83
+ if row["provenance"] == "random_fill" and not include_random_fill:
84
+ continue # dropped
85
+ ```
86
+
87
+ Our builder marks same-category assignments `existing` and the random remainder
88
+ `random_fill`. With 32 per target that is **59 same-category and 197 random**, so the
89
+ default keeps only 59. **Pass `--include-random-fill` to get all 256.**
90
+
91
+ ---
92
+
93
+ ## Run it
94
+
95
+ ```bash
96
+ python build_vace_objwise_gr00t_dataset.py \
97
+ --vace-root /path/to/vace_aug32_work \
98
+ --manifest /path/to/vace_aug32_work/assignments.jsonl \
99
+ --source-root $PWD/data/pickplace_target_human \
100
+ --canonical-meta-source $PWD/dataset_build/canonical_dataset \
101
+ --output /path/to/gr00t_views/vace_per_target_32_256eps \
102
+ --include-random-fill \
103
+ --skip-three-view
104
+
105
+ python repair_mimicgen_task_ids.py --dataset /path/to/gr00t_views/vace_per_target_32_256eps
106
+ python verify_geom_objwise_gr00t_dataset.py --dataset /path/to/gr00t_views/vace_per_target_32_256eps
107
+ ```
108
+
109
+ `--copy-videos` materialises the videos instead of symlinking — needed if the dataset will
110
+ be moved away from the VACE output tree.
111
+
112
+ ### `repair_mimicgen_task_ids.py` is not optional
113
+
114
+ Its docstring: the writer "stores all parquet task columns as 0 while episodes.jsonl stores
115
+ the intended language per episode", and training reads the task id from the parquet — so
116
+ without this pass **every augmented episode trains as task 0** and the language conditioning
117
+ silently collapses. It also renames the modality keys to the canonical
118
+ `left_view` / `right_view` / `wrist_view` that `single_panda_gripper` expects.
119
+
120
+ ---
121
+
122
+ ## What the result must look like
123
+
124
+ ```
125
+ <output>/
126
+ meta/{info,modality,episodes,tasks,stats}.json all five, asserted by the loader
127
+ data/chunk-000/episode_NNNNNN.parquet
128
+ videos/chunk-000/observation.images.robot0_agentview_left/episode_NNNNNN.mp4
129
+ .../robot0_agentview_right/... .../robot0_eye_in_hand/...
130
+ extras/episode_NNNNNN/... eval only, training never reads it
131
+ ```
132
+
133
+ parquet columns (11): `observation.state` (list, **16**), `action` (list, **12**),
134
+ `next.reward`, `next.done`, `timestamp`, `frame_index`, `episode_index`, `index`,
135
+ `task_index`, `annotation.human.task_description` (**int index into tasks.jsonl**, not the
136
+ string), `annotation.human.task_name`.
137
+
138
+ `modality.json` slicing:
139
+
140
+ ```
141
+ state base_position 0:3 | base_rotation 3:7 (quat) | EE_pos_rel 7:10
142
+ | EE_rot_rel 10:14 (quat) | gripper_qpos 14:16
143
+ action base_motion 0:4 | control_mode 4:5 | EE_pos 5:8 | EE_rot 8:11 | gripper_close 11:12
144
+ video left_view / right_view / wrist_view -> observation.images.robot0_agentview_{left,right}
145
+ and observation.images.robot0_eye_in_hand
146
+ ```
147
+
148
+ ---
149
+
150
+ ## Then training
151
+
152
+ `train_robocasa/` is a separate self-contained bundle (49 MB + the `mygr00t` conda env and a
153
+ 7.1 GB base checkpoint, both referenced by absolute path). Point it at the dataset by
154
+ absolute path rather than by name:
155
+
156
+ ```bash
157
+ cd train_robocasa
158
+ DATASET_PATH=/path/to/gr00t_views/vace_per_target_32_256eps \
159
+ GPUS=0,1,2,3 MAX_STEPS=30000 SAVE_STEPS=5000 PER_GPU_BATCH=16 \
160
+ bash scripts/train_groot15_single_dataset.sh
161
+ ```
162
+
163
+ `DATASETNAME` (the alternative) resolves against
164
+ `/lp-dev/jonghoon/robocasa_full/gr00t_views/`, which will not exist elsewhere.
165
+
166
+ Check first with `bash scripts/smoke_test.sh <gpu>` — imports, dataset load, and a real
167
+ 20-step fine-tune.
168
+
169
+ ---
170
+
171
+ ## Checklist for the other machine
172
+
173
+ - [ ] `run_aug32.sh` finished; every assignment has three `out_video.mp4`
174
+ - [ ] `assignments.jsonl` carries `provenance` (re-run `build_aug32_assignments.py` if it predates 2026-09-19)
175
+ - [ ] `source_episodes.tar` unpacked — **the parquet (`data/`) is read from it**; the archive
176
+ was repacked 2026-09-19 to include it, an earlier copy has only `videos/` and `extras/`
177
+ - [ ] `dataset_build/canonical_dataset/` present (its `meta` symlink must resolve; use
178
+ `cp -rL` when moving)
179
+ - [ ] `build_vace_objwise_gr00t_dataset.py` + `repair_mimicgen_task_ids.py` + a verifier copied over
180
+ - [ ] python with `numpy`, `pandas`; `ffmpeg` unless `--skip-three-view`
181
+ - [ ] decided: `--include-random-fill` (256) or not (59)
182
+ - [ ] decided: merge with the 502 base or train on the augmented set alone
augment/vace/README.md ADDED
@@ -0,0 +1,211 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # VACE augmentation — 8 objects × 32 episodes, 3 cameras
2
+
3
+ Swaps a target mesh into RoboCasa episodes with the VACE video-diffusion model, producing
4
+ the same folder layout as the MimicGen augmentation run so the two baselines can be
5
+ compared file-for-file.
6
+
7
+ **Same-category first, random for the remainder.** For `donut_5`, episodes that already
8
+ contain a donut (`donut_8`, `donut_9`, …) are used first — those are the swaps VACE was
9
+ designed for. Only after that pool runs out does the rest of the 32 come from randomly
10
+ drawn episodes of any category (apple → donut_5, milk → donut_5, …). The split is recorded
11
+ per assignment as `fill: same_category | random`.
12
+
13
+ How far the same-category pool goes, after `make_gt_masks.py` filled in the 58 missing
14
+ same-category masks (episodes with a complete set of GT masks, excluding the target mesh):
15
+
16
+ | target | same-category pool | same-category used | random fill | reaches 32 same-category? |
17
+ |---|---|---|---|---|
18
+ | MeasuringCup009 | 57 | **32** | 0 | **yes** |
19
+ | steak_8 | 18 | 18 | 14 | no |
20
+ | SoapDispenser010 | 5 | 5 | 27 | no |
21
+ | teapot_6 | 5 | 5 | 27 | no |
22
+ | teapot_7 | 5 | 5 | 27 | no |
23
+ | donut_5 | 3 | 3 | 29 | no |
24
+ | Jar023 | 2 | 2 | 30 | no |
25
+ | SyrupBottle005 | 2 | 2 | 30 | no |
26
+ | **total** | | **72** | **184** | |
27
+
28
+ Only MeasuringCup009 can be filled from its own category. This is a property of the
29
+ dataset, not of the masks: across all 2,517 episodes there are only 5 teapot episodes,
30
+ 3 donut, 2 jar, 2 syrup_bottle. Generating the other 1,830 missing masks would not move
31
+ these numbers — only 58 of the 1,888 were same-category for these eight targets (55
32
+ measuring_cup, 3 steak), and those are already done.
33
+
34
+ ## Run
35
+
36
+ ```bash
37
+ ./run_aug32.sh <work_dir> <out_dir> <gpu> [num_wan_workers]
38
+
39
+ # one GPU
40
+ ./run_aug32.sh /scratch/vace_aug32 \
41
+ /home/nvidia/minha/pose6daug/baseline/mimicgen/output_aug_256 2
42
+
43
+ # four GPUs: split-wan writes four manifests, one worker each
44
+ ./run_aug32.sh /scratch/vace_aug32 /path/out 2 4 # then move the workers onto 4 GPUs by hand
45
+ ```
46
+
47
+ Every stage is idempotent — re-running after an interruption resumes rather than redoing.
48
+
49
+ | stage | what it does | cost |
50
+ |---|---|---|
51
+ | 1 `build` | pick sources, same-category first | seconds |
52
+ | 2 `prep` | render the target mesh's reference views, cut frames, per-view manifests | ~4 s / assignment |
53
+ | 3 `split` | chop into 81-frame WAN jobs | seconds |
54
+ | 4 `wan` | the diffusion pass | **everything** |
55
+ | 5 `stitch` | glue each camera's chunks back together | seconds |
56
+ | 6 `collect` | copy into the output layout | seconds |
57
+
58
+ **Runtime.** 256 assignments × 3 cameras ≈ **3,084 WAN jobs**. At the measured 51.4 s/job
59
+ that is **~44 h on one GPU**, ~11 h on four. Levers, in order of how much they buy and how
60
+ little they cost:
61
+
62
+ - `--offload_model False` is already passed by the script. The upstream default is `True`
63
+ whenever `world_size == 1`, which moves the DiT to CPU and back around every job: 81.8 s
64
+ vs 51.4 s measured. Never drop this flag.
65
+ - `--sample_steps 25` (upstream default 50) → 32.1 s/job, ~27 h on one GPU. Quality
66
+ trade-off; look at the plates before committing.
67
+ - Dropping the two exo cameras cuts the job count to a third. At 256×256 the target-object
68
+ mask is ~0.2 % of an exo frame and VACE tends to erase the object instead of replacing it;
69
+ only `robot0_eye_in_hand` (~9 %) reliably gets a real swap. If the exo views are not
70
+ needed, edit `CAMS` in `../vace_object_insert_batch.py`.
71
+
72
+ ## Output
73
+
74
+ ```
75
+ <out_dir>/<task>-ep<N>_<object the episode had>_<object it becomes>/
76
+ robot0_agentview_left/robot0_agentview_left.mp4
77
+ robot0_agentview_right/robot0_agentview_right.mp4
78
+ robot0_eye_in_hand/robot0_eye_in_hand.mp4
79
+ .done
80
+ ```
81
+
82
+ e.g. `c2cab-ep000151_donut_donut_5` (same-category) and
83
+ `snk2c-ep000004_egg_teapot_7` (random fill). Task tags are `c2cab`, `c2sto`, `drw2c`,
84
+ `snk2c`, `tst2c`; the hyphen keeps the name to three underscore-separated parts, matching
85
+ MimicGen's `ep098_apple_teapot_6`. `.done` carries the real task, episode, source mesh,
86
+ target, language instruction, and which fill it came from.
87
+
88
+ ## Files
89
+
90
+ | file | |
91
+ |---|---|
92
+ | `build_aug32_assignments.py` | source selection — the only new logic |
93
+ | `collect_outputs.py` | VACE plates → the output layout |
94
+ | `run_aug32.sh` | all six stages |
95
+ | `SETUP.md` | **start here on a new machine** — the setup path, in order |
96
+ | `preflight.sh` | checks environments, weights, datasets, GT-mask coverage, GPU |
97
+ | `make_gt_masks.py` | re-renders the missing GT masks from the simulator |
98
+ | `vace_wan_inference_batch.patch` | the upstream diff, for porting onto a fresh clone |
99
+ | `vendor/render_mesh_views_side.py` | mesh reference-view renderer, vendored from the original machine |
100
+
101
+ Stages 2–5 are `../vace_object_insert_batch.py` and `../VACE/vace/vace_wan_inference.py`
102
+ unmodified; this directory adds selection and collection around them.
103
+
104
+ ## Prerequisites
105
+
106
+ Run `./preflight.sh` — it checks every item below and prints GT-mask coverage and GPU
107
+ state. Nothing here needs to be installed or cloned; it all already exists on this box and
108
+ is referenced by absolute path, the same convention the other baselines use.
109
+
110
+ ### Environments (do NOT create a venv)
111
+
112
+ | what | python |
113
+ |---|---|
114
+ | batch driver (`vace_object_insert_batch.py`, the builder, the collector) | `/lp-dev/jonghoon/robocasa_calib/envs/robocasa/bin/python` (3.10, mujoco + robocasa) |
115
+ | WAN inference | `/lp-dev/jonghoon/VACE/venv/bin/python` (3.12, torch 2.5.1+cu124, flash_attn 2.8.3) |
116
+ | mesh reference-view rendering (called by `prep-worker`) | `/lp-dev/jonghoon/isaac-gr00t/venv/bin/python` — override with `VACE_RENDER_PYTHON`; it only needs `trimesh`, `pyrender`, `numpy`, `pillow` (measured: 4.12.1 / 0.1.45 / 1.26.4 / 12.1.1), so any env with those will do |
117
+
118
+ `prep-worker` renders the target mesh's reference views by importing `render_views` from
119
+ `render_mesh_views_side.py`, which lived outside this tree on the original machine. A copy
120
+ ships in **`vendor/render_mesh_views_side.py`**; point at it with
121
+
122
+ ```bash
123
+ export VACE_RENDER_SCRIPT_DIR=$PWD/aug32/vendor
124
+ export VACE_RENDER_PYTHON=/path/to/a/python/with/trimesh+pyrender
125
+ ```
126
+
127
+ ### VACE code and weights (already cloned — do NOT clone again)
128
+
129
+ `../VACE/` is a fork of `https://github.com/ali-vilab/VACE.git` carrying the batch flags
130
+ (`--batch_manifest / --skip_existing / --batch_continue_on_error`) this pipeline depends on;
131
+ a fresh upstream clone will not have them. Weights come in through
132
+ `../VACE/models -> /lp-dev/jonghoon/VACE-home-models` (31 GB: `Wan2.1-VACE-1.3B`,
133
+ `VACE-Annotators`). ~27 GB of GPU memory is needed with `--offload_model False`.
134
+
135
+ ### Datasets
136
+
137
+ | path | what |
138
+ |---|---|
139
+ | `/home/nvidia/jonghoon/robocasa_full/pickplace_target_human/` | source episodes — `videos/chunk-000/observation.images.<camera>/episode_NNNNNN.mp4` and `extras/episode_NNNNNN/ep_meta.json` (symlink to `/lp-dev/...`) |
140
+ | `/home/nvidia/jonghoon/robocasa_full/aug_materials/robocasa/<task>/ep<N>/` | `mask_gt_target_object_<camera>.mp4` — the GT masks |
141
+ | `/lp-dev/jonghoon/robocasa_calib/repos/robocasa/robocasa/models/assets/objects/{objaverse,lightwheel}/` | target meshes |
142
+
143
+ GT-mask coverage, measured: `PickPlaceCounterToCabinet` 502, `CounterToStove` 50,
144
+ `DrawerToCounter` 43, `SinkToCounter` 34, `ToasterToCounter` 0 — 629 usable episodes, and
145
+ the builder draws only from those.
146
+
147
+ ## Porting to another machine
148
+
149
+ Do **not** clone `ali-vilab/VACE` fresh — upstream has no `--batch_manifest`, which every
150
+ stage past `split-wan` depends on.
151
+
152
+ **Copy the whole `baseline/vace/` directory**, not just `aug32/`. Nothing outside it is
153
+ needed except the 31 GB of WAN weights. To rebuild from a clone instead:
154
+ `git clone https://github.com/ali-vilab/VACE.git && git checkout 48eb44f`, then
155
+ `git apply vace_wan_inference_batch.patch`, and copy `build_vace_wan_batch_manifest.py`,
156
+ `vace_inference_full.py` and this `aug32/` directory alongside it.
157
+
158
+ ### Unpack the data
159
+
160
+ ```bash
161
+ cd baseline/vace && mkdir -p data && cd data
162
+ tar -xf ../datasets_pack/source_episodes.tar # 2.7 GB -> pickplace_target_human/
163
+ tar -xzf ../datasets_pack/aug_materials_ext.tar.gz # 121 MB -> aug_materials_ext/
164
+ tar -xzf ../datasets_pack/target_meshes.tar.gz # 14 MB -> objaverse/, lightwheel/
165
+ cd ..
166
+ ```
167
+
168
+ ### Point everything at it
169
+
170
+ Every path is an environment variable with the original machine's value as the default, so
171
+ this is the whole configuration:
172
+
173
+ | variable | default | set it to |
174
+ |---|---|---|
175
+ | `VACE_ROBOCASA_ROOT` | original machine | `$PWD/data/pickplace_target_human` |
176
+ | `VACE_AUG_MATERIALS` | original machine | `$PWD/data/aug_materials_ext` |
177
+ | `VACE_ASSET_ROOTS` | original machine | `$PWD/data/lightwheel:$PWD/data/objaverse` |
178
+ | `VACE_PY_ROBOCASA` | original machine | a python with `robocasa`, `mujoco`, `trimesh`, `pyrender` |
179
+ | `VACE_PY_WAN` | original machine | a python with torch 2.5.1+cu124 and `flash_attn` |
180
+ | `VACE_RENDER_SCRIPT_DIR` | **`aug32/vendor/`** | — already inside the tree |
181
+ | `VACE_RENDER_PYTHON` | **`VACE_PY_ROBOCASA`** | — only if that env lacks trimesh/pyrender |
182
+
183
+ ```bash
184
+ export VACE_ROBOCASA_ROOT=$PWD/data/pickplace_target_human
185
+ export VACE_AUG_MATERIALS=$PWD/data/aug_materials_ext
186
+ export VACE_ASSET_ROOTS=$PWD/data/lightwheel:$PWD/data/objaverse
187
+ export VACE_PY_ROBOCASA=/path/to/robocasa/python
188
+ export VACE_PY_WAN=/path/to/wan/python
189
+ ./aug32/preflight.sh # says MISS for anything still wrong
190
+ ```
191
+
192
+ The mesh reference-view renderer (`render_mesh_views_side.py`, which `prep-worker` imports)
193
+ is vendored at `aug32/vendor/` and defaults to the driver's own interpreter, so it needs no
194
+ separate environment — `trimesh` and `pyrender` are its only requirements beyond numpy/PIL.
195
+
196
+ ### Still needed from elsewhere
197
+
198
+ The 31 GB of WAN weights (`Wan2.1-VACE-1.3B`, `VACE-Annotators`) — link them in at
199
+ `VACE/models`. And, only if you want to re-run `make_gt_masks.py` there, the meshes of
200
+ objects other than the eight targets, since that rebuilds each episode's own scene.
201
+
202
+ ### Gotchas
203
+
204
+ - An episode without GT masks makes `prep-worker` raise `FileNotFoundError`. The builder
205
+ filters on this, so it cannot happen through `run_aug32.sh` — but it will if you hand-edit
206
+ `assignments.jsonl`.
207
+ - The WAN stage must run with `vace/VACE` as the working directory (imports and `--ckpt_dir`
208
+ are relative to it). `run_aug32.sh` does the `cd`.
209
+ - `--save_file` in `vace_wan_inference.py` is written relative to the *current* directory,
210
+ not `--save_dir`. The batch path does not use it; a one-off call should pass an absolute
211
+ path.
augment/vace/SETUP.md ADDED
@@ -0,0 +1,138 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # First hour on a new machine
2
+
3
+ Do these in order. Everything else — how the pipeline works, why the sources are picked the
4
+ way they are, runtime numbers — is in `README.md`; this file is only the setup path.
5
+
6
+ Assumes you have this repo and nothing else.
7
+
8
+ ---
9
+
10
+ ## 0. What you must bring yourself
11
+
12
+ | | size | why it is not here |
13
+ |---|---|---|
14
+ | **WAN weights** — `Wan2.1-VACE-1.3B`, `VACE-Annotators` | 31 GB | too large to ship; get them from the VACE release or copy from the original machine |
15
+ | **GPU** with ≥ 27 GB free | | `--offload_model False` keeps the DiT resident |
16
+
17
+ Everything else is in this repo.
18
+
19
+ ---
20
+
21
+ ## 1. Unpack the data (~3 GB, 2 min)
22
+
23
+ ```bash
24
+ cd <repo>/ # the directory holding aug32/, VACE/, datasets_pack/
25
+ mkdir -p data && cd data
26
+ tar -xf ../datasets_pack/source_episodes.tar # 2.7 GB -> pickplace_target_human/
27
+ tar -xzf ../datasets_pack/aug_materials_ext.tar.gz # 121 MB -> aug_materials_ext/
28
+ tar -xzf ../datasets_pack/target_meshes.tar.gz # 14 MB -> objaverse/ lightwheel/
29
+ cd ..
30
+ ```
31
+
32
+ Check:
33
+
34
+ ```
35
+ data/pickplace_target_human/<5 tasks>/{videos,extras}
36
+ data/aug_materials_ext/<4 tasks>/ep<N>/mask_gt_target_object_<camera>.mp4
37
+ data/{objaverse,lightwheel}/<category>/<model>/model.xml
38
+ ```
39
+
40
+ ---
41
+
42
+ ## 2. Link the WAN weights
43
+
44
+ ```bash
45
+ ln -s /path/to/VACE-home-models VACE/models
46
+ ls VACE/models # must show Wan2.1-VACE-1.3B and VACE-Annotators
47
+ ```
48
+
49
+ ---
50
+
51
+ ## 3. Build two Python environments
52
+
53
+ **A — batch driver.** Needs `robocasa`, `robosuite`, `mujoco`, `trimesh`, `pyrender`,
54
+ `opencv-python`, `numpy`. This one also renders the target meshes' reference views, so
55
+ `trimesh` + `pyrender` are not optional. Versions that work: mujoco 3.3.1, robosuite 1.5.2,
56
+ robocasa 1.0.0, trimesh 4.12.1, pyrender 0.1.45, numpy 1.26.4, Python 3.10.
57
+
58
+ **B — WAN inference.** Python 3.12, `torch 2.5.1+cu124`, `flash_attn 2.8.3`, plus
59
+ `VACE/requirements.txt`. `wan` must be importable (it is a site-package in the original
60
+ environment, installed from the VACE repo).
61
+
62
+ ```bash
63
+ export VACE_PY_ROBOCASA=/path/to/env-A/bin/python
64
+ export VACE_PY_WAN=/path/to/env-B/bin/python
65
+ ```
66
+
67
+ ---
68
+
69
+ ## 4. Export the data paths
70
+
71
+ ```bash
72
+ cd <repo>
73
+ export VACE_ROBOCASA_ROOT=$PWD/data/pickplace_target_human
74
+ export VACE_AUG_MATERIALS=$PWD/data/aug_materials_ext
75
+ export VACE_ASSET_ROOTS=$PWD/data/lightwheel:$PWD/data/objaverse
76
+ ```
77
+
78
+ Leave `VACE_RENDER_SCRIPT_DIR` and `VACE_RENDER_PYTHON` alone — they default to
79
+ `aug32/vendor/` and to `VACE_PY_ROBOCASA`, both already correct.
80
+
81
+ Put all five exports in a file you can `source`; every later step needs them.
82
+
83
+ ---
84
+
85
+ ## 5. Preflight
86
+
87
+ ```bash
88
+ ./aug32/preflight.sh
89
+ ```
90
+
91
+ Every line must read `OK`. It checks both interpreters, the vendored renderer and its
92
+ imports, the VACE fork and weights, the three data roots, per-task GT-mask coverage, and
93
+ free GPU memory. A `MISS` names the exact path that is wrong.
94
+
95
+ ---
96
+
97
+ ## 6. Smoke test before the real batch
98
+
99
+ One assignment end to end, ~10 min on one GPU, to catch a broken path before a multi-hour
100
+ run:
101
+
102
+ ```bash
103
+ mkdir -p /tmp/vace_smoke
104
+ $VACE_PY_ROBOCASA aug32/build_aug32_assignments.py \
105
+ --out-root /tmp/vace_smoke --per-mesh 1 --targets teapot/teapot_6
106
+ head -1 /tmp/vace_smoke/assignments.jsonl > /tmp/vace_smoke/one.jsonl
107
+ mv /tmp/vace_smoke/one.jsonl /tmp/vace_smoke/assignments.jsonl
108
+ ./aug32/run_aug32.sh /tmp/vace_smoke /tmp/vace_smoke_out 0
109
+ ```
110
+
111
+ Expect `/tmp/vace_smoke_out/<task>-ep<N>_<src>_teapot_6/` with three camera folders, each
112
+ holding one mp4.
113
+
114
+ ---
115
+
116
+ ## 7. The real run
117
+
118
+ ```bash
119
+ ./aug32/run_aug32.sh /scratch/vace_aug32 /scratch/output_aug_256 0
120
+ ```
121
+
122
+ 8 targets × 32 episodes × 3 cameras ≈ 3,084 WAN jobs, **~44 h on one GPU** at the measured
123
+ 51.4 s/job. Split it: `run_aug32.sh ... <gpu> 4` writes four worker manifests, and the four
124
+ WAN processes can be moved onto four GPUs (~11 h). Every stage is idempotent, so an
125
+ interrupted run resumes where it stopped.
126
+
127
+ ---
128
+
129
+ ## Common failures
130
+
131
+ | symptom | cause |
132
+ |---|---|
133
+ | `FileNotFoundError: Missing GT target-object mask video` | `VACE_AUG_MATERIALS` not exported, or pointing above `aug_materials_ext/` |
134
+ | `ModuleNotFoundError: No module named 'trimesh'` during `prep-worker` | env A lacks trimesh/pyrender — install them, or point `VACE_RENDER_PYTHON` at an env that has them |
135
+ | `unrecognized arguments: --batch_manifest` | `VACE/` was replaced with a fresh upstream clone; apply `aug32/vace_wan_inference_batch.patch` |
136
+ | WAN job ~82 s instead of ~51 s | `--offload_model False` got dropped; it is in `run_aug32.sh`, do not remove it |
137
+ | exo cameras come out with the object *erased* rather than swapped | expected at 256×256 — the target mask is ~0.2 % of an exo frame. Only `robot0_eye_in_hand` (~9 %) reliably swaps |
138
+ | assignments have fewer same-category sources than expected | the dataset only holds 5 teapot / 3 donut / 2 jar / 2 syrup_bottle episodes. Only MeasuringCup009 can fill 32 from its own category |
augment/vace/build_aug32_assignments.py ADDED
@@ -0,0 +1,148 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """Build VACE assignments: same-category sources first, random episodes for the remainder.
3
+
4
+ For each target mesh we want N augmentations. Episodes whose own object is the same
5
+ category as the target are used first (donut_8, donut_9 ... -> donut_5), because those are
6
+ the swaps VACE was designed for. Only once that pool is exhausted does the rest get filled
7
+ from randomly drawn episodes of any category (apple, milk, ... -> donut_5).
8
+
9
+ Only episodes with a complete set of GT target-object mask videos are eligible --
10
+ `prep-worker` fails on any other. No source episode is used twice for the same target.
11
+
12
+ The stock `vace_object_insert_batch.py build` cannot express this: it draws a target's
13
+ sources from its own category only, and round-robins with wraparound so a short pool comes
14
+ back as duplicates. This writes the same `assignments.jsonl` the stock builder writes, so
15
+ every later stage (`prep-worker`, `split-wan`, the WAN batch, `stitch-segments`) is the
16
+ unmodified upstream code.
17
+
18
+ python build_aug32_assignments.py --out-root DIR [--per-mesh 32] [--seed 0]
19
+ """
20
+ from __future__ import annotations
21
+
22
+ import argparse
23
+ import importlib.util
24
+ import json
25
+ import random
26
+ import sys
27
+ from collections import Counter
28
+ from pathlib import Path
29
+
30
+ HERE = Path(__file__).resolve().parent
31
+ spec = importlib.util.spec_from_file_location(
32
+ "vace_batch", HERE.parent / "vace_object_insert_batch.py")
33
+ mg = importlib.util.module_from_spec(spec)
34
+ spec.loader.exec_module(mg)
35
+
36
+ TARGETS = [
37
+ "donut/donut_5",
38
+ "jar/Jar023",
39
+ "measuring_cup/MeasuringCup009",
40
+ "soap_dispenser/SoapDispenser010",
41
+ "steak/steak_8",
42
+ "syrup_bottle/SyrupBottle005",
43
+ "teapot/teapot_6",
44
+ "teapot/teapot_7",
45
+ ]
46
+
47
+ # short, underscore-free task tags so the folder name stays three underscore-separated parts
48
+ TASK_TAG = {
49
+ "PickPlaceCounterToCabinet": "c2cab",
50
+ "PickPlaceCounterToStove": "c2sto",
51
+ "PickPlaceDrawerToCounter": "drw2c",
52
+ "PickPlaceSinkToCounter": "snk2c",
53
+ "PickPlaceToasterToCounter": "tst2c",
54
+ }
55
+
56
+
57
+ class AnyCategory:
58
+ """Container that accepts every category, so the stock scan keeps all episodes."""
59
+
60
+ def __contains__(self, value):
61
+ return value is not None
62
+
63
+
64
+ def has_gt_masks(row: dict) -> bool:
65
+ ep_dir = mg.AUG_MATERIALS_ROOT / row["task"] / f"ep{int(row['episode']):06d}"
66
+ return all(
67
+ (ep_dir / f"mask_gt_target_object_{cam}.mp4").exists()
68
+ or (ep_dir / "encoded_videos" / f"mask_gt_target_object_{cam}.mp4").exists()
69
+ for cam in mg.CAMS
70
+ )
71
+
72
+
73
+ def run_name_for(src: dict, target_model: str) -> str:
74
+ """<task tag>-ep<N>_<object that episode had>_<object it becomes>, as MimicGen names them."""
75
+ tag = TASK_TAG.get(src["task"], mg.sanitize(src["task"]))
76
+ return (f"{tag}-ep{int(src['episode']):06d}"
77
+ f"_{mg.sanitize(src['source_category'])}"
78
+ f"_{mg.sanitize(target_model)}")
79
+
80
+
81
+ def main() -> int:
82
+ ap = argparse.ArgumentParser()
83
+ ap.add_argument("--out-root", type=Path, required=True)
84
+ ap.add_argument("--per-mesh", type=int, default=32)
85
+ ap.add_argument("--targets", nargs="*", default=TARGETS)
86
+ ap.add_argument("--dataset-root", type=Path, default=mg.ROBOCASA_ROOT)
87
+ ap.add_argument("--seed", type=int, default=0)
88
+ a = ap.parse_args()
89
+
90
+ by_cat = mg.scan_sources(a.dataset_root, AnyCategory())
91
+ pool = [r for rows in by_cat.values() for r in rows if has_gt_masks(r)]
92
+ pool.sort(key=lambda r: (r["task"], r["episode"]))
93
+ print(f"GT 마스크 보유 소스: {len(pool)}개", file=sys.stderr, flush=True)
94
+
95
+ rng = random.Random(a.seed)
96
+ a.out_root.mkdir(parents=True, exist_ok=True)
97
+ assignments, summary = [], {}
98
+ idx = 0
99
+ for target_mesh in a.targets:
100
+ category, model = target_mesh.split("/", 1)
101
+ same = [r for r in pool
102
+ if r["source_category"] == category and r["source_model"] != model]
103
+ rng.shuffle(same)
104
+ chosen = same[: a.per_mesh]
105
+
106
+ if len(chosen) < a.per_mesh:
107
+ used = {(r["task"], r["episode"]) for r in chosen}
108
+ rest = [r for r in pool
109
+ if (r["task"], r["episode"]) not in used and r["source_model"] != model]
110
+ rng.shuffle(rest)
111
+ chosen += rest[: a.per_mesh - len(chosen)]
112
+
113
+ n_same = sum(1 for r in chosen if r["source_category"] == category)
114
+ summary[target_mesh] = dict(
115
+ requested=a.per_mesh, assigned=len(chosen),
116
+ same_category=n_same, random_fill=len(chosen) - n_same,
117
+ same_category_pool=len(same),
118
+ by_task=dict(sorted(Counter(r["task"] for r in chosen).items())),
119
+ )
120
+ for sample_idx, src in enumerate(chosen):
121
+ name = run_name_for(src, model)
122
+ assignments.append(dict(
123
+ assignment_idx=idx, sample_idx_for_target=sample_idx,
124
+ target_mesh=target_mesh, target_category=category, target_model=model,
125
+ source=src, case_dir=str(a.out_root / name), run_name=name,
126
+ # `provenance` is the key build_vace_objwise_gr00t_dataset.py filters on:
127
+ # it drops "random_fill" rows unless --include-random-fill is passed.
128
+ provenance=("existing" if src["source_category"] == category else "random_fill"),
129
+ fill=("same_category" if src["source_category"] == category else "random"),
130
+ ))
131
+ idx += 1
132
+
133
+ with (a.out_root / "assignments.jsonl").open("w", encoding="utf-8") as f:
134
+ for row in assignments:
135
+ f.write(json.dumps(row, ensure_ascii=False) + "\n")
136
+ (a.out_root / "assignment_summary.json").write_text(
137
+ json.dumps(summary, indent=2, ensure_ascii=False))
138
+
139
+ print(f"\n{'타깃':34s} {'배정':>5s} {'동일카테고리':>10s} {'랜덤보충':>8s} {'동일풀':>7s}")
140
+ for t, s in summary.items():
141
+ print(f" {t:32s} {s['assigned']:5d} {s['same_category']:10d} "
142
+ f"{s['random_fill']:8d} {s['same_category_pool']:7d}")
143
+ print(f"\n배정 {len(assignments)}건 -> {a.out_root / 'assignments.jsonl'}")
144
+ return 0
145
+
146
+
147
+ if __name__ == "__main__":
148
+ raise SystemExit(main())
augment/vace/collect_outputs.py ADDED
@@ -0,0 +1,78 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """Collect finished VACE plates into the MimicGen output layout.
3
+
4
+ VACE writes one stitched video per camera under
5
+
6
+ <case_dir>/work_<camera>/inference_gtmask_<tag>/out_video.mp4
7
+
8
+ which this copies to the same shape the MimicGen augmentation run uses:
9
+
10
+ <out>/<task>-ep<N>_<object the episode had>_<object it becomes>/<camera>/<camera>.mp4
11
+
12
+ plus a `.done` recording where the plate came from. An assignment whose three cameras are
13
+ not all present is skipped and listed at the end, so a partial WAN batch can be collected
14
+ and re-collected later without producing half-populated folders.
15
+
16
+ python collect_outputs.py --assignments DIR/assignments.jsonl --out DIR_OUT
17
+ """
18
+ from __future__ import annotations
19
+
20
+ import argparse
21
+ import glob
22
+ import json
23
+ import os
24
+ import shutil
25
+ from pathlib import Path
26
+
27
+ CAMERAS = ["robot0_agentview_left", "robot0_agentview_right", "robot0_eye_in_hand"]
28
+
29
+
30
+ def plate_for(case_dir: str, camera: str) -> str | None:
31
+ hits = sorted(glob.glob(os.path.join(case_dir, f"work_{camera}", "*", "out_video.mp4")))
32
+ return hits[0] if hits else None
33
+
34
+
35
+ def main() -> int:
36
+ ap = argparse.ArgumentParser()
37
+ ap.add_argument("--assignments", type=Path, required=True)
38
+ ap.add_argument("--out", type=Path, required=True)
39
+ ap.add_argument("--force", action="store_true")
40
+ a = ap.parse_args()
41
+
42
+ rows = [json.loads(l) for l in a.assignments.open()]
43
+ a.out.mkdir(parents=True, exist_ok=True)
44
+ done, missing = 0, []
45
+ for r in rows:
46
+ plates = {c: plate_for(r["case_dir"], c) for c in CAMERAS}
47
+ if any(v is None for v in plates.values()):
48
+ missing.append((r["run_name"], [c for c, v in plates.items() if v is None]))
49
+ continue
50
+ ep_dir = a.out / r["run_name"]
51
+ if (ep_dir / ".done").exists() and not a.force:
52
+ done += 1
53
+ continue
54
+ for c, src in plates.items():
55
+ d = ep_dir / c
56
+ d.mkdir(parents=True, exist_ok=True)
57
+ shutil.copy2(src, d / f"{c}.mp4")
58
+ src = r["source"]
59
+ (ep_dir / ".done").write_text(json.dumps(dict(
60
+ method="vace", fill=r.get("fill"),
61
+ source_task=src["task"], source_episode=src["episode"],
62
+ source_object=src["source_mesh"], target_object=r["target_mesh"],
63
+ lang=src.get("lang"), case_dir=r["case_dir"]), indent=1))
64
+ done += 1
65
+ print(f" {r['run_name']}", flush=True)
66
+
67
+ print(f"\n수집 {done} / {len(rows)}")
68
+ if missing:
69
+ print(f"미완 {len(missing)}건 (카메라 누락):")
70
+ for name, cams in missing[:20]:
71
+ print(f" {name}: {', '.join(c.replace('robot0_', '') for c in cams)}")
72
+ if len(missing) > 20:
73
+ print(f" ... 외 {len(missing) - 20}건")
74
+ return 0
75
+
76
+
77
+ if __name__ == "__main__":
78
+ raise SystemExit(main())
augment/vace/make_gt_masks.py ADDED
@@ -0,0 +1,179 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """Render the missing GT target-object mask videos straight from the simulator.
3
+
4
+ `prep-worker` needs `aug_materials/robocasa/<task>/ep<N>/mask_gt_target_object_<cam>.mp4`
5
+ for every source episode, and those only exist for 629 of the 2,517 episodes — which is
6
+ what caps the VACE batch. They are reproducible: every episode ships `model.xml.gz` and
7
+ `states.npz`, so restoring each frame and rendering a segmentation pass gives the same
8
+ mask. This writes them in the same shape as the delivered ones (256x256, 20 fps, h264,
9
+ white where the target object is).
10
+
11
+ The target is every geom whose name starts with `obj_`, matching
12
+ `vace_object_insert_batch.render_target_mask` (which only ever rendered frame 0, as a
13
+ visibility prefilter — this renders the whole episode).
14
+
15
+ # what is missing
16
+ python make_gt_masks.py --list
17
+
18
+ # render it, 8 workers, writing beside the delivered masks
19
+ python make_gt_masks.py --workers 8
20
+ python make_gt_masks.py --workers 8 --out-root /somewhere/else # or to a new tree
21
+ """
22
+ from __future__ import annotations
23
+
24
+ import argparse
25
+ import json
26
+ import os
27
+ import sys
28
+ from pathlib import Path
29
+
30
+ os.environ.setdefault("MUJOCO_GL", "egl")
31
+ os.environ.setdefault("PYOPENGL_PLATFORM", "egl")
32
+
33
+ import importlib.util
34
+ from concurrent.futures import ProcessPoolExecutor, as_completed
35
+
36
+ HERE = Path(__file__).resolve().parent
37
+ spec = importlib.util.spec_from_file_location(
38
+ "vace_batch", HERE.parent / "vace_object_insert_batch.py")
39
+ mg = importlib.util.module_from_spec(spec)
40
+ spec.loader.exec_module(mg)
41
+
42
+ CAMS = mg.CAMS
43
+ FPS = 20
44
+ RES = 256
45
+
46
+
47
+ def mask_path(root: Path, task: str, episode: int, cam: str) -> Path:
48
+ return root / task / f"ep{episode:06d}" / f"mask_gt_target_object_{cam}.mp4"
49
+
50
+
51
+ def episodes(dataset_root: Path):
52
+ for task_dir in sorted(p for p in dataset_root.iterdir() if p.is_dir()):
53
+ extras = task_dir / "extras"
54
+ if not extras.exists():
55
+ continue
56
+ for ep_dir in sorted(extras.glob("episode_*")):
57
+ try:
58
+ yield task_dir.name, int(ep_dir.name.split("_")[-1]), ep_dir
59
+ except ValueError:
60
+ continue
61
+
62
+
63
+ def missing(dataset_root: Path, out_root: Path, read_root: Path, categories=None):
64
+ out = []
65
+ for task, ep, ep_dir in episodes(dataset_root):
66
+ if categories:
67
+ info = mg.target_obj_info(ep_dir)
68
+ if not info or info["category"] not in categories:
69
+ continue
70
+ need = [c for c in CAMS
71
+ if not mask_path(read_root, task, ep, c).exists()
72
+ and not (read_root / task / f"ep{ep:06d}" / "encoded_videos"
73
+ / f"mask_gt_target_object_{c}.mp4").exists()
74
+ and not mask_path(out_root, task, ep, c).exists()]
75
+ if need:
76
+ out.append((task, ep, str(ep_dir), need))
77
+ return out
78
+
79
+
80
+ def render_one(job):
81
+ task, ep, ep_dir, cams, out_root = job
82
+ import cv2
83
+ import mujoco
84
+ import numpy as np
85
+
86
+ try:
87
+ sim, states = mg.sim_from_episode(Path(ep_dir))
88
+ model, data = sim.model._model, sim.data._data
89
+ target = {gid for gid in range(model.ngeom)
90
+ if (mujoco.mj_id2name(model, mujoco.mjtObj.mjOBJ_GEOM, gid) or "").startswith("obj_")}
91
+ if not target:
92
+ return dict(task=task, episode=ep, status="failed", error="no obj_ geoms")
93
+ opt = mujoco.MjvOption()
94
+ opt.sitegroup[:] = 0
95
+ renderer = mujoco.Renderer(model, height=RES, width=RES)
96
+ renderer.enable_segmentation_rendering()
97
+ written = {}
98
+ try:
99
+ for cam in cams:
100
+ cid = mujoco.mj_name2id(model, mujoco.mjtObj.mjOBJ_CAMERA, cam)
101
+ if cid < 0:
102
+ return dict(task=task, episode=ep, status="failed", error=f"no camera {cam}")
103
+ dst = mask_path(Path(out_root), task, ep, cam)
104
+ dst.parent.mkdir(parents=True, exist_ok=True)
105
+ vw = cv2.VideoWriter(str(dst), cv2.VideoWriter_fourcc(*"mp4v"),
106
+ FPS, (RES, RES), True)
107
+ area = 0
108
+ for t in range(len(states)):
109
+ sim.set_state_from_flattened(states[t])
110
+ sim.forward()
111
+ renderer.update_scene(data, camera=cid, scene_option=opt)
112
+ seg = mg.render_segmentation_safe(renderer)
113
+ m = (np.isin(seg[..., 0], list(target)).astype(np.uint8) * 255)
114
+ area = max(area, int((m > 127).sum()))
115
+ vw.write(np.repeat(m[:, :, None], 3, axis=2))
116
+ vw.release()
117
+ written[cam] = dict(frames=len(states), max_area=area)
118
+ finally:
119
+ renderer.close()
120
+ return dict(task=task, episode=ep, status="ok", frames=len(states), cams=written)
121
+ except Exception as e: # one bad episode must not take the batch down
122
+ return dict(task=task, episode=ep, status="failed", error=f"{type(e).__name__}: {e}")
123
+
124
+
125
+ def main() -> int:
126
+ ap = argparse.ArgumentParser()
127
+ ap.add_argument("--dataset-root", type=Path, default=mg.ROBOCASA_ROOT)
128
+ ap.add_argument("--read-root", type=Path, default=mg.AUG_MATERIALS_ROOT,
129
+ help="where the delivered masks live (checked, never written)")
130
+ ap.add_argument("--out-root", type=Path, default=mg.AUG_MATERIALS_ROOT,
131
+ help="where new masks are written; default = beside the delivered ones")
132
+ ap.add_argument("--workers", type=int, default=8)
133
+ ap.add_argument("--limit", type=int, default=None)
134
+ ap.add_argument("--list", action="store_true", help="only report what is missing")
135
+ ap.add_argument("--categories", nargs="*", default=None,
136
+ help="only episodes whose own object is one of these categories -- the "
137
+ "others can never serve as a same-category source")
138
+ ap.add_argument("--log", type=Path, default=None)
139
+ a = ap.parse_args()
140
+
141
+ todo = missing(a.dataset_root, a.out_root, a.read_root,
142
+ set(a.categories) if a.categories else None)
143
+ from collections import Counter
144
+ per = Counter(t for t, _, _, _ in todo)
145
+ total = sum(1 for _ in episodes(a.dataset_root))
146
+ print(f"에피소드 {total}개 중 마스크 미보유 {len(todo)}개")
147
+ for t, n in sorted(per.items()):
148
+ print(f" {t:30s} {n:5d}")
149
+ if a.list:
150
+ return 0
151
+ if a.limit:
152
+ todo = todo[: a.limit]
153
+ if not todo:
154
+ return 0
155
+
156
+ jobs = [(t, e, d, cams, str(a.out_root)) for t, e, d, cams in todo]
157
+ log = open(a.log, "w") if a.log else None
158
+ ok = fail = 0
159
+ with ProcessPoolExecutor(max_workers=a.workers) as ex:
160
+ futs = {ex.submit(render_one, j): j for j in jobs}
161
+ for i, fut in enumerate(as_completed(futs), 1):
162
+ r = fut.result()
163
+ ok += r["status"] == "ok"
164
+ fail += r["status"] == "failed"
165
+ if log:
166
+ log.write(json.dumps(r) + "\n")
167
+ log.flush()
168
+ if r["status"] == "failed":
169
+ print(f" FAIL {r['task']}/ep{r['episode']:06d}: {r['error'][:90]}", flush=True)
170
+ if i % 25 == 0 or i == len(jobs):
171
+ print(f" {i}/{len(jobs)} ok={ok} fail={fail}", flush=True)
172
+ if log:
173
+ log.close()
174
+ print(f"\n완료: ok {ok}, failed {fail}")
175
+ return 0 if fail == 0 else 1
176
+
177
+
178
+ if __name__ == "__main__":
179
+ raise SystemExit(main())
augment/vace/preflight.sh ADDED
@@ -0,0 +1,60 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env bash
2
+ # Check everything run_aug32.sh needs before committing a multi-hour batch to a GPU.
3
+ # Exits non-zero if anything required is missing.
4
+ set -uo pipefail
5
+
6
+ HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
7
+ VACE="$(cd "$HERE/.." && pwd)"
8
+ fail=0
9
+
10
+ ok() { printf ' \033[32mOK \033[0m %s\n' "$1"; }
11
+ bad() { printf ' \033[31mMISS\033[0m %s\n' "$1"; fail=1; }
12
+ have() { [ -e "$1" ] && ok "$2" || bad "$2 ($1)"; }
13
+ runs() { [ -x "$1" ] && ok "$2 ($("$1" -V 2>&1))" || bad "$2 ($1)"; }
14
+
15
+ echo "환경 (가상환경 — 새로 만들 필요 없음)"
16
+ runs "${VACE_PY_ROBOCASA:-/lp-dev/jonghoon/robocasa_calib/envs/robocasa/bin/python}" "batch driver (robocasa, mujoco)"
17
+ runs "${VACE_PY_WAN:-/lp-dev/jonghoon/VACE/venv/bin/python}" "WAN inference (torch+flash_attn)"
18
+ have "${VACE_RENDER_SCRIPT_DIR:-$HERE/vendor}/render_mesh_views_side.py" "render_mesh_views_side.py (vendored)"
19
+ "${VACE_RENDER_PYTHON:-${VACE_PY_ROBOCASA:-/lp-dev/jonghoon/robocasa_calib/envs/robocasa/bin/python}}" \
20
+ -c "import trimesh,pyrender" 2>/dev/null && ok "trimesh + pyrender (mesh view rendering)" \
21
+ || bad "trimesh + pyrender missing from the render python"
22
+
23
+ echo
24
+ echo "VACE 코드와 가중치"
25
+ have "$VACE/VACE/vace/vace_wan_inference.py" "VACE fork (ali-vilab/VACE + batch flags)"
26
+ have "$VACE/VACE/models/Wan2.1-VACE-1.3B" "Wan2.1-VACE-1.3B weights (via models symlink)"
27
+ have "$VACE/VACE/models/VACE-Annotators" "VACE-Annotators"
28
+ have "$VACE/vace_object_insert_batch.py" "batch driver"
29
+
30
+ echo
31
+ echo "데이터셋"
32
+ have "${VACE_ROBOCASA_ROOT:-/home/nvidia/jonghoon/robocasa_full/pickplace_target_human}" "소스 에피소드 (영상 + extras)"
33
+ have "${VACE_AUG_MATERIALS:-/home/nvidia/jonghoon/robocasa_full/aug_materials/robocasa}" "GT target-object 마스크"
34
+ for r in $(echo "${VACE_ASSET_ROOTS:-/lp-dev/jonghoon/robocasa_calib/repos/robocasa/robocasa/models/assets/objects/lightwheel:/lp-dev/jonghoon/robocasa_calib/repos/robocasa/robocasa/models/assets/objects/objaverse}" | tr ":" " "); do
35
+ have "$r" "메쉬 $(basename "$r")"
36
+ done
37
+
38
+ echo
39
+ echo "GT 마스크 커버리지 (prep-worker 가 요구 — 없으면 그 에피소드는 실패)"
40
+ A="${VACE_AUG_MATERIALS:-/home/nvidia/jonghoon/robocasa_full/aug_materials/robocasa}"
41
+ for t in PickPlaceCounterToCabinet PickPlaceCounterToStove PickPlaceDrawerToCounter \
42
+ PickPlaceSinkToCounter PickPlaceToasterToCounter; do
43
+ n=0
44
+ for d in "$A/$t"/ep*; do
45
+ [ -f "$d/mask_gt_target_object_robot0_agentview_left.mp4" ] &&
46
+ [ -f "$d/mask_gt_target_object_robot0_agentview_right.mp4" ] &&
47
+ [ -f "$d/mask_gt_target_object_robot0_eye_in_hand.mp4" ] && n=$((n+1))
48
+ done 2>/dev/null
49
+ printf ' %-30s %4d episodes\n' "$t" "$n"
50
+ done
51
+
52
+ echo
53
+ echo "GPU"
54
+ nvidia-smi --query-gpu=index,memory.used,memory.total,utilization.gpu --format=csv,noheader 2>/dev/null |
55
+ awk -F', ' '{printf " gpu %s %s / %s util %s\n",$1,$2,$3,$4}'
56
+ echo " 필요: 약 27 GB (Wan2.1-VACE-1.3B, --offload_model False 기준)"
57
+
58
+ echo
59
+ [ $fail -eq 0 ] && echo "모두 준비됨." || echo "누락 항목 있음 — 위 MISS 확인."
60
+ exit $fail
augment/vace/run_aug32.sh ADDED
@@ -0,0 +1,67 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env bash
2
+ # VACE augmentation: 8 target objects x 32 episodes, three cameras each.
3
+ #
4
+ # ./run_aug32.sh <work_dir> <out_dir> <gpu> [num_wan_workers]
5
+ #
6
+ # Stages, in order. Each is idempotent, so re-running after an interruption resumes.
7
+ # 1 build same-category sources first, random episodes for the remainder
8
+ # 2 prep render the target mesh's reference views, cut frames, build per-view manifests
9
+ # 3 split chop into 81-frame WAN jobs
10
+ # 4 wan the diffusion pass -- the only slow stage
11
+ # 5 stitch glue each camera's chunks back into one video
12
+ # 6 collect copy into <out_dir>/<ep>_<src obj>_<target>/<camera>/<camera>.mp4
13
+ set -euo pipefail
14
+
15
+ HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
16
+ VACE="$(cd "$HERE/.." && pwd)"
17
+
18
+ WORK="${1:?usage: run_aug32.sh <work_dir> <out_dir> <gpu> [num_wan_workers]}"
19
+ OUT="${2:?usage: run_aug32.sh <work_dir> <out_dir> <gpu> [num_wan_workers]}"
20
+ GPU="${3:?usage: run_aug32.sh <work_dir> <out_dir> <gpu> [num_wan_workers]}"
21
+ WORKERS="${4:-1}"
22
+
23
+ RC="${VACE_PY_ROBOCASA:-/lp-dev/jonghoon/robocasa_calib/envs/robocasa/bin/python}"
24
+ VP="${VACE_PY_WAN:-/lp-dev/jonghoon/VACE/venv/bin/python}"
25
+ export MUJOCO_GL=egl PYOPENGL_PLATFORM=egl
26
+
27
+ # Data paths come from the environment; unset means "the original machine's path".
28
+ # On any other box unpack ../datasets_pack and `export` these first -- see README "Porting":
29
+ # VACE_ROBOCASA_ROOT VACE_AUG_MATERIALS VACE_ASSET_ROOTS
30
+ # VACE_PY_ROBOCASA VACE_PY_WAN (interpreters)
31
+
32
+ mkdir -p "$WORK"
33
+
34
+ echo "== 1/6 build =================================================="
35
+ "$RC" "$HERE/build_aug32_assignments.py" --out-root "$WORK" --per-mesh 32
36
+
37
+ echo "== 2/6 prep (gpu $GPU) ========================================"
38
+ cd "$VACE"
39
+ "$RC" vace_object_insert_batch.py prep-worker \
40
+ --assignments "$WORK/assignments.jsonl" --out-root "$WORK" \
41
+ --worker-id 0 --num-workers 1 --gpu "$GPU" | tee "$WORK/prep.log" | \
42
+ grep -c '"status": "ok"' | xargs -I{} echo " prepared views: {}"
43
+
44
+ echo "== 3/6 split =================================================="
45
+ "$RC" vace_object_insert_batch.py split-wan \
46
+ --assignments "$WORK/assignments.jsonl" --out-root "$WORK" --num-workers "$WORKERS"
47
+
48
+ echo "== 4/6 wan (gpu $GPU) ========================================="
49
+ # --offload_model False keeps the DiT resident; the default True costs ~30 s per job
50
+ cd "$VACE/VACE"
51
+ for w in $(seq -f "%02g" 0 $((WORKERS - 1))); do
52
+ CUDA_VISIBLE_DEVICES="$GPU" "$VP" vace/vace_wan_inference.py \
53
+ --model_name vace-1.3B --ckpt_dir models/Wan2.1-VACE-1.3B \
54
+ --batch_manifest "$WORK/batch_manifests/wan_worker_$w.jsonl" \
55
+ --offload_model False --skip_existing true --batch_continue_on_error true \
56
+ > "$WORK/wan_$w.log" 2>&1 &
57
+ done
58
+ wait
59
+
60
+ echo "== 5/6 stitch ================================================="
61
+ cd "$VACE"
62
+ "$RC" vace_object_insert_batch.py stitch-segments \
63
+ --assignments "$WORK/assignments.jsonl" --out-root "$WORK"
64
+
65
+ echo "== 6/6 collect ================================================"
66
+ "$RC" "$HERE/collect_outputs.py" --assignments "$WORK/assignments.jsonl" --out "$OUT"
67
+ echo "done -> $OUT"
dataset/build_vace_objwise_gr00t_dataset.py ADDED
@@ -0,0 +1,509 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """Build a GR00T/LeRobot dataset from object-wise VACE augmentation videos.
3
+
4
+ The VACE output stores one assignment directory per source trajectory and target
5
+ mesh, with three final view videos under work_<view>/<inference_dir>/out_video.mp4.
6
+ This builder keeps the original source state/action parquet rows, rewrites only
7
+ episode/global indices and task ids, links the VACE videos into the LeRobot video
8
+ layout, and creates one side-by-side three_view.mp4 preview per augmented episode.
9
+ """
10
+
11
+ from __future__ import annotations
12
+
13
+ import argparse
14
+ import json
15
+ import math
16
+ import os
17
+ import shutil
18
+ import statistics
19
+ import subprocess
20
+ import time
21
+ from collections import Counter, defaultdict
22
+ from datetime import datetime, timezone
23
+ from pathlib import Path
24
+ from typing import Any
25
+
26
+ import numpy as np
27
+ import pandas as pd
28
+
29
+
30
+ TASKS = (
31
+ "PickPlaceCounterToCabinet",
32
+ "PickPlaceCounterToStove",
33
+ "PickPlaceDrawerToCounter",
34
+ "PickPlaceSinkToCounter",
35
+ "PickPlaceToasterToCounter",
36
+ )
37
+
38
+ VIEW_KEYS = (
39
+ "robot0_agentview_left",
40
+ "robot0_agentview_right",
41
+ "robot0_eye_in_hand",
42
+ )
43
+
44
+ VIDEO_FEATURES = {
45
+ "robot0_agentview_left": "observation.images.robot0_agentview_left",
46
+ "robot0_agentview_right": "observation.images.robot0_agentview_right",
47
+ "robot0_eye_in_hand": "observation.images.robot0_eye_in_hand",
48
+ }
49
+
50
+ TASK_COLUMNS = (
51
+ "task_index",
52
+ "annotation.human.task_description",
53
+ "annotation.human.task_name",
54
+ )
55
+
56
+
57
+ def parse_args() -> argparse.Namespace:
58
+ parser = argparse.ArgumentParser(description=__doc__)
59
+ parser.add_argument(
60
+ "--vace-root",
61
+ type=Path,
62
+ default=Path("/home/nvidia/jonghoon/robocasa_full/augmented_video/groot1.5_objwise_vace"),
63
+ )
64
+ parser.add_argument(
65
+ "--source-root",
66
+ type=Path,
67
+ default=Path("/home/nvidia/jonghoon/robocasa_full/pickplace_target_human"),
68
+ )
69
+ parser.add_argument(
70
+ "--canonical-meta-source",
71
+ type=Path,
72
+ default=Path("/home/nvidia/jonghoon/robocasa_full/gr00t_views/PickPlaceCounterToCabinet_all_502eps"),
73
+ help="Dataset whose GR00T-compatible info/modality/view_manifest contract is copied.",
74
+ )
75
+ parser.add_argument(
76
+ "--manifest",
77
+ type=Path,
78
+ action="append",
79
+ default=None,
80
+ help="Assignment JSONL. Defaults to assignments_target16.jsonl and assignments_teapot6_extra.jsonl.",
81
+ )
82
+ parser.add_argument("--output", type=Path, required=True)
83
+ parser.add_argument(
84
+ "--include-random-fill",
85
+ action="store_true",
86
+ help="Include assignments whose provenance is random_fill. Default keeps only clean same-category assignments.",
87
+ )
88
+ parser.add_argument(
89
+ "--copy-videos",
90
+ action="store_true",
91
+ help="Copy per-view VACE videos instead of symlinking them.",
92
+ )
93
+ parser.add_argument("--force", action="store_true")
94
+ parser.add_argument(
95
+ "--skip-three-view",
96
+ action="store_true",
97
+ help="Do not render extras/episode_xxxxxx/three_view.mp4 previews.",
98
+ )
99
+ return parser.parse_args()
100
+
101
+
102
+ def load_json(path: Path) -> dict[str, Any]:
103
+ return json.loads(path.read_text())
104
+
105
+
106
+ def load_jsonl(path: Path) -> list[dict[str, Any]]:
107
+ with path.open() as f:
108
+ return [json.loads(line) for line in f if line.strip()]
109
+
110
+
111
+ def write_jsonl(path: Path, rows: list[dict[str, Any]]) -> None:
112
+ path.parent.mkdir(parents=True, exist_ok=True)
113
+ with path.open("w") as f:
114
+ for row in rows:
115
+ f.write(json.dumps(row, sort_keys=False) + "\n")
116
+
117
+
118
+ def copy_or_link(src: Path, dst: Path, copy: bool = False) -> None:
119
+ if not src.exists():
120
+ raise FileNotFoundError(src)
121
+ dst.parent.mkdir(parents=True, exist_ok=True)
122
+ if dst.exists() or dst.is_symlink():
123
+ if dst.is_dir() and not dst.is_symlink():
124
+ shutil.rmtree(dst)
125
+ else:
126
+ dst.unlink()
127
+ if copy:
128
+ shutil.copy2(src, dst)
129
+ else:
130
+ dst.symlink_to(src.resolve())
131
+
132
+
133
+ def source_episode_path(dataset_root: Path, episode_index: int, chunks_size: int) -> Path:
134
+ chunk = episode_index // chunks_size
135
+ return dataset_root / "data" / f"chunk-{chunk:03d}" / f"episode_{episode_index:06d}.parquet"
136
+
137
+
138
+ def load_source_catalog(source_root: Path) -> dict[str, dict[str, Any]]:
139
+ catalog: dict[str, dict[str, Any]] = {}
140
+ for task in TASKS:
141
+ root = source_root / task
142
+ if not root.exists():
143
+ continue
144
+ meta = root / "meta"
145
+ info = load_json(meta / "info.json")
146
+ catalog[task] = {
147
+ "root": root,
148
+ "meta": meta,
149
+ "info": info,
150
+ "chunks_size": int(info.get("chunks_size", 1000)),
151
+ "episodes": {int(r["episode_index"]): r for r in load_jsonl(meta / "episodes.jsonl")},
152
+ "episode_stats": (
153
+ {int(r["episode_index"]): r for r in load_jsonl(meta / "episodes_stats.jsonl")}
154
+ if (meta / "episodes_stats.jsonl").exists()
155
+ else {}
156
+ ),
157
+ "tasks": {int(r["task_index"]): r["task"] for r in load_jsonl(meta / "tasks.jsonl")},
158
+ }
159
+ return catalog
160
+
161
+
162
+ def default_manifests(vace_root: Path) -> list[Path]:
163
+ manifests = [vace_root / "assignments_target16.jsonl"]
164
+ teapot = vace_root / "assignments_teapot6_extra.jsonl"
165
+ if teapot.exists():
166
+ manifests.append(teapot)
167
+ return manifests
168
+
169
+
170
+ def load_assignments(paths: list[Path], include_random_fill: bool) -> list[dict[str, Any]]:
171
+ rows: list[dict[str, Any]] = []
172
+ for path in paths:
173
+ for row in load_jsonl(path):
174
+ row = dict(row)
175
+ row.setdefault("assignment_manifest", str(path))
176
+ row.setdefault("provenance", "existing")
177
+ if row["provenance"] == "random_fill" and not include_random_fill:
178
+ continue
179
+ rows.append(row)
180
+ rows.sort(key=lambda r: (str(r.get("target_mesh", "")), int(r.get("assignment_idx", 10**9)), str(r.get("case_dir", ""))))
181
+ return rows
182
+
183
+
184
+ def final_view_video(case_dir: Path, view: str) -> tuple[Path, int | None, dict[str, Any]]:
185
+ work = case_dir / f"work_{view}"
186
+ manifest_path = work / "manifest.json"
187
+ manifest: dict[str, Any] = {}
188
+ if manifest_path.exists():
189
+ manifest = load_json(manifest_path)
190
+ inference_dir = manifest.get("inference_save_dir")
191
+ if inference_dir:
192
+ out = Path(inference_dir) / "out_video.mp4"
193
+ if out.exists():
194
+ return out, manifest.get("video_info", {}).get("nb_frames"), manifest
195
+ candidates = [p for p in work.glob("*/out_video.mp4") if "/segments/" not in str(p)]
196
+ if not candidates:
197
+ raise FileNotFoundError(f"No final out_video.mp4 for {case_dir} view={view}")
198
+ candidates.sort(key=lambda p: p.stat().st_mtime, reverse=True)
199
+ return candidates[0], manifest.get("video_info", {}).get("nb_frames"), manifest
200
+
201
+
202
+ def create_three_view(left: Path, right: Path, wrist: Path, out: Path) -> None:
203
+ out.parent.mkdir(parents=True, exist_ok=True)
204
+ tmp = out.with_suffix(".tmp.mp4")
205
+ if tmp.exists():
206
+ tmp.unlink()
207
+ cmd = [
208
+ "ffmpeg",
209
+ "-y",
210
+ "-loglevel",
211
+ "error",
212
+ "-i",
213
+ str(left),
214
+ "-i",
215
+ str(right),
216
+ "-i",
217
+ str(wrist),
218
+ "-filter_complex",
219
+ "[0:v]scale=256:256[v0];[1:v]scale=256:256[v1];[2:v]scale=256:256[v2];[v0][v1][v2]hstack=inputs=3[v]",
220
+ "-map",
221
+ "[v]",
222
+ "-an",
223
+ "-c:v",
224
+ "libx264",
225
+ "-preset",
226
+ "veryfast",
227
+ "-crf",
228
+ "18",
229
+ "-pix_fmt",
230
+ "yuv420p",
231
+ "-shortest",
232
+ str(tmp),
233
+ ]
234
+ subprocess.run(cmd, check=True)
235
+ tmp.replace(out)
236
+
237
+
238
+ def remap_episode_stats(row: dict[str, Any], new_episode_index: int, global_start: int, length: int) -> dict[str, Any]:
239
+ row = json.loads(json.dumps(row))
240
+ row["episode_index"] = new_episode_index
241
+ stats = row.setdefault("stats", {})
242
+ stats["episode_index"] = {
243
+ "min": [new_episode_index],
244
+ "max": [new_episode_index],
245
+ "mean": [float(new_episode_index)],
246
+ "std": [0.0],
247
+ "count": [length],
248
+ }
249
+ stats["index"] = {
250
+ "min": [global_start],
251
+ "max": [global_start + length - 1],
252
+ "mean": [global_start + (length - 1) / 2.0],
253
+ "std": [statistics.pstdev(range(global_start, global_start + length))],
254
+ "count": [length],
255
+ }
256
+ return row
257
+
258
+
259
+ def column_values_to_matrix(values: pd.Series) -> np.ndarray:
260
+ return np.vstack([np.asarray(v, dtype=np.float64).reshape(1, -1) for v in values])
261
+
262
+
263
+ def calculate_stats(parquet_paths: list[Path]) -> dict[str, dict[str, list[float]]]:
264
+ accum: dict[str, list[np.ndarray]] = defaultdict(list)
265
+ for path in parquet_paths:
266
+ df = pd.read_parquet(path)
267
+ for column in df.columns:
268
+ accum[column].append(column_values_to_matrix(df[column]))
269
+ stats: dict[str, dict[str, list[float]]] = {}
270
+ for column, chunks in accum.items():
271
+ values = np.concatenate(chunks, axis=0)
272
+ stats[column] = {
273
+ "mean": np.mean(values, axis=0).tolist(),
274
+ "std": np.std(values, axis=0).tolist(),
275
+ "min": np.min(values, axis=0).tolist(),
276
+ "max": np.max(values, axis=0).tolist(),
277
+ "q01": np.quantile(values, 0.01, axis=0).tolist(),
278
+ "q99": np.quantile(values, 0.99, axis=0).tolist(),
279
+ }
280
+ return stats
281
+
282
+
283
+ def main() -> None:
284
+ args = parse_args()
285
+ manifests = args.manifest or default_manifests(args.vace_root)
286
+ assignments = load_assignments(manifests, include_random_fill=args.include_random_fill)
287
+ if not assignments:
288
+ raise RuntimeError("No assignments selected")
289
+
290
+ if args.output.exists():
291
+ if not args.force:
292
+ raise FileExistsError(f"{args.output} exists; pass --force")
293
+ backup = args.output.with_name(f"{args.output.name}.bak_{time.strftime('%Y%m%d_%H%M%S')}")
294
+ args.output.rename(backup)
295
+ print(f"Moved existing output to {backup}")
296
+
297
+ source_catalog = load_source_catalog(args.source_root)
298
+ canonical_meta = args.canonical_meta_source / "meta"
299
+ out_meta = args.output / "meta"
300
+ out_data = args.output / "data" / "chunk-000"
301
+ out_videos = args.output / "videos" / "chunk-000"
302
+ out_meta.mkdir(parents=True, exist_ok=True)
303
+ out_data.mkdir(parents=True, exist_ok=True)
304
+
305
+ global_task_id: dict[str, int] = {}
306
+ global_tasks: list[dict[str, Any]] = []
307
+
308
+ def task_id(task_text: str) -> int:
309
+ if task_text not in global_task_id:
310
+ idx = len(global_task_id)
311
+ global_task_id[task_text] = idx
312
+ global_tasks.append({"task_index": idx, "task": task_text})
313
+ return global_task_id[task_text]
314
+
315
+ new_episodes: list[dict[str, Any]] = []
316
+ new_episode_stats: list[dict[str, Any]] = []
317
+ dataset_manifest: list[dict[str, Any]] = []
318
+ parquet_paths: list[Path] = []
319
+ source_counts: Counter[str] = Counter()
320
+ provenance_counts: Counter[str] = Counter()
321
+ global_index = 0
322
+
323
+ for new_ep, row in enumerate(assignments):
324
+ source = row["source"]
325
+ source_task = source["task"]
326
+ source_episode = int(source["episode"])
327
+ if source_task not in source_catalog:
328
+ raise KeyError(f"Unknown source task {source_task} for {row.get('case_dir')}")
329
+ src = source_catalog[source_task]
330
+ src_parquet = source_episode_path(src["root"], source_episode, src["chunks_size"])
331
+ if not src_parquet.exists():
332
+ raise FileNotFoundError(src_parquet)
333
+ source_episode_meta = src["episodes"][source_episode]
334
+ df = pd.read_parquet(src_parquet).copy()
335
+ length = len(df)
336
+ if int(source_episode_meta["length"]) != length:
337
+ raise RuntimeError(f"Source length mismatch for {src_parquet}: parquet={length} meta={source_episode_meta['length']}")
338
+
339
+ case_dir = Path(row["case_dir"])
340
+ view_outputs: dict[str, str] = {}
341
+ view_frame_counts: dict[str, int | None] = {}
342
+ view_manifests: dict[str, Any] = {}
343
+ for view in VIEW_KEYS:
344
+ video, frame_count, view_manifest = final_view_video(case_dir, view)
345
+ if frame_count is not None and int(frame_count) != length:
346
+ raise RuntimeError(f"Frame mismatch for {case_dir} {view}: vace={frame_count} parquet={length}")
347
+ feature = VIDEO_FEATURES[view]
348
+ dst = out_videos / feature / f"episode_{new_ep:06d}.mp4"
349
+ copy_or_link(video, dst, copy=args.copy_videos)
350
+ view_outputs[view] = str(dst)
351
+ view_frame_counts[view] = int(frame_count) if frame_count is not None else None
352
+ view_manifests[view] = view_manifest
353
+
354
+ if "episode_index" in df.columns:
355
+ df["episode_index"] = np.full(length, new_ep, dtype=np.int64)
356
+ if "frame_index" in df.columns:
357
+ df["frame_index"] = np.arange(length, dtype=np.int64)
358
+ if "index" in df.columns:
359
+ df["index"] = np.arange(global_index, global_index + length, dtype=np.int64)
360
+
361
+ for column in TASK_COLUMNS:
362
+ if column in df.columns:
363
+ mapping = {}
364
+ for old_idx in sorted(pd.unique(df[column])):
365
+ old_idx_int = int(old_idx)
366
+ try:
367
+ mapping[old_idx_int] = task_id(src["tasks"][old_idx_int])
368
+ except KeyError as exc:
369
+ raise KeyError(f"{src['root']} has no task index {old_idx_int} for column {column}") from exc
370
+ df[column] = df[column].map(lambda v: mapping[int(v)]).astype("int64")
371
+
372
+ dst_parquet = out_data / f"episode_{new_ep:06d}.parquet"
373
+ df.to_parquet(dst_parquet, index=False)
374
+ parquet_paths.append(dst_parquet)
375
+
376
+ new_episode = {
377
+ "episode_index": new_ep,
378
+ "tasks": source_episode_meta.get("tasks", []),
379
+ "length": length,
380
+ }
381
+ new_episodes.append(new_episode)
382
+ if source_episode in src["episode_stats"]:
383
+ new_episode_stats.append(remap_episode_stats(src["episode_stats"][source_episode], new_ep, global_index, length))
384
+
385
+ extras_dir = args.output / "extras" / f"episode_{new_ep:06d}"
386
+ extras_dir.mkdir(parents=True, exist_ok=True)
387
+ if not args.skip_three_view:
388
+ create_three_view(
389
+ Path(view_outputs["robot0_agentview_left"]),
390
+ Path(view_outputs["robot0_agentview_right"]),
391
+ Path(view_outputs["robot0_eye_in_hand"]),
392
+ extras_dir / "three_view.mp4",
393
+ )
394
+
395
+ compact_row = {
396
+ k: v
397
+ for k, v in row.items()
398
+ if k
399
+ in {
400
+ "assignment_idx",
401
+ "sample_idx_for_target",
402
+ "target_mesh",
403
+ "target_category",
404
+ "target_model",
405
+ "case_dir",
406
+ "run_name",
407
+ "provenance",
408
+ "assignment_manifest",
409
+ }
410
+ }
411
+ dataset_manifest.append(
412
+ {
413
+ "episode_index": new_ep,
414
+ "source_task": source_task,
415
+ "source_episode_index": source_episode,
416
+ "source_length": length,
417
+ "global_index_start": global_index,
418
+ "global_index_end": global_index + length - 1,
419
+ "source": source,
420
+ "assignment": compact_row,
421
+ "vace_case_dir": str(case_dir),
422
+ "vace_view_frame_counts": view_frame_counts,
423
+ "videos": {
424
+ VIDEO_FEATURES[view]: view_outputs[view]
425
+ for view in VIEW_KEYS
426
+ },
427
+ "three_view": str(extras_dir / "three_view.mp4") if not args.skip_three_view else None,
428
+ }
429
+ )
430
+
431
+ source_counts[f"{source_task}/episode_{source_episode:06d}"] += 1
432
+ provenance_counts[str(row.get("provenance", "existing"))] += 1
433
+ global_index += length
434
+
435
+ write_jsonl(out_meta / "episodes.jsonl", new_episodes)
436
+ write_jsonl(out_meta / "tasks.jsonl", global_tasks)
437
+ if new_episode_stats:
438
+ write_jsonl(out_meta / "episodes_stats.jsonl", new_episode_stats)
439
+
440
+ # Copy the GR00T-compatible embodiment/modality contract, not the raw
441
+ # RoboCasa view naming contract.
442
+ for name in ("modality.json", "view_manifest.json"):
443
+ src = canonical_meta / name
444
+ if src.exists():
445
+ shutil.copy2(src, out_meta / name)
446
+ raw_embodiment = args.source_root / "PickPlaceCounterToCabinet" / "meta" / "embodiment.json"
447
+ if raw_embodiment.exists():
448
+ shutil.copy2(raw_embodiment, out_meta / "embodiment.json")
449
+
450
+ info = load_json(canonical_meta / "info.json")
451
+ chunks_size = int(info.get("chunks_size", 1000))
452
+ video_features = [k for k, v in info.get("features", {}).items() if v.get("dtype") == "video"]
453
+ info["total_episodes"] = len(new_episodes)
454
+ info["total_frames"] = global_index
455
+ info["total_videos"] = len(new_episodes) * len(video_features)
456
+ info["total_chunks"] = max(1, math.ceil(len(new_episodes) / chunks_size))
457
+ info["total_tasks"] = len(global_tasks)
458
+ info["splits"] = {"train": f"0:{len(new_episodes)}"}
459
+ info["created_from_vace_objectwise"] = {
460
+ "created_at": datetime.now(timezone.utc).isoformat(),
461
+ "vace_root": str(args.vace_root),
462
+ "manifests": [str(p) for p in manifests],
463
+ "source_root": str(args.source_root),
464
+ "canonical_meta_source": str(args.canonical_meta_source),
465
+ "include_random_fill": bool(args.include_random_fill),
466
+ "episode_count": len(new_episodes),
467
+ "provenance_counts": dict(sorted(provenance_counts.items())),
468
+ }
469
+ (out_meta / "info.json").write_text(json.dumps(info, indent=2) + "\n")
470
+
471
+ stats = calculate_stats(parquet_paths)
472
+ (out_meta / "stats.json").write_text(json.dumps(stats, indent=2) + "\n")
473
+ (out_meta / "relative_stats.json").write_text("{}\n")
474
+ (out_meta / "vace_objwise_dataset_manifest.json").write_text(
475
+ json.dumps(
476
+ {
477
+ "created_at": datetime.now(timezone.utc).isoformat(),
478
+ "output": str(args.output),
479
+ "episode_count": len(new_episodes),
480
+ "total_frames": global_index,
481
+ "include_random_fill": bool(args.include_random_fill),
482
+ "provenance_counts": dict(sorted(provenance_counts.items())),
483
+ "target_mesh_counts": dict(sorted(Counter(r["assignment"]["target_mesh"] for r in dataset_manifest).items())),
484
+ "source_episode_counts": dict(sorted(source_counts.items())),
485
+ "episodes": dataset_manifest,
486
+ },
487
+ indent=2,
488
+ )
489
+ + "\n"
490
+ )
491
+
492
+ print(
493
+ json.dumps(
494
+ {
495
+ "output": str(args.output),
496
+ "episodes": len(new_episodes),
497
+ "frames": global_index,
498
+ "videos_per_view": len(new_episodes),
499
+ "three_view_videos": 0 if args.skip_three_view else len(new_episodes),
500
+ "provenance_counts": dict(sorted(provenance_counts.items())),
501
+ "total_tasks": len(global_tasks),
502
+ },
503
+ indent=2,
504
+ )
505
+ )
506
+
507
+
508
+ if __name__ == "__main__":
509
+ main()
dataset/merge_gr00t_view_datasets.py ADDED
@@ -0,0 +1,339 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """Merge multiple GR00T/LeRobot view datasets into one physical dataset.
3
+
4
+ The merge preserves every source row value except dataset-local columns:
5
+ episode_index, frame_index, index, and task-id columns are remapped into the
6
+ merged dataset. Videos are symlinked from the source datasets.
7
+ """
8
+
9
+ from __future__ import annotations
10
+
11
+ import argparse
12
+ import json
13
+ import math
14
+ import shutil
15
+ import statistics
16
+ import time
17
+ from collections import Counter, defaultdict
18
+ from datetime import datetime, timezone
19
+ from pathlib import Path
20
+ from typing import Any
21
+
22
+ import numpy as np
23
+ import pandas as pd
24
+
25
+
26
+ TASK_COLUMNS = (
27
+ "task_index",
28
+ "annotation.human.task_description",
29
+ "annotation.human.task_name",
30
+ )
31
+
32
+
33
+ def parse_args() -> argparse.Namespace:
34
+ parser = argparse.ArgumentParser(description=__doc__)
35
+ parser.add_argument("--source", type=Path, action="append", required=True)
36
+ parser.add_argument("--output", type=Path, required=True)
37
+ parser.add_argument("--copy-videos", action="store_true")
38
+ parser.add_argument("--force", action="store_true")
39
+ return parser.parse_args()
40
+
41
+
42
+ def load_json(path: Path) -> dict[str, Any]:
43
+ return json.loads(path.read_text())
44
+
45
+
46
+ def load_jsonl(path: Path) -> list[dict[str, Any]]:
47
+ with path.open() as f:
48
+ return [json.loads(line) for line in f if line.strip()]
49
+
50
+
51
+ def write_jsonl(path: Path, rows: list[dict[str, Any]]) -> None:
52
+ path.parent.mkdir(parents=True, exist_ok=True)
53
+ with path.open("w") as f:
54
+ for row in rows:
55
+ f.write(json.dumps(row, sort_keys=False) + "\n")
56
+
57
+
58
+ def copy_or_link(src: Path, dst: Path, copy: bool = False) -> None:
59
+ if not src.exists():
60
+ raise FileNotFoundError(src)
61
+ dst.parent.mkdir(parents=True, exist_ok=True)
62
+ if dst.exists() or dst.is_symlink():
63
+ if dst.is_dir() and not dst.is_symlink():
64
+ shutil.rmtree(dst)
65
+ else:
66
+ dst.unlink()
67
+ if copy:
68
+ shutil.copy2(src, dst)
69
+ else:
70
+ dst.symlink_to(src.resolve())
71
+
72
+
73
+ def episode_path(dataset_root: Path, episode_index: int, chunks_size: int) -> Path:
74
+ chunk = episode_index // chunks_size
75
+ return dataset_root / "data" / f"chunk-{chunk:03d}" / f"episode_{episode_index:06d}.parquet"
76
+
77
+
78
+ def video_features(info: dict[str, Any]) -> list[str]:
79
+ return [key for key, spec in info.get("features", {}).items() if spec.get("dtype") == "video"]
80
+
81
+
82
+ def source_video_path(dataset_root: Path, feature: str, episode_index: int, chunks_size: int) -> Path:
83
+ chunk = episode_index // chunks_size
84
+ return dataset_root / "videos" / f"chunk-{chunk:03d}" / feature / f"episode_{episode_index:06d}.mp4"
85
+
86
+
87
+ def remap_episode_stats(row: dict[str, Any], new_episode_index: int, global_start: int, length: int) -> dict[str, Any]:
88
+ row = json.loads(json.dumps(row))
89
+ row["episode_index"] = new_episode_index
90
+ stats = row.setdefault("stats", {})
91
+ stats["episode_index"] = {
92
+ "min": [new_episode_index],
93
+ "max": [new_episode_index],
94
+ "mean": [float(new_episode_index)],
95
+ "std": [0.0],
96
+ "count": [length],
97
+ }
98
+ stats["index"] = {
99
+ "min": [global_start],
100
+ "max": [global_start + length - 1],
101
+ "mean": [global_start + (length - 1) / 2.0],
102
+ "std": [statistics.pstdev(range(global_start, global_start + length))],
103
+ "count": [length],
104
+ }
105
+ return row
106
+
107
+
108
+ def column_values_to_matrix(values: pd.Series) -> np.ndarray:
109
+ return np.vstack([np.asarray(v, dtype=np.float64).reshape(1, -1) for v in values])
110
+
111
+
112
+ def calculate_stats(parquet_paths: list[Path]) -> dict[str, dict[str, list[float]]]:
113
+ accum: dict[str, list[np.ndarray]] = defaultdict(list)
114
+ for path in parquet_paths:
115
+ df = pd.read_parquet(path)
116
+ for column in df.columns:
117
+ accum[column].append(column_values_to_matrix(df[column]))
118
+ stats: dict[str, dict[str, list[float]]] = {}
119
+ for column, chunks in accum.items():
120
+ values = np.concatenate(chunks, axis=0)
121
+ stats[column] = {
122
+ "mean": np.mean(values, axis=0).tolist(),
123
+ "std": np.std(values, axis=0).tolist(),
124
+ "min": np.min(values, axis=0).tolist(),
125
+ "max": np.max(values, axis=0).tolist(),
126
+ "q01": np.quantile(values, 0.01, axis=0).tolist(),
127
+ "q99": np.quantile(values, 0.99, axis=0).tolist(),
128
+ }
129
+ return stats
130
+
131
+
132
+ def validate_contract(sources: list[Path]) -> tuple[dict[str, Any], list[str]]:
133
+ infos = [load_json(src / "meta" / "info.json") for src in sources]
134
+ base = infos[0]
135
+ base_features = base.get("features", {})
136
+ base_video_features = video_features(base)
137
+ for src, info in zip(sources[1:], infos[1:]):
138
+ if info.get("features", {}) != base_features:
139
+ raise RuntimeError(f"Feature contract differs for {src}")
140
+ if set(video_features(info)) != set(base_video_features):
141
+ raise RuntimeError(f"Video feature set differs for {src}")
142
+ return base, base_video_features
143
+
144
+
145
+ def main() -> None:
146
+ args = parse_args()
147
+ sources = [src.resolve() for src in args.source]
148
+ for src in sources:
149
+ if not (src / "meta" / "info.json").exists():
150
+ raise FileNotFoundError(src / "meta" / "info.json")
151
+ base_info, vid_features = validate_contract(sources)
152
+
153
+ if args.output.exists():
154
+ if not args.force:
155
+ raise FileExistsError(f"{args.output} exists; pass --force")
156
+ backup = args.output.with_name(f"{args.output.name}.bak_{time.strftime('%Y%m%d_%H%M%S')}")
157
+ args.output.rename(backup)
158
+ print(f"Moved existing output to {backup}")
159
+
160
+ out_meta = args.output / "meta"
161
+ out_data_root = args.output / "data"
162
+ out_video_root = args.output / "videos"
163
+ out_meta.mkdir(parents=True, exist_ok=True)
164
+
165
+ for name in ("modality.json", "view_manifest.json", "embodiment.json", "relative_stats.json"):
166
+ src_file = sources[0] / "meta" / name
167
+ if src_file.exists():
168
+ shutil.copy2(src_file, out_meta / name)
169
+
170
+ global_task_id: dict[str, int] = {}
171
+ global_tasks: list[dict[str, Any]] = []
172
+
173
+ def task_id(task_text: str) -> int:
174
+ if task_text not in global_task_id:
175
+ idx = len(global_task_id)
176
+ global_task_id[task_text] = idx
177
+ global_tasks.append({"task_index": idx, "task": task_text})
178
+ return global_task_id[task_text]
179
+
180
+ new_episodes: list[dict[str, Any]] = []
181
+ new_episode_stats: list[dict[str, Any]] = []
182
+ merge_manifest: list[dict[str, Any]] = []
183
+ parquet_paths: list[Path] = []
184
+ source_counts: Counter[str] = Counter()
185
+ source_frame_counts: Counter[str] = Counter()
186
+ global_index = 0
187
+ new_episode_index = 0
188
+ out_chunks_size = int(base_info.get("chunks_size", 1000))
189
+
190
+ for source_idx, src_root in enumerate(sources):
191
+ src_name = src_root.name
192
+ src_info = load_json(src_root / "meta" / "info.json")
193
+ src_chunks_size = int(src_info.get("chunks_size", 1000))
194
+ src_episodes = {int(r["episode_index"]): r for r in load_jsonl(src_root / "meta" / "episodes.jsonl")}
195
+ src_tasks = {int(r["task_index"]): r["task"] for r in load_jsonl(src_root / "meta" / "tasks.jsonl")}
196
+ src_episode_stats = (
197
+ {int(r["episode_index"]): r for r in load_jsonl(src_root / "meta" / "episodes_stats.jsonl")}
198
+ if (src_root / "meta" / "episodes_stats.jsonl").exists()
199
+ else {}
200
+ )
201
+ for src_episode_index in sorted(src_episodes):
202
+ src_episode_meta = src_episodes[src_episode_index]
203
+ src_parquet = episode_path(src_root, src_episode_index, src_chunks_size)
204
+ if not src_parquet.exists():
205
+ raise FileNotFoundError(src_parquet)
206
+ df = pd.read_parquet(src_parquet).copy()
207
+ length = len(df)
208
+ if int(src_episode_meta["length"]) != length:
209
+ raise RuntimeError(f"Length mismatch for {src_parquet}: parquet={length}, meta={src_episode_meta['length']}")
210
+
211
+ if "episode_index" in df.columns:
212
+ df["episode_index"] = np.full(length, new_episode_index, dtype=np.int64)
213
+ if "frame_index" in df.columns:
214
+ df["frame_index"] = np.arange(length, dtype=np.int64)
215
+ if "index" in df.columns:
216
+ df["index"] = np.arange(global_index, global_index + length, dtype=np.int64)
217
+
218
+ task_column_mappings: dict[str, dict[int, int]] = {}
219
+ for column in TASK_COLUMNS:
220
+ if column in df.columns:
221
+ mapping = {}
222
+ for old_idx in sorted(pd.unique(df[column])):
223
+ old_idx_int = int(old_idx)
224
+ mapping[old_idx_int] = task_id(src_tasks[old_idx_int])
225
+ df[column] = df[column].map(lambda v: mapping[int(v)]).astype("int64")
226
+ task_column_mappings[column] = mapping
227
+
228
+ out_parquet = episode_path(args.output, new_episode_index, out_chunks_size)
229
+ out_parquet.parent.mkdir(parents=True, exist_ok=True)
230
+ df.to_parquet(out_parquet, index=False)
231
+ parquet_paths.append(out_parquet)
232
+
233
+ video_links: dict[str, str] = {}
234
+ source_video_links: dict[str, str] = {}
235
+ for feature in vid_features:
236
+ src_video = source_video_path(src_root, feature, src_episode_index, src_chunks_size)
237
+ dst_video = source_video_path(args.output, feature, new_episode_index, out_chunks_size)
238
+ copy_or_link(src_video, dst_video, copy=args.copy_videos)
239
+ video_links[feature] = str(dst_video)
240
+ source_video_links[feature] = str(src_video)
241
+
242
+ extras_src = src_root / "extras" / f"episode_{src_episode_index:06d}"
243
+ extras_dst = args.output / "extras" / f"episode_{new_episode_index:06d}"
244
+ if extras_src.exists():
245
+ extras_dst.mkdir(parents=True, exist_ok=True)
246
+ for child in extras_src.iterdir():
247
+ copy_or_link(child, extras_dst / child.name, copy=args.copy_videos)
248
+
249
+ new_episodes.append(
250
+ {
251
+ "episode_index": new_episode_index,
252
+ "tasks": src_episode_meta.get("tasks", []),
253
+ "length": length,
254
+ }
255
+ )
256
+ if src_episode_index in src_episode_stats:
257
+ new_episode_stats.append(
258
+ remap_episode_stats(src_episode_stats[src_episode_index], new_episode_index, global_index, length)
259
+ )
260
+
261
+ merge_manifest.append(
262
+ {
263
+ "episode_index": new_episode_index,
264
+ "source_dataset_index": source_idx,
265
+ "source_dataset_name": src_name,
266
+ "source_dataset_root": str(src_root),
267
+ "source_episode_index": src_episode_index,
268
+ "source_length": length,
269
+ "source_parquet": str(src_parquet),
270
+ "global_index_start": global_index,
271
+ "global_index_end": global_index + length - 1,
272
+ "task_column_mappings": task_column_mappings,
273
+ "videos": video_links,
274
+ "source_videos": source_video_links,
275
+ }
276
+ )
277
+ source_counts[src_name] += 1
278
+ source_frame_counts[src_name] += length
279
+ global_index += length
280
+ new_episode_index += 1
281
+
282
+ write_jsonl(out_meta / "episodes.jsonl", new_episodes)
283
+ write_jsonl(out_meta / "tasks.jsonl", global_tasks)
284
+ if new_episode_stats:
285
+ write_jsonl(out_meta / "episodes_stats.jsonl", new_episode_stats)
286
+
287
+ info = json.loads(json.dumps(base_info))
288
+ info["total_episodes"] = len(new_episodes)
289
+ info["total_frames"] = global_index
290
+ info["total_videos"] = len(new_episodes) * len(vid_features)
291
+ info["total_chunks"] = max(1, math.ceil(len(new_episodes) / out_chunks_size))
292
+ info["total_tasks"] = len(global_tasks)
293
+ info["splits"] = {"train": f"0:{len(new_episodes)}"}
294
+ info["created_from_merged_gr00t_datasets"] = {
295
+ "created_at": datetime.now(timezone.utc).isoformat(),
296
+ "sources": [str(src) for src in sources],
297
+ "source_episode_counts": dict(source_counts),
298
+ "source_frame_counts": dict(source_frame_counts),
299
+ "video_policy": "symlink" if not args.copy_videos else "copy",
300
+ }
301
+ (out_meta / "info.json").write_text(json.dumps(info, indent=2) + "\n")
302
+
303
+ stats = calculate_stats(parquet_paths)
304
+ (out_meta / "stats.json").write_text(json.dumps(stats, indent=2) + "\n")
305
+ (out_meta / "merged_dataset_manifest.json").write_text(
306
+ json.dumps(
307
+ {
308
+ "created_at": datetime.now(timezone.utc).isoformat(),
309
+ "output": str(args.output),
310
+ "sources": [str(src) for src in sources],
311
+ "episode_count": len(new_episodes),
312
+ "total_frames": global_index,
313
+ "source_episode_counts": dict(source_counts),
314
+ "source_frame_counts": dict(source_frame_counts),
315
+ "episodes": merge_manifest,
316
+ },
317
+ indent=2,
318
+ )
319
+ + "\n"
320
+ )
321
+
322
+ print(
323
+ json.dumps(
324
+ {
325
+ "output": str(args.output),
326
+ "episodes": len(new_episodes),
327
+ "frames": global_index,
328
+ "videos": len(new_episodes) * len(vid_features),
329
+ "tasks": len(global_tasks),
330
+ "source_episode_counts": dict(source_counts),
331
+ "source_frame_counts": dict(source_frame_counts),
332
+ },
333
+ indent=2,
334
+ )
335
+ )
336
+
337
+
338
+ if __name__ == "__main__":
339
+ main()
dataset/render_to_gr00t.py ADDED
@@ -0,0 +1,353 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """Render mesh-swapped MimicGen demos into GR00T 1.5 LeRobot-style dataset.
3
+
4
+ Reads gen HDF5s from $WORKSPACE/gen_per_target_sam3d_50/<cat>_<model>/<cat>_<model>_gen/demo.hdf5,
5
+ writes parquet+meta+videos+extras to $DATASET_OUT.
6
+
7
+ This is a lightly-modified copy of /lp-dev/jonghoon/mimicgen_augment/render_per_target.py
8
+ that takes input/output paths from env vars (or defaults), so we can produce a fresh dataset
9
+ without clobbering the existing groot1.5_mimcgen_per_target.
10
+ """
11
+ import os
12
+ os.environ["MUJOCO_GL"] = "egl"
13
+
14
+ import h5py, json, pathlib, glob, sys, gzip, re
15
+ import numpy as np
16
+ import pyarrow.parquet as pq
17
+ import pyarrow as pa
18
+
19
+ import robosuite, robocasa
20
+ from robomimic.envs.env_robosuite import EnvRobosuite
21
+ import robomimic.utils.obs_utils as ObsUtils
22
+ ObsUtils.initialize_obs_utils_with_obs_specs({'obs': {'low_dim': [], 'rgb': []}})
23
+
24
+ sys.path.insert(0, "/lp-dev/jonghoon/mimicgen_augment")
25
+ from swap_and_render import (
26
+ extract_obs_state, write_meta, CAMERAS, IMG_SIZE,
27
+ _bbox_diag, _bbox_diag_multi, _find_canonical_visual,
28
+ )
29
+ from build_sam3d_sources import find_sam3d_mesh as _find_sam3d_mesh_untextured
30
+
31
+ # The meshes build_sam3d_sources points at carry vertex colours but no UVs, so MuJoCo
32
+ # renders them flat grey. /data/minha/pose6daug/mesh_textured holds the same
33
+ # reconstructions re-baked with UVs + material_0.png, in the same <model>_to_* layout.
34
+ TEXTURED_PAIR_ROOT = pathlib.Path(os.environ.get(
35
+ "TEXTURED_PAIR_ROOT", "/data/minha/pose6daug/mesh_textured/pairs"))
36
+
37
+
38
+ def find_sam3d_mesh(target_cat, target_model):
39
+ for root in sorted(TEXTURED_PAIR_ROOT.glob("*")):
40
+ for pair_dir in sorted(root.glob(f"{target_model}_to_*")):
41
+ obj = pair_dir / "target_rotated" / "original.obj"
42
+ if obj.exists():
43
+ return obj
44
+ return _find_sam3d_mesh_untextured(target_cat, target_model)
45
+ import mujoco, imageio
46
+
47
+ GEN_DIR = pathlib.Path(os.environ.get(
48
+ "GEN_DIR", "/lp-dev/jonghoon/mimicgen_aug_groot15/gen_per_target_sam3d_50"))
49
+ DATASET_ROOT = pathlib.Path(os.environ.get(
50
+ "DATASET_OUT", "/lp-dev/jonghoon/mimicgen_aug_groot15/dataset/groot1.5_mimcgen_per_target_50"))
51
+
52
+ TARGETS = [
53
+ ("donut", "donut_5"),
54
+ ("jar", "Jar023"),
55
+ ("measuring_cup", "MeasuringCup009"),
56
+ ("soap_dispenser", "SoapDispenser010"),
57
+ ("steak", "steak_8"),
58
+ ("syrup_bottle", "SyrupBottle005"),
59
+ ("teapot", "teapot_6"),
60
+ ("teapot", "teapot_7"),
61
+ ]
62
+
63
+
64
+ def swap_visual_to_sam3d(model_xml, sam3d_obj_path):
65
+ tex = pathlib.Path(sam3d_obj_path).parent / "material_0.png"
66
+ swap_texture = tex if tex.exists() else None
67
+ canonical_file, canonical_scale_xyz = _find_canonical_visual(model_xml)
68
+ if canonical_file is None:
69
+ return None
70
+ sam3d_diag = _bbox_diag(str(sam3d_obj_path))
71
+ # _find_canonical_visual returns a LIST of visual submeshes (a robocasa object can be
72
+ # split into several, e.g. teapot = lid + base) and its docstring says to take the union
73
+ # bbox with _bbox_diag_multi. Upstream passed that list to _bbox_diag, which wants one
74
+ # path, so the diagonal came back None and swap_visual_to_sam3d silently returned None --
75
+ # the swap never happened, on any target.
76
+ canonical_diag = (_bbox_diag_multi(canonical_file)
77
+ if isinstance(canonical_file, (list, tuple))
78
+ else _bbox_diag(canonical_file))
79
+ if sam3d_diag is None or canonical_diag is None:
80
+ return None
81
+ mean_canonical = float(np.mean(canonical_scale_xyz))
82
+ target_scale = (canonical_diag * mean_canonical) / sam3d_diag
83
+ scale_str = f'scale="{target_scale:.6f} {target_scale:.6f} {target_scale:.6f}"'
84
+
85
+ out = []
86
+ for line in model_xml.split("\n"):
87
+ s = line.strip()
88
+ if s.startswith("<mesh") and 'name="obj_' in s and "collision" not in s.lower():
89
+ line = re.sub(r'file="[^"]+"', f'file="{sam3d_obj_path}"', line)
90
+ line = re.sub(r' scale="[^"]+"', f' {scale_str}', line)
91
+ if "scale=" not in line:
92
+ line = line.rstrip("/>").rstrip() + f" {scale_str}/>"
93
+ elif s.startswith("<texture") and 'name="obj_' in s:
94
+ if swap_texture is not None:
95
+ # the replacement mesh is UV-mapped and ships its own baked texture
96
+ line = re.sub(r'file="[^"]+"', f'file="{swap_texture}"', line)
97
+ line = re.sub(r' builtin="[^"]+"', "", line)
98
+ else:
99
+ line = re.sub(r'file="[^"]+"', 'builtin="flat"', line)
100
+ if "width=" not in line:
101
+ line = line.rstrip("/>").rstrip() + ' width="2" height="2"/>'
102
+ elif s.startswith("<material") and 'name="obj_' in s:
103
+ if swap_texture is None:
104
+ # no texture to apply -- flat grey beats whatever the stale binding gives
105
+ forced_rgba = "0.7 0.7 0.7 1.0"
106
+ if re.search(r'rgba="[^"]+"', line):
107
+ line = re.sub(r'rgba="[^"]+"', f'rgba="{forced_rgba}"', line)
108
+ else:
109
+ line = line.rstrip("/>").rstrip() + f' rgba="{forced_rgba}"/>'
110
+ line = re.sub(r' texture="[^"]+"', "", line)
111
+ elif re.search(r'rgba="[^"]+"', line):
112
+ line = re.sub(r'rgba="[^"]+"', 'rgba="1 1 1 1"', line)
113
+ out.append(line)
114
+ return "\n".join(out)
115
+
116
+
117
+ def render_demo(env, demo_grp, out_root, episode_index, row_info, global_frame_offset, sam3d_obj_path=None):
118
+ out_root = pathlib.Path(out_root)
119
+ model_xml = demo_grp.attrs["model_file"]
120
+ if sam3d_obj_path is not None:
121
+ swapped = swap_visual_to_sam3d(model_xml, str(sam3d_obj_path))
122
+ if swapped is not None:
123
+ model_xml = swapped
124
+
125
+ states = demo_grp["states"][0]
126
+ state = dict(model=model_xml, states=states)
127
+ if "ep_meta" in demo_grp.attrs:
128
+ state["ep_meta"] = demo_grp.attrs["ep_meta"]
129
+
130
+ try:
131
+ env.reset_to(state)
132
+ except Exception as e:
133
+ print(f" Reset failed: {e}")
134
+ return False, 0
135
+
136
+ all_states = demo_grp["states"][:]
137
+ all_actions = demo_grp["actions"][:]
138
+ sim = env.base_env.sim
139
+ mj_model = sim.model._model
140
+ mj_data = sim.data._data
141
+ n_frames = len(all_states)
142
+
143
+ cam_ids = {cam: mujoco.mj_name2id(mj_model, mujoco.mjtObj.mjOBJ_CAMERA, cam) for cam in CAMERAS}
144
+ scene_option = mujoco.MjvOption()
145
+ scene_option.geomgroup[:] = 0
146
+ scene_option.geomgroup[1] = 1
147
+ scene_option.sitegroup[:] = 0
148
+
149
+ renderer = mujoco.Renderer(mj_model, height=IMG_SIZE, width=IMG_SIZE)
150
+ ep_tag = f"episode_{episode_index:06d}"
151
+ vid_dir = out_root / "videos" / "chunk-000"
152
+ writers = {}
153
+ for cam in CAMERAS:
154
+ cam_dir = vid_dir / f"observation.images.{cam}"
155
+ cam_dir.mkdir(parents=True, exist_ok=True)
156
+ writers[cam] = imageio.get_writer(
157
+ str(cam_dir / f"{ep_tag}.mp4"), fps=20, codec="libx264",
158
+ macro_block_size=1, ffmpeg_params=["-pix_fmt", "yuv420p"],
159
+ )
160
+
161
+ obs_states = []
162
+ for t in range(n_frames):
163
+ sim.set_state_from_flattened(all_states[t])
164
+ sim.forward()
165
+ obs_states.append(extract_obs_state(env))
166
+ for cam in CAMERAS:
167
+ renderer.update_scene(mj_data, camera=cam_ids[cam], scene_option=scene_option)
168
+ writers[cam].append_data(renderer.render())
169
+
170
+ renderer.close()
171
+ for w in writers.values():
172
+ w.close()
173
+
174
+ obs_states = np.array(obs_states)
175
+ try:
176
+ ep_meta_obj = json.loads(demo_grp.attrs.get("ep_meta", "{}"))
177
+ except Exception:
178
+ ep_meta_obj = {}
179
+
180
+ # The HDF5 holds actions in the simulator's own order,
181
+ # [eef_pos(3), eef_rot(3), gripper, base(3), torso, base_mode]
182
+ # while meta/modality.json declares the LeRobot order,
183
+ # [base_motion(4), control_mode, eef_pos(3), eef_rot(3), gripper_close].
184
+ # Upstream wrote them through unchanged, so every slice landed on the wrong dimensions:
185
+ # arm motion was read as base motion and base_mode as the gripper, which trains a policy
186
+ # that drives the base away and never closes the gripper (measured 0/160 at eval).
187
+ actions = all_actions[:, [7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6]]
188
+ if len(actions) < n_frames:
189
+ pad = np.zeros((n_frames - len(actions), actions.shape[1]))
190
+ actions = np.vstack([actions, pad])
191
+
192
+ records = []
193
+ for t in range(n_frames):
194
+ is_last = (t == n_frames - 1)
195
+ records.append({
196
+ "observation.state": obs_states[t].tolist(),
197
+ "action": actions[t].tolist(),
198
+ "next.reward": 1.0 if is_last else 0.0,
199
+ "next.done": is_last,
200
+ "timestamp": t / 20.0,
201
+ "frame_index": t,
202
+ "episode_index": episode_index,
203
+ "index": global_frame_offset + t,
204
+ "task_index": 0,
205
+ "annotation.human.task_description": 0,
206
+ "annotation.human.task_name": 0,
207
+ })
208
+
209
+ schema = pa.schema([
210
+ ("observation.state", pa.list_(pa.float64())),
211
+ ("action", pa.list_(pa.float64())),
212
+ ("next.reward", pa.float32()),
213
+ ("next.done", pa.bool_()),
214
+ ("timestamp", pa.float32()),
215
+ ("frame_index", pa.int64()),
216
+ ("episode_index", pa.int64()),
217
+ ("index", pa.int64()),
218
+ ("task_index", pa.int64()),
219
+ ("annotation.human.task_description", pa.int64()),
220
+ ("annotation.human.task_name", pa.int64()),
221
+ ])
222
+ table = pa.Table.from_pylist(records, schema=schema)
223
+ data_dir = out_root / "data" / "chunk-000"
224
+ data_dir.mkdir(parents=True, exist_ok=True)
225
+ pq.write_table(table, str(data_dir / f"{ep_tag}.parquet"))
226
+
227
+ extras_dir = out_root / "extras" / ep_tag
228
+ extras_dir.mkdir(parents=True, exist_ok=True)
229
+ aug_meta = dict(ep_meta_obj)
230
+ aug_meta["augmentation"] = {
231
+ "type": "mimicgen_per_target_sam3d_50",
232
+ "target_mesh": row_info["target_mesh"],
233
+ "source_demo": row_info["source_demo"],
234
+ }
235
+ (extras_dir / "ep_meta.json").write_text(json.dumps(aug_meta, indent=2))
236
+ with gzip.open(str(extras_dir / "model.xml.gz"), "wt") as f:
237
+ f.write(model_xml)
238
+ np.savez_compressed(str(extras_dir / "states.npz"), states=all_states)
239
+
240
+ return True, n_frames
241
+
242
+
243
+ def main():
244
+ DATASET_ROOT.mkdir(parents=True, exist_ok=True)
245
+ print(f"GEN_DIR={GEN_DIR}")
246
+ print(f"DATASET_ROOT={DATASET_ROOT}")
247
+
248
+ target_demos = {}
249
+ for cat, model in TARGETS:
250
+ name = f"{cat}_{model}"
251
+ gen_paths = sorted(glob.glob(f"{GEN_DIR}/{name}*/{name}_gen/demo.hdf5"))
252
+ if not gen_paths:
253
+ print(f" WARN: no demos for {cat}/{model} in {GEN_DIR}/{name}")
254
+ target_demos[(cat, model)] = []
255
+ continue
256
+ target_demos[(cat, model)] = gen_paths
257
+
258
+ first_hdf5 = None
259
+ for paths in target_demos.values():
260
+ if paths:
261
+ first_hdf5 = paths[0]
262
+ break
263
+ if first_hdf5 is None:
264
+ print("No demos found at all!")
265
+ return
266
+
267
+ fh0 = h5py.File(first_hdf5, "r")
268
+ env_meta = json.loads(fh0["data"].attrs["env_args"])
269
+ fh0.close()
270
+ env_kwargs = dict(env_meta["env_kwargs"])
271
+ env_kwargs["has_offscreen_renderer"] = True
272
+ env_kwargs["use_camera_obs"] = False
273
+ env = EnvRobosuite(env_name=env_meta["env_name"], **env_kwargs)
274
+
275
+ episodes_info = []
276
+ task_descriptions = set()
277
+ global_frame_offset = 0
278
+ success_count = 0
279
+ ep_idx = 0
280
+
281
+ only = os.environ.get("TARGET_FILTER") # "<cat>/<model>" -> shard this run to one target
282
+ for cat, model in TARGETS:
283
+ if only and f"{cat}/{model}" != only:
284
+ continue
285
+ gen_paths = target_demos[(cat, model)]
286
+ target_label = f"{cat}/{model}"
287
+ # The visual-mesh swap is OFF here. Upstream replaces the object's visual mesh with
288
+ # a reconstruction from an eval rollout and forces it grey; neither the Ours nor the
289
+ # VACE dataset builder does that, and our episodes were generated (and already
290
+ # inspected in output_aug_256) with the plain RoboCasa asset. USE_MESH_SWAP=1 restores
291
+ # the upstream behaviour.
292
+ sam3d_mesh = find_sam3d_mesh(cat, model) if os.environ.get("USE_MESH_SWAP") == "1" else None
293
+ print(f" -> {target_label}: mesh swap = {sam3d_mesh}")
294
+ n_for_target = 0
295
+ for hp in gen_paths:
296
+ fh = h5py.File(hp, "r")
297
+ for dk in sorted(fh["data"].keys(), key=lambda x: int(x.split("_")[1])):
298
+ row_info = {
299
+ "target_mesh": target_label,
300
+ "source_demo": f"{hp}::{dk}",
301
+ }
302
+
303
+ ep_tag = f"episode_{ep_idx:06d}"
304
+ parquet_path = DATASET_ROOT / "data" / "chunk-000" / f"{ep_tag}.parquet"
305
+ if parquet_path.exists():
306
+ tbl = pq.read_table(str(parquet_path))
307
+ n = len(tbl)
308
+ em_path = DATASET_ROOT / "extras" / ep_tag / "ep_meta.json"
309
+ td = "Pick and place"
310
+ if em_path.exists():
311
+ em = json.loads(em_path.read_text())
312
+ td = em.get("lang", td)
313
+ episodes_info.append({"episode_index": ep_idx, "tasks": [td], "length": n})
314
+ task_descriptions.add(td)
315
+ global_frame_offset += n
316
+ success_count += 1
317
+ n_for_target += 1
318
+ ep_idx += 1
319
+ print(f" ep{ep_idx-1:03d} [{target_label}] {dk} — already done")
320
+ continue
321
+
322
+ print(f" ep{ep_idx:03d} [{target_label}] {dk}", flush=True)
323
+ ok, n_frames = render_demo(
324
+ env, fh[f"data/{dk}"],
325
+ DATASET_ROOT, ep_idx, row_info, global_frame_offset,
326
+ sam3d_obj_path=sam3d_mesh,
327
+ )
328
+ if ok:
329
+ success_count += 1
330
+ n_for_target += 1
331
+ em_path = DATASET_ROOT / "extras" / ep_tag / "ep_meta.json"
332
+ em = json.loads(em_path.read_text()) if em_path.exists() else {}
333
+ td = em.get("lang", f"Pick and place {cat}")
334
+ episodes_info.append({"episode_index": ep_idx, "tasks": [td], "length": n_frames})
335
+ task_descriptions.add(td)
336
+ global_frame_offset += n_frames
337
+ print(f" OK ({n_frames} frames)")
338
+ else:
339
+ print(f" FAILED")
340
+ ep_idx += 1
341
+ fh.close()
342
+ print(f" -> {target_label}: {n_for_target} episodes")
343
+
344
+ env.close()
345
+
346
+ if episodes_info:
347
+ write_meta(DATASET_ROOT, episodes_info, sorted(task_descriptions))
348
+ print(f"Wrote meta/ with {len(episodes_info)} episodes")
349
+ print(f"Done: {success_count} episodes total")
350
+
351
+
352
+ if __name__ == "__main__":
353
+ main()
dataset/repair_action_order.py ADDED
@@ -0,0 +1,71 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """Reorder the action column of an already-built gr00t_views dataset, in place.
3
+
4
+ Cheaper than rebuilding: the videos and observation.state are correct, only `action` was
5
+ written in simulator order. Rewrites every episode parquet and recomputes meta/stats.json for
6
+ the action key (mean/std/min/max/q01/q99), leaving every other stats entry untouched.
7
+
8
+ Refuses to run unless the dataset currently looks like simulator order, so it cannot be
9
+ applied twice.
10
+ """
11
+ import argparse, glob, json, os, shutil, sys
12
+ import numpy as np
13
+ import pandas as pd
14
+
15
+ FIX = [7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6]
16
+
17
+
18
+ def looks_robosuite(a):
19
+ zero = [i for i in range(12) if np.abs(a[:, i]).max() < 1e-6]
20
+ const = [i for i in range(12) if len(np.unique(np.round(a[:, i], 4))) == 1]
21
+ two = [i for i in range(12) if len(np.unique(np.round(a[:, i], 4))) == 2]
22
+ return set([7, 8, 9, 10]).issubset(zero) and 11 in const and 6 in two
23
+
24
+
25
+ def main():
26
+ ap = argparse.ArgumentParser()
27
+ ap.add_argument("--dataset", required=True)
28
+ ap.add_argument("--dry-run", action="store_true")
29
+ a = ap.parse_args()
30
+
31
+ paths = sorted(glob.glob(os.path.join(a.dataset, "data", "chunk-*", "*.parquet")))
32
+ if not paths:
33
+ sys.exit(f"no parquet under {a.dataset}")
34
+
35
+ probe = np.vstack([np.stack(pd.read_parquet(p, columns=["action"])["action"].tolist())
36
+ for p in paths[:10]])
37
+ if not looks_robosuite(probe):
38
+ sys.exit("dataset does not look like simulator order -- refusing "
39
+ "(already fixed, or a layout this tool does not know)")
40
+
41
+ print(f"{len(paths)} episodes to rewrite")
42
+ if a.dry_run:
43
+ return
44
+
45
+ all_actions = []
46
+ for i, p in enumerate(paths):
47
+ df = pd.read_parquet(p)
48
+ arr = np.stack(df["action"].tolist())[:, FIX]
49
+ df["action"] = list(arr)
50
+ df.to_parquet(p, index=False)
51
+ all_actions.append(arr)
52
+ if (i + 1) % 64 == 0:
53
+ print(f" {i+1}/{len(paths)}", flush=True)
54
+ A = np.vstack(all_actions)
55
+
56
+ sp = os.path.join(a.dataset, "meta", "stats.json")
57
+ shutil.copy(sp, sp + ".bak_action_order")
58
+ stats = json.load(open(sp))
59
+ stats["action"] = {
60
+ "mean": A.mean(0).tolist(), "std": A.std(0).tolist(),
61
+ "min": A.min(0).tolist(), "max": A.max(0).tolist(),
62
+ "q01": np.percentile(A, 1, axis=0).tolist(),
63
+ "q99": np.percentile(A, 99, axis=0).tolist(),
64
+ }
65
+ json.dump(stats, open(sp, "w"), indent=2)
66
+ print(f"rewrote {len(paths)} parquet, recomputed action stats over {A.shape[0]} frames")
67
+ print(f" stats backup: {sp}.bak_action_order")
68
+
69
+
70
+ if __name__ == "__main__":
71
+ raise SystemExit(main())
dataset/repair_mimicgen_task_ids.py ADDED
@@ -0,0 +1,208 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """Repair per-episode task-id columns and modality aliases in MimicGen GR00T datasets.
3
+
4
+ The MimicGen swap/render writer stores all parquet task columns as 0 while
5
+ episodes.jsonl stores the intended language per episode. GR00T training reads
6
+ the parquet task ids through tasks.jsonl, so every episode becomes task 0 unless
7
+ we remap these columns.
8
+
9
+ It also writes raw RoboCasa camera names as modality keys. The
10
+ single_panda_gripper config expects the canonical GR00T view names
11
+ left_view/right_view/wrist_view, each pointing to the raw observation key.
12
+ """
13
+
14
+ from __future__ import annotations
15
+
16
+ import argparse
17
+ import json
18
+ from collections import Counter
19
+ from pathlib import Path
20
+
21
+ import numpy as np
22
+ import pandas as pd
23
+
24
+
25
+ TASK_COLUMNS = (
26
+ "task_index",
27
+ "annotation.human.task_description",
28
+ "annotation.human.task_name",
29
+ )
30
+
31
+ CANONICAL_VIDEO_MODALITY = {
32
+ "left_view": {"original_key": "observation.images.robot0_agentview_left"},
33
+ "right_view": {"original_key": "observation.images.robot0_agentview_right"},
34
+ "wrist_view": {"original_key": "observation.images.robot0_eye_in_hand"},
35
+ }
36
+
37
+ CANONICAL_ANNOTATION_MODALITY = {
38
+ "human.action.task_description": {"original_key": "annotation.human.task_description"},
39
+ }
40
+
41
+ CANONICAL_STATE_MODALITY = {
42
+ "base_position": {"original_key": "observation.state", "start": 0, "end": 3},
43
+ "base_rotation": {
44
+ "original_key": "observation.state",
45
+ "start": 3,
46
+ "end": 7,
47
+ "rotation_type": "quaternion",
48
+ },
49
+ "end_effector_position_relative": {"original_key": "observation.state", "start": 7, "end": 10},
50
+ "end_effector_rotation_relative": {
51
+ "original_key": "observation.state",
52
+ "start": 10,
53
+ "end": 14,
54
+ "rotation_type": "quaternion",
55
+ },
56
+ "gripper_qpos": {"original_key": "observation.state", "start": 14, "end": 16},
57
+ }
58
+
59
+ CANONICAL_ACTION_MODALITY = {
60
+ "base_motion": {"original_key": "action", "start": 0, "end": 4},
61
+ "control_mode": {"original_key": "action", "start": 4, "end": 5},
62
+ "end_effector_position": {"original_key": "action", "start": 5, "end": 8},
63
+ "end_effector_rotation": {"original_key": "action", "start": 8, "end": 11},
64
+ "gripper_close": {"original_key": "action", "start": 11, "end": 12},
65
+ }
66
+
67
+
68
+ def parse_args() -> argparse.Namespace:
69
+ parser = argparse.ArgumentParser(description=__doc__)
70
+ parser.add_argument("--dataset", type=Path, required=True)
71
+ parser.add_argument("--dry-run", action="store_true")
72
+ return parser.parse_args()
73
+
74
+
75
+ def load_jsonl(path: Path) -> list[dict]:
76
+ with path.open() as f:
77
+ return [json.loads(line) for line in f if line.strip()]
78
+
79
+
80
+ def episode_path(dataset_root: Path, episode_index: int, chunks_size: int) -> Path:
81
+ chunk = episode_index // chunks_size
82
+ return dataset_root / "data" / f"chunk-{chunk:03d}" / f"episode_{episode_index:06d}.parquet"
83
+
84
+
85
+ def main() -> None:
86
+ args = parse_args()
87
+ dataset = args.dataset.resolve()
88
+ meta = dataset / "meta"
89
+ info_path = meta / "info.json"
90
+ if not info_path.exists():
91
+ raise FileNotFoundError(f"missing {info_path}")
92
+
93
+ info = json.loads(info_path.read_text())
94
+ chunks_size = int(info.get("chunks_size", 1000))
95
+ episodes = load_jsonl(meta / "episodes.jsonl")
96
+ tasks_path = meta / "tasks.jsonl"
97
+ tasks = load_jsonl(tasks_path)
98
+
99
+ episode_task_texts: list[str] = []
100
+ for episode in episodes:
101
+ for task_text in episode.get("tasks") or []:
102
+ if task_text not in episode_task_texts:
103
+ episode_task_texts.append(str(task_text))
104
+
105
+ existing_task_to_id = {row["task"]: int(row["task_index"]) for row in tasks}
106
+ missing_tasks = [task for task in episode_task_texts if task not in existing_task_to_id]
107
+ tasks_rebuilt = False
108
+ if missing_tasks:
109
+ # Some subset datasets were written with tasks.jsonl containing only task 0
110
+ # even though episodes.jsonl had the full per-episode language. Rebuild the
111
+ # task table from episodes.jsonl so parquet ids can be repaired coherently.
112
+ tasks = [{"task_index": idx, "task": task} for idx, task in enumerate(episode_task_texts)]
113
+ tasks_rebuilt = True
114
+ if not args.dry_run:
115
+ tasks_path.write_text("".join(json.dumps(row, sort_keys=False) + "\n" for row in tasks))
116
+ info["total_tasks"] = len(tasks)
117
+ info_path.write_text(json.dumps(info, indent=2, sort_keys=False) + "\n")
118
+
119
+ task_to_id = {row["task"]: int(row["task_index"]) for row in tasks}
120
+
121
+ modality_path = meta / "modality.json"
122
+ modality_changed = False
123
+ if modality_path.exists():
124
+ modality = json.loads(modality_path.read_text())
125
+ if modality.get("state") != CANONICAL_STATE_MODALITY:
126
+ modality["state"] = CANONICAL_STATE_MODALITY
127
+ modality_changed = True
128
+ if modality.get("action") != CANONICAL_ACTION_MODALITY:
129
+ modality["action"] = CANONICAL_ACTION_MODALITY
130
+ modality_changed = True
131
+ if modality.get("video") != CANONICAL_VIDEO_MODALITY:
132
+ modality["video"] = CANONICAL_VIDEO_MODALITY
133
+ modality_changed = True
134
+ if modality.get("annotation") != CANONICAL_ANNOTATION_MODALITY:
135
+ modality["annotation"] = CANONICAL_ANNOTATION_MODALITY
136
+ modality_changed = True
137
+ if modality_changed and not args.dry_run:
138
+ modality_path.write_text(json.dumps(modality, indent=2, sort_keys=False) + "\n")
139
+
140
+ rewrites: list[dict] = []
141
+ failures: list[str] = []
142
+ text_counts: Counter[str] = Counter()
143
+
144
+ for episode in episodes:
145
+ episode_index = int(episode["episode_index"])
146
+ task_texts = episode.get("tasks") or []
147
+ if len(task_texts) != 1:
148
+ failures.append(f"episode {episode_index}: expected exactly one task, got {task_texts!r}")
149
+ continue
150
+ task_text = str(task_texts[0])
151
+ if task_text not in task_to_id:
152
+ failures.append(f"episode {episode_index}: task text missing from tasks.jsonl: {task_text!r}")
153
+ continue
154
+ task_id = int(task_to_id[task_text])
155
+ path = episode_path(dataset, episode_index, chunks_size)
156
+ if not path.exists():
157
+ failures.append(f"episode {episode_index}: missing parquet {path}")
158
+ continue
159
+
160
+ df = pd.read_parquet(path)
161
+ before = {}
162
+ changed = False
163
+ for column in TASK_COLUMNS:
164
+ if column not in df.columns:
165
+ failures.append(f"episode {episode_index}: missing task column {column}")
166
+ continue
167
+ unique_before = sorted({int(v) for v in df[column].unique()})
168
+ before[column] = unique_before
169
+ desired = np.full(len(df), task_id, dtype=np.int64)
170
+ if unique_before != [task_id]:
171
+ df[column] = desired
172
+ changed = True
173
+ if changed and not args.dry_run:
174
+ df.to_parquet(path, index=False)
175
+ rewrites.append(
176
+ {
177
+ "episode_index": episode_index,
178
+ "task": task_text,
179
+ "task_id": task_id,
180
+ "before": before,
181
+ "changed": changed,
182
+ "length": len(df),
183
+ }
184
+ )
185
+ text_counts[task_text] += 1
186
+
187
+ report = {
188
+ "dataset": str(dataset),
189
+ "dry_run": bool(args.dry_run),
190
+ "episodes_checked": len(episodes),
191
+ "tasks_rebuilt": tasks_rebuilt,
192
+ "missing_tasks_before_rebuild": missing_tasks,
193
+ "modality_changed": modality_changed,
194
+ "rewritten_episodes": sum(1 for row in rewrites if row["changed"]),
195
+ "task_counts": dict(text_counts),
196
+ "failures": failures,
197
+ "passed": not failures,
198
+ }
199
+ out = meta / "mimicgen_task_id_repair.json"
200
+ if not args.dry_run:
201
+ out.write_text(json.dumps({**report, "episodes": rewrites}, indent=2, sort_keys=True) + "\n")
202
+ print(json.dumps(report, indent=2, sort_keys=True))
203
+ if failures:
204
+ raise SystemExit(1)
205
+
206
+
207
+ if __name__ == "__main__":
208
+ main()
dataset/run_convert.sh ADDED
@@ -0,0 +1,22 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env bash
2
+ # gr00t_views conversion, SAM3D textured mesh (USE_MESH_SWAP=1), one output per shard;
3
+ # merged afterwards. Each shard needs its own GEN_DIR or the glob would pick up the others.
4
+ set -u
5
+ N=/tmp/claude-1001/-home-nvidia/68c27ffb-706b-4cb3-8bf2-4618a200d378/scratchpad/natural256
6
+ B=/home/nvidia/minha/pose6daug/baseline/mimicgen
7
+ PY=/lp-dev/jonghoon/mimicgen_augment/envs/mimicgen/bin/python
8
+ GPUS=(0 2 3)
9
+ mkdir -p "$N/logs" "$B/gr00t_views/natural_shards"
10
+ i=0
11
+ for d in "$N"/gen/shard_*; do
12
+ k=$(basename "$d")
13
+ tgt=$(basename "$(ls -d "$d"/*/ | head -1)")
14
+ g=${GPUS[$((i % ${#GPUS[@]}))]}
15
+ MUJOCO_GL=egl PYOPENGL_PLATFORM=egl MUJOCO_EGL_DEVICE_ID=$g \
16
+ GEN_DIR="$d" DATASET_OUT="$B/gr00t_views/natural_shards/$k" USE_MESH_SWAP=1 \
17
+ taskset -c 0-254 nohup "$PY" -u "$B/gr00t_build/render_to_gr00t.py" \
18
+ > "$N/logs/conv_$k.log" 2>&1 &
19
+ i=$((i+1))
20
+ done
21
+ wait
22
+ echo "[nat] convert done $(date -u +%FT%TZ)"
dataset/verify_geom_objwise_gr00t_dataset.py ADDED
@@ -0,0 +1,233 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """Verify a geometric object-wise GR00T dataset preserves source episode data.
3
+
4
+ Expected differences from the source parquets are limited to:
5
+ - episode_index remapped to the new dataset episode id
6
+ - frame_index reset to 0..N-1
7
+ - index remapped to the new global frame range
8
+ - task id columns remapped to this combined dataset's tasks.jsonl ids
9
+
10
+ All state/action/reward/done/timestamp values must match the source episode.
11
+ Videos must resolve to the augmented render output recorded by the builder.
12
+ """
13
+
14
+ from __future__ import annotations
15
+
16
+ import argparse
17
+ import json
18
+ import subprocess
19
+ from pathlib import Path
20
+ from typing import Any
21
+
22
+ import numpy as np
23
+ import pandas as pd
24
+
25
+
26
+ TASK_COLUMNS = (
27
+ "task_index",
28
+ "annotation.human.task_description",
29
+ "annotation.human.task_name",
30
+ )
31
+
32
+ INDEX_COLUMNS = ("episode_index", "frame_index", "index")
33
+
34
+
35
+ def parse_args() -> argparse.Namespace:
36
+ parser = argparse.ArgumentParser(description=__doc__)
37
+ parser.add_argument("--dataset", type=Path, required=True)
38
+ parser.add_argument("--source-root", type=Path, default=Path("/home/nvidia/jonghoon/robocasa_full/pickplace_target_human"))
39
+ parser.add_argument("--render-root", type=Path, default=Path("/home/nvidia/jonghoon/robocasa_full/augmented_video/groot1.5_objwise_ours"))
40
+ parser.add_argument("--check-video-frames", action="store_true")
41
+ parser.add_argument("--allow-copied-videos", action="store_true")
42
+ return parser.parse_args()
43
+
44
+
45
+ def load_json(path: Path) -> dict[str, Any]:
46
+ return json.loads(path.read_text())
47
+
48
+
49
+ def load_jsonl(path: Path) -> list[dict[str, Any]]:
50
+ with path.open() as f:
51
+ return [json.loads(line) for line in f if line.strip()]
52
+
53
+
54
+ def source_tasks(source_root: Path) -> dict[str, dict[int, str]]:
55
+ out: dict[str, dict[int, str]] = {}
56
+ for meta in sorted(source_root.glob("*/meta/tasks.jsonl")):
57
+ task_name = meta.parent.parent.name
58
+ out[task_name] = {int(r["task_index"]): r["task"] for r in load_jsonl(meta)}
59
+ return out
60
+
61
+
62
+ def output_tasks(dataset: Path) -> dict[int, str]:
63
+ return {int(r["task_index"]): r["task"] for r in load_jsonl(dataset / "meta" / "tasks.jsonl")}
64
+
65
+
66
+ def dataset_episode_path(dataset: Path, episode_index: int, chunks_size: int) -> Path:
67
+ chunk = episode_index // chunks_size
68
+ return dataset / "data" / f"chunk-{chunk:03d}" / f"episode_{episode_index:06d}.parquet"
69
+
70
+
71
+ def arrays_equal(left: Any, right: Any) -> bool:
72
+ left_arr = np.asarray(left)
73
+ right_arr = np.asarray(right)
74
+ if left_arr.shape != right_arr.shape:
75
+ return False
76
+ return bool(np.array_equal(left_arr, right_arr))
77
+
78
+
79
+ def series_equal(left: pd.Series, right: pd.Series) -> tuple[bool, str | None]:
80
+ if len(left) != len(right):
81
+ return False, f"length mismatch {len(left)} != {len(right)}"
82
+ for idx, (lv, rv) in enumerate(zip(left.tolist(), right.tolist())):
83
+ if not arrays_equal(lv, rv):
84
+ return False, f"row {idx}: {lv!r} != {rv!r}"
85
+ return True, None
86
+
87
+
88
+ def frame_count(path: Path) -> int | None:
89
+ commands = [
90
+ [
91
+ "ffprobe",
92
+ "-v",
93
+ "error",
94
+ "-count_frames",
95
+ "-select_streams",
96
+ "v:0",
97
+ "-show_entries",
98
+ "stream=nb_read_frames",
99
+ "-of",
100
+ "csv=p=0",
101
+ str(path),
102
+ ],
103
+ [
104
+ "ffprobe",
105
+ "-v",
106
+ "error",
107
+ "-select_streams",
108
+ "v:0",
109
+ "-show_entries",
110
+ "stream=nb_frames",
111
+ "-of",
112
+ "csv=p=0",
113
+ str(path),
114
+ ],
115
+ ]
116
+ for cmd in commands:
117
+ proc = subprocess.run(cmd, check=False, text=True, capture_output=True)
118
+ value = proc.stdout.strip().splitlines()[0] if proc.stdout.strip() else ""
119
+ if value and value != "N/A":
120
+ try:
121
+ return int(value)
122
+ except ValueError:
123
+ continue
124
+ return None
125
+
126
+
127
+ def main() -> None:
128
+ args = parse_args()
129
+ info = load_json(args.dataset / "meta" / "info.json")
130
+ chunks_size = int(info.get("chunks_size", 1000))
131
+ manifest_path = args.dataset / "meta" / "geom_objwise_dataset_manifest.json"
132
+ manifest = load_json(manifest_path)
133
+ src_tasks = source_tasks(args.source_root)
134
+ dst_tasks = output_tasks(args.dataset)
135
+
136
+ failures: list[str] = []
137
+ checked_rows = 0
138
+ changed_column_counts = {name: 0 for name in INDEX_COLUMNS + TASK_COLUMNS}
139
+ video_checks = 0
140
+ frame_checks = 0
141
+
142
+ for episode in manifest["episodes"]:
143
+ new_ep = int(episode["episode_index"])
144
+ source_task = str(episode["source_task"])
145
+ source_episode = int(episode["source_episode_index"])
146
+ length = int(episode["source_length"])
147
+ global_start = int(episode["global_index_start"])
148
+ src_parquet = Path(episode["source_parquet"])
149
+ dst_parquet = dataset_episode_path(args.dataset, new_ep, chunks_size)
150
+ src_df = pd.read_parquet(src_parquet)
151
+ dst_df = pd.read_parquet(dst_parquet)
152
+ if len(src_df) != length or len(dst_df) != length:
153
+ failures.append(f"episode {new_ep}: length mismatch source={len(src_df)} dst={len(dst_df)} manifest={length}")
154
+ continue
155
+ if list(src_df.columns) != list(dst_df.columns):
156
+ failures.append(f"episode {new_ep}: column mismatch")
157
+ continue
158
+
159
+ for column in src_df.columns:
160
+ if column == "episode_index":
161
+ expected = np.full(length, new_ep, dtype=np.int64)
162
+ if not np.array_equal(dst_df[column].to_numpy(), expected):
163
+ failures.append(f"episode {new_ep}: episode_index remap invalid")
164
+ changed_column_counts[column] += 1
165
+ elif column == "frame_index":
166
+ expected = np.arange(length, dtype=np.int64)
167
+ if not np.array_equal(dst_df[column].to_numpy(), expected):
168
+ failures.append(f"episode {new_ep}: frame_index remap invalid")
169
+ changed_column_counts[column] += 1
170
+ elif column == "index":
171
+ expected = np.arange(global_start, global_start + length, dtype=np.int64)
172
+ if not np.array_equal(dst_df[column].to_numpy(), expected):
173
+ failures.append(f"episode {new_ep}: index remap invalid")
174
+ changed_column_counts[column] += 1
175
+ elif column in TASK_COLUMNS:
176
+ if column in src_df.columns:
177
+ for row_idx, (src_id, dst_id) in enumerate(zip(src_df[column].tolist(), dst_df[column].tolist())):
178
+ src_text = src_tasks[source_task][int(src_id)]
179
+ dst_text = dst_tasks[int(dst_id)]
180
+ if src_text != dst_text:
181
+ failures.append(
182
+ f"episode {new_ep}: {column} row {row_idx} task text mismatch {src_text!r} != {dst_text!r}"
183
+ )
184
+ break
185
+ changed_column_counts[column] += 1
186
+ else:
187
+ ok, reason = series_equal(src_df[column], dst_df[column])
188
+ if not ok:
189
+ failures.append(f"episode {new_ep}: data column {column} changed: {reason}")
190
+ break
191
+
192
+ for feature, dst_video in episode["videos"].items():
193
+ dst_path = Path(dst_video)
194
+ expected_source = Path(episode["resolved_augmented_video_sources"][feature]).resolve()
195
+ if not dst_path.exists():
196
+ failures.append(f"episode {new_ep}: missing output video {dst_path}")
197
+ continue
198
+ if not args.allow_copied_videos and not dst_path.is_symlink():
199
+ failures.append(f"episode {new_ep}: output video is not symlink: {dst_path}")
200
+ if dst_path.resolve() != expected_source:
201
+ failures.append(f"episode {new_ep}: video link mismatch {dst_path.resolve()} != {expected_source}")
202
+ if args.source_root.resolve() in dst_path.resolve().parents:
203
+ failures.append(f"episode {new_ep}: video resolves under source root: {dst_path.resolve()}")
204
+ if args.render_root.resolve() not in expected_source.parents:
205
+ failures.append(f"episode {new_ep}: augmented video source is outside render root: {expected_source}")
206
+ if args.check_video_frames:
207
+ frames = frame_count(dst_path.resolve())
208
+ if frames != length:
209
+ failures.append(f"episode {new_ep}: video frame count {frames} != parquet length {length} for {feature}")
210
+ frame_checks += 1
211
+ video_checks += 1
212
+ checked_rows += length
213
+
214
+ summary = {
215
+ "dataset": str(args.dataset),
216
+ "episodes_checked": len(manifest["episodes"]),
217
+ "rows_checked": checked_rows,
218
+ "video_links_checked": video_checks,
219
+ "video_frame_counts_checked": frame_checks,
220
+ "expected_changed_columns_checked": changed_column_counts,
221
+ "failures": failures[:50],
222
+ "failure_count": len(failures),
223
+ "passed": not failures,
224
+ }
225
+ out = args.dataset / "meta" / "geom_objwise_verification.json"
226
+ out.write_text(json.dumps(summary, indent=2) + "\n")
227
+ print(json.dumps(summary, indent=2))
228
+ if failures:
229
+ raise SystemExit(1)
230
+
231
+
232
+ if __name__ == "__main__":
233
+ main()
eval/eval_groot15_exact_replay.sh ADDED
@@ -0,0 +1,389 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env bash
2
+ set -uEo pipefail
3
+
4
+ # Usually this is the training dataset name used to produce the checkpoint.
5
+ # It is used for default labels only; MODEL_PATH controls the checkpoint.
6
+ DATASETNAME="${DATASETNAME:-PickPlaceCounterToCabinet_150eps_augmented_groot15_v3_100_clean}"
7
+
8
+ MYGROOT_ROOT="${MYGROOT_ROOT:-/home/nvidia/jonghoon/myGR00T}"
9
+ MYGROOT_PY="${MYGROOT_PY:-/data/nvidia/gripper_augmentator/conda-envs/mygr00t/bin/python}"
10
+ ROBOCASA_ROOT="${ROBOCASA_ROOT:-/lp-dev/jonghoon/robocasa_calib/repos/robocasa}"
11
+ ROBOSUITE_ROOT="${ROBOSUITE_ROOT:-/lp-dev/jonghoon/robocasa_calib/repos/robosuite_robocasa_dev}"
12
+ ROBOCASA_PY="${ROBOCASA_PY:-/lp-dev/jonghoon/robocasa_calib/envs/robocasa/bin/python}"
13
+ EVAL_CLIENT="${EVAL_CLIENT:-$MYGROOT_ROOT/scripts/eval_robocasa_replay_state.py}"
14
+
15
+ OUTPUT_BASE="${OUTPUT_BASE:-/lp-dev/jonghoon/myGR00T_outputs}"
16
+ EVAL_OUTPUT_BASE="${EVAL_OUTPUT_BASE:-/lp-dev/jonghoon/isaac-gr00t/eval_results}"
17
+ CHECKPOINT_STEP="${CHECKPOINT_STEP:-30000}"
18
+ MODEL_RUN_NAME="${MODEL_RUN_NAME:-}"
19
+ if [[ -z "${MODEL_PATH:-}" ]]; then
20
+ if [[ -n "$MODEL_RUN_NAME" ]]; then
21
+ MODEL_PATH="${OUTPUT_BASE}/${MODEL_RUN_NAME}/checkpoint-${CHECKPOINT_STEP}"
22
+ else
23
+ MODEL_PATH="${OUTPUT_BASE}/pnpcountertocab_all502_gbs64_wandb_60k_save5k_20260427_193901/checkpoint-60000"
24
+ fi
25
+ fi
26
+
27
+ REPLAY_STATE_ROOT="${REPLAY_STATE_ROOT:-/lp-dev/jonghoon/isaac-gr00t/eval_results/pnpcountertocab_nojar_pseudo100_exact_replay_episode_set_20260430}"
28
+ EVAL_TASK="${EVAL_TASK:-PnPCounterToCab}"
29
+ ENV_NAME="${ENV_NAME:-PickPlaceCounterToCabinet}"
30
+ FINETUNED_TASK="${FINETUNED_TASK:-ft-${ENV_NAME}_${DATASETNAME}_ckpt${CHECKPOINT_STEP}}"
31
+ N_EPISODES="${N_EPISODES:-100}"
32
+ EXPECTED_REPLAY_STATES="${EXPECTED_REPLAY_STATES:-$N_EPISODES}"
33
+ SEED_BASE="${SEED_BASE:-42}"
34
+ MAX_EPISODE_STEPS="${MAX_EPISODE_STEPS:-750}"
35
+ ACTION_HORIZON="${ACTION_HORIZON:-16}"
36
+ GPUS="${GPUS:-0,1,2,3}"
37
+ PORT_BASE="${PORT_BASE:-8777}"
38
+ SAVE_VIDEOS="${SAVE_VIDEOS:-1}"
39
+ POLICY_SEED="${POLICY_SEED-42}"
40
+ RUN_ID="${RUN_ID:-$(date -u +%Y%m%d_%H%M%S)}"
41
+ EVAL_RUN_NAME="${EVAL_RUN_NAME:-mygr00t_${DATASETNAME}_ckpt${CHECKPOINT_STEP}_eval-${EVAL_TASK}_exact${N_EPISODES}_replay_${RUN_ID}}"
42
+ ROOT="${ROOT:-${EVAL_OUTPUT_BASE}/${EVAL_RUN_NAME}}"
43
+ MAIN_LOG="$ROOT/launcher.log"
44
+
45
+ SERVER_PIDS=()
46
+ CLIENT_PIDS=()
47
+ CLEANUP_RAN=0
48
+
49
+ ts() { date -u +"%Y-%m-%dT%H:%M:%SZ"; }
50
+ log() { echo "[$(ts)] $*" | tee -a "$MAIN_LOG"; }
51
+
52
+ cleanup() {
53
+ if [[ "$CLEANUP_RAN" -eq 1 ]]; then
54
+ return
55
+ fi
56
+ CLEANUP_RAN=1
57
+ local pid
58
+ for pid in "${CLIENT_PIDS[@]:-}"; do kill "$pid" 2>/dev/null || true; done
59
+ for pid in "${SERVER_PIDS[@]:-}"; do kill "$pid" 2>/dev/null || true; done
60
+ sleep 2
61
+ for pid in "${CLIENT_PIDS[@]:-}" "${SERVER_PIDS[@]:-}"; do kill -9 "$pid" 2>/dev/null || true; done
62
+ for pid in "${CLIENT_PIDS[@]:-}" "${SERVER_PIDS[@]:-}"; do wait "$pid" 2>/dev/null || true; done
63
+ }
64
+ trap cleanup EXIT
65
+ trap 'cleanup; exit 130' INT
66
+ trap 'cleanup; exit 143' TERM
67
+
68
+ pick_port() {
69
+ local port="$1"
70
+ while ss -ltn "sport = :$port" | grep -q LISTEN; do
71
+ port=$((port + 1))
72
+ done
73
+ echo "$port"
74
+ }
75
+
76
+ wait_for_port() {
77
+ local port="$1"
78
+ for _ in $(seq 1 240); do
79
+ if nc -z 127.0.0.1 "$port" 2>/dev/null; then
80
+ return 0
81
+ fi
82
+ sleep 5
83
+ done
84
+ return 1
85
+ }
86
+
87
+ mkdir -p "$ROOT/workers"
88
+ : > "$MAIN_LOG"
89
+
90
+ if [[ ! -f "$MODEL_PATH/model.safetensors.index.json" ]]; then
91
+ log "FATAL: checkpoint is missing or incomplete: $MODEL_PATH"
92
+ exit 1
93
+ fi
94
+
95
+ if ! python3 - "$REPLAY_STATE_ROOT" "$EXPECTED_REPLAY_STATES" <<'PY'
96
+ import json
97
+ import sys
98
+ from pathlib import Path
99
+
100
+ root = Path(sys.argv[1])
101
+ expected = int(sys.argv[2])
102
+ paths = sorted(root.glob("workers/worker_*/replay_states/episode_*_replay.json"))
103
+ paths.extend(sorted(root.glob("replay_states/episode_*_replay.json")))
104
+ rows = [json.loads(path.read_text()) for path in paths]
105
+ missing = [
106
+ r.get("episode_index")
107
+ for r in rows
108
+ if not Path(str(r.get("model_xml_gz", ""))).is_file()
109
+ or not Path(str(r.get("state_npz", ""))).is_file()
110
+ ]
111
+ raise SystemExit(0 if len(rows) == expected and not missing else 1)
112
+ PY
113
+ then
114
+ log "FATAL: replay source must have $EXPECTED_REPLAY_STATES replay jsons with model XML and state NPZ artifacts: $REPLAY_STATE_ROOT"
115
+ exit 1
116
+ fi
117
+
118
+ IFS=',' read -r -a GPU_LIST <<< "$GPUS"
119
+ N_WORKERS="${#GPU_LIST[@]}"
120
+ if [[ "$N_WORKERS" -lt 1 ]]; then
121
+ log "FATAL: no GPUs specified"
122
+ exit 1
123
+ fi
124
+
125
+ BASE_COUNT=$((N_EPISODES / N_WORKERS))
126
+ REMAINDER=$((N_EPISODES % N_WORKERS))
127
+
128
+ export PYTHONPATH="$MYGROOT_ROOT:${PYTHONPATH:-}"
129
+ CLIENT_PYTHONPATH="$ROBOSUITE_ROOT:$ROBOCASA_ROOT:$MYGROOT_ROOT:${PYTHONPATH:-}"
130
+ export MUJOCO_GL=egl
131
+ export PYOPENGL_PLATFORM=egl
132
+
133
+ python3 - "$ROOT/run_info.json" <<PY
134
+ import json, sys, time
135
+ info = {
136
+ "dataset_name": "$DATASETNAME",
137
+ "model_path": "$MODEL_PATH",
138
+ "finetuned_task": "$FINETUNED_TASK",
139
+ "replay_state_root": "$REPLAY_STATE_ROOT",
140
+ "eval_task": "$EVAL_TASK",
141
+ "env_name": "$ENV_NAME",
142
+ "n_episodes_requested": int("$N_EPISODES"),
143
+ "expected_replay_states": int("$EXPECTED_REPLAY_STATES"),
144
+ "seed_base": int("$SEED_BASE"),
145
+ "gpus": "$GPUS".split(","),
146
+ "n_workers": int("$N_WORKERS"),
147
+ "max_episode_steps": int("$MAX_EPISODE_STEPS"),
148
+ "action_horizon": int("$ACTION_HORIZON"),
149
+ "obj_instance_split": "target",
150
+ "legacy_object_split_label": "B",
151
+ "layout_and_style_ids": [[i, i] for i in range(1, 11)],
152
+ "save_videos": bool(int("$SAVE_VIDEOS")),
153
+ "policy_seed": None if "$POLICY_SEED" == "" else int("$POLICY_SEED"),
154
+ "canonical_episode_definition": "episode_index restores model XML and flattened MuJoCo state from replay_state_root",
155
+ "replay_mode": "exact_model_xml_plus_mujoco_state",
156
+ "exact_replay_artifacts": "source workers/worker_*/replay_states/episode_*.xml.gz plus *_state.npz",
157
+ "codebase": "myGR00T",
158
+ "data_config": "single_panda_gripper",
159
+ "embodiment_tag": "new_embodiment",
160
+ "started_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
161
+ }
162
+ open(sys.argv[1], "w").write(json.dumps(info, indent=2, sort_keys=True) + "\\n")
163
+ PY
164
+
165
+ MANIFEST="$ROOT/seed_shard_manifest.jsonl"
166
+ : > "$MANIFEST"
167
+
168
+ log "launching exact-state replay eval"
169
+ log "root=$ROOT"
170
+ log "dataset_name=$DATASETNAME"
171
+ log "model=$MODEL_PATH"
172
+ log "replay_state_root=$REPLAY_STATE_ROOT"
173
+ log "episodes=$N_EPISODES workers=$N_WORKERS gpus=$GPUS seed_base=$SEED_BASE"
174
+
175
+ POLICY_SERVER_SEED_ARGS=()
176
+ POLICY_CLIENT_SEED_ARGS=()
177
+ if [[ -n "$POLICY_SEED" ]]; then
178
+ POLICY_SERVER_SEED_ARGS+=(--seed "$POLICY_SEED")
179
+ POLICY_CLIENT_SEED_ARGS+=(--policy_seed "$POLICY_SEED")
180
+ fi
181
+
182
+ offset=0
183
+ port="$PORT_BASE"
184
+ WORKER_DIRS=()
185
+ WORKER_COUNTS=()
186
+ WORKER_OFFSETS=()
187
+ WORKER_SEEDS=()
188
+ WORKER_PORTS=()
189
+
190
+ for worker_id in "${!GPU_LIST[@]}"; do
191
+ count="$BASE_COUNT"
192
+ if [[ "$worker_id" -lt "$REMAINDER" ]]; then
193
+ count=$((count + 1))
194
+ fi
195
+ seed=$((SEED_BASE + worker_id))
196
+ gpu="${GPU_LIST[$worker_id]}"
197
+ worker_dir="$ROOT/workers/worker_$(printf '%02d' "$worker_id")_seed_${seed}"
198
+ mkdir -p "$worker_dir"
199
+ for local_idx in $(seq 0 $((count - 1))); do
200
+ printf '{"episode_index":%d,"worker_id":"%02d","episode_seed":%d,"local_episode_index":%d}\n' \
201
+ "$((offset + local_idx))" "$worker_id" "$seed" "$local_idx" >> "$MANIFEST"
202
+ done
203
+
204
+ port="$(pick_port "$port")"
205
+ WORKER_DIRS+=("$worker_dir")
206
+ WORKER_COUNTS+=("$count")
207
+ WORKER_OFFSETS+=("$offset")
208
+ WORKER_SEEDS+=("$seed")
209
+ WORKER_PORTS+=("$port")
210
+
211
+ log "starting server worker=$worker_id gpu=$gpu seed=$seed episodes=$count offset=$offset port=$port"
212
+ (
213
+ cd "$MYGROOT_ROOT"
214
+ OMP_NUM_THREADS=4 MKL_NUM_THREADS=4 OPENBLAS_NUM_THREADS=4 \
215
+ CUDA_VISIBLE_DEVICES="$gpu" "$MYGROOT_PY" scripts/inference_service.py \
216
+ --server \
217
+ --port "$port" \
218
+ --model_path "$MODEL_PATH" \
219
+ --data_config single_panda_gripper \
220
+ --embodiment_tag new_embodiment \
221
+ "${POLICY_SERVER_SEED_ARGS[@]}"
222
+ ) > "$worker_dir/server.log" 2>&1 &
223
+ SERVER_PIDS+=("$!")
224
+ echo "${SERVER_PIDS[-1]}" > "$worker_dir/server.pid"
225
+
226
+ offset=$((offset + count))
227
+ port=$((port + 1))
228
+ done
229
+
230
+ for worker_id in "${!WORKER_PORTS[@]}"; do
231
+ port="${WORKER_PORTS[$worker_id]}"
232
+ worker_dir="${WORKER_DIRS[$worker_id]}"
233
+ if ! wait_for_port "$port"; then
234
+ log "FATAL: policy server for worker=$worker_id did not open port=$port"
235
+ tail -n 80 "$worker_dir/server.log" | tee -a "$MAIN_LOG" || true
236
+ cleanup
237
+ exit 1
238
+ fi
239
+ log "server ready worker=$worker_id port=$port"
240
+ done
241
+
242
+ for worker_id in "${!GPU_LIST[@]}"; do
243
+ gpu="${GPU_LIST[$worker_id]}"
244
+ port="${WORKER_PORTS[$worker_id]}"
245
+ worker_dir="${WORKER_DIRS[$worker_id]}"
246
+ count="${WORKER_COUNTS[$worker_id]}"
247
+ offset="${WORKER_OFFSETS[$worker_id]}"
248
+ seed="${WORKER_SEEDS[$worker_id]}"
249
+ SAVE_VIDEO_ARGS=()
250
+ if [[ "$SAVE_VIDEOS" -eq 1 ]]; then
251
+ SAVE_VIDEO_ARGS+=(--save_videos)
252
+ fi
253
+ log "starting client worker=$worker_id gpu=$gpu seed=$seed episodes=$count offset=$offset port=$port"
254
+ PYTHONPATH="$CLIENT_PYTHONPATH" CUDA_VISIBLE_DEVICES="$gpu" "$ROBOCASA_PY" "$EVAL_CLIENT" \
255
+ --host 127.0.0.1 \
256
+ --port "$port" \
257
+ --model_path "$MODEL_PATH" \
258
+ --output_dir "$worker_dir" \
259
+ --env_name "$ENV_NAME" \
260
+ --eval_task "$EVAL_TASK" \
261
+ --finetuned_task "$FINETUNED_TASK" \
262
+ --replay_state_root "$REPLAY_STATE_ROOT" \
263
+ --n_episodes "$count" \
264
+ --episode_offset "$offset" \
265
+ --worker_id "$(printf '%02d' "$worker_id")" \
266
+ --seed "$seed" \
267
+ --max_episode_steps "$MAX_EPISODE_STEPS" \
268
+ --action_horizon "$ACTION_HORIZON" \
269
+ --robots PandaOmron \
270
+ --generative_textures \
271
+ "${POLICY_CLIENT_SEED_ARGS[@]}" \
272
+ "${SAVE_VIDEO_ARGS[@]}" \
273
+ > "$worker_dir/client.log" 2>&1 &
274
+ CLIENT_PIDS+=("$!")
275
+ echo "${CLIENT_PIDS[-1]}" > "$worker_dir/client.pid"
276
+ done
277
+
278
+ client_rc=0
279
+ for worker_id in "${!CLIENT_PIDS[@]}"; do
280
+ pid="${CLIENT_PIDS[$worker_id]}"
281
+ if wait "$pid"; then
282
+ log "client finished worker=$worker_id rc=0"
283
+ else
284
+ rc="$?"
285
+ log "client failed worker=$worker_id rc=$rc"
286
+ client_rc="$rc"
287
+ fi
288
+ done
289
+
290
+ cleanup
291
+
292
+ if [[ "$client_rc" -ne 0 ]]; then
293
+ log "FATAL: at least one worker failed"
294
+ exit "$client_rc"
295
+ fi
296
+
297
+ python3 - "$ROOT" <<'PY'
298
+ import csv
299
+ import json
300
+ import sys
301
+ import time
302
+ from pathlib import Path
303
+
304
+ root = Path(sys.argv[1])
305
+ workers = sorted((root / "workers").glob("worker_*"))
306
+
307
+ def load_jsonl(path):
308
+ rows = []
309
+ if path.exists():
310
+ with path.open() as f:
311
+ for line in f:
312
+ line = line.strip()
313
+ if line:
314
+ rows.append(json.loads(line))
315
+ return rows
316
+
317
+ episodes = []
318
+ full_meta = []
319
+ for worker in workers:
320
+ episodes.extend(load_jsonl(worker / "episodes.jsonl"))
321
+ full_meta.extend(load_jsonl(worker / "episodes_full_meta.jsonl"))
322
+
323
+ episodes.sort(key=lambda r: int(r["episode_index"]))
324
+ full_meta.sort(key=lambda r: int(r["episode_index"]))
325
+
326
+ with (root / "episodes.jsonl").open("w") as f:
327
+ for row in episodes:
328
+ f.write(json.dumps(row, sort_keys=True) + "\n")
329
+
330
+ with (root / "episodes_full_meta.jsonl").open("w") as f:
331
+ for row in full_meta:
332
+ f.write(json.dumps(row, sort_keys=True) + "\n")
333
+
334
+ fieldnames = [
335
+ "episode_index",
336
+ "local_episode_index",
337
+ "worker_id",
338
+ "episode_seed",
339
+ "success",
340
+ "reward",
341
+ "policy_steps",
342
+ "sim_step_budget",
343
+ "video_path",
344
+ "layout_id",
345
+ "style_id",
346
+ "language",
347
+ "target_mesh_keys",
348
+ ]
349
+ with (root / "simulation_results.csv").open("w", newline="") as f:
350
+ writer = csv.DictWriter(f, fieldnames=fieldnames)
351
+ writer.writeheader()
352
+ for row in episodes:
353
+ writer.writerow(
354
+ {
355
+ "episode_index": row.get("episode_index"),
356
+ "local_episode_index": row.get("local_episode_index"),
357
+ "worker_id": row.get("worker_id"),
358
+ "episode_seed": row.get("episode_seed"),
359
+ "success": int(bool(row.get("success"))),
360
+ "reward": row.get("reward"),
361
+ "policy_steps": row.get("policy_steps"),
362
+ "sim_step_budget": row.get("sim_step_budget"),
363
+ "video_path": row.get("video_path", ""),
364
+ "layout_id": row.get("layout_id"),
365
+ "style_id": row.get("style_id"),
366
+ "language": row.get("language"),
367
+ "target_mesh_keys": ";".join(
368
+ str(obj.get("mesh_key"))
369
+ for obj in row.get("objects", [])
370
+ if obj.get("mesh_key")
371
+ ),
372
+ }
373
+ )
374
+
375
+ run_info = json.loads((root / "run_info.json").read_text())
376
+ n_success = sum(1 for row in episodes if row.get("success"))
377
+ summary = {
378
+ **run_info,
379
+ "finished_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
380
+ "n_episodes": len(episodes),
381
+ "n_success": int(n_success),
382
+ "success_rate": float(n_success / max(len(episodes), 1)),
383
+ }
384
+ (root / "summary.json").write_text(json.dumps(summary, indent=2, sort_keys=True) + "\n")
385
+ print(json.dumps(summary, indent=2, sort_keys=True))
386
+ PY
387
+
388
+ log "merged worker outputs"
389
+ log "done root=$ROOT"
eval/eval_robocasa_replay_state_3view.py ADDED
@@ -0,0 +1,590 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """Run RoboCasa eval from saved model XML + MuJoCo state artifacts.
3
+
4
+ This is the strict replay path for apples-to-apples evaluation after a
5
+ reconstructed episode bank has been created by `eval_robocasa_reconstruct_xyz.py`.
6
+ Each episode is restored from:
7
+
8
+ - `replay_states/episode_XXXXXX.xml.gz`
9
+ - `replay_states/episode_XXXXXX_state.npz`
10
+ - `replay_states/episode_XXXXXX_replay.json`
11
+ """
12
+
13
+ from __future__ import annotations
14
+
15
+ import argparse
16
+ import csv
17
+ import gzip
18
+ import json
19
+ import pickle
20
+ import time
21
+ from io import BytesIO
22
+ from pathlib import Path
23
+ from typing import Any
24
+
25
+ import gymnasium as gym
26
+ import numpy as np
27
+ import robocasa # noqa: F401 - registers RoboCasa environments with robosuite
28
+ import torch
29
+ import zmq
30
+ from tqdm import trange
31
+
32
+ from gr00t.eval.wrappers.multistep_wrapper import MultiStepWrapper
33
+ from gr00t.eval.wrappers.record_video import RecordVideo
34
+ from gr00t.eval.wrappers.robocasa_wrapper import RoboCasaWrapper, load_robocasa_gym_env
35
+
36
+ TARGET_LAYOUT_AND_STYLE_IDS = [[i, i] for i in range(1, 11)]
37
+ TARGET_OBJ_INSTANCE_SPLIT = "target"
38
+ ACTION_SEED_KEY = "_policy_action_seed"
39
+ ACTION_SEED_EPISODE_STRIDE = 1_000_003
40
+ TORCH_SEED_MODULUS = 2**63 - 1
41
+
42
+
43
+ class PolicyClient:
44
+ """Minimal client for myGR00T's RobotInferenceServer."""
45
+
46
+ def __init__(self, host: str, port: int):
47
+ self.context = zmq.Context()
48
+ self.socket = self.context.socket(zmq.REQ)
49
+ self.socket.connect(f"tcp://{host}:{port}")
50
+
51
+ @staticmethod
52
+ def to_bytes(data: dict[str, Any]) -> bytes:
53
+ buffer = BytesIO()
54
+ torch.save(data, buffer)
55
+ return buffer.getvalue()
56
+
57
+ @staticmethod
58
+ def from_bytes(data: bytes) -> dict[str, Any]:
59
+ buffer = BytesIO(data)
60
+ return torch.load(buffer, weights_only=False)
61
+
62
+ def call(self, endpoint: str, data: dict[str, Any] | None = None) -> dict[str, Any]:
63
+ request: dict[str, Any] = {"endpoint": endpoint}
64
+ if data is not None:
65
+ request["data"] = data
66
+ self.socket.send(self.to_bytes(request))
67
+ message = self.socket.recv()
68
+ if message == b"ERROR":
69
+ raise RuntimeError("policy server returned ERROR")
70
+ return self.from_bytes(message)
71
+
72
+ def get_action(self, observations: dict[str, Any]) -> dict[str, Any]:
73
+ return self.call("get_action", observations)
74
+
75
+ def close(self) -> None:
76
+ self.socket.close()
77
+ self.context.term()
78
+
79
+
80
+ def as_bool(value: Any) -> bool:
81
+ if isinstance(value, np.ndarray):
82
+ return bool(np.any(value))
83
+ if isinstance(value, (list, tuple)):
84
+ return any(as_bool(v) for v in value)
85
+ if isinstance(value, dict):
86
+ return any(as_bool(v) for v in value.values())
87
+ return bool(value)
88
+
89
+
90
+ def json_default(value: Any) -> Any:
91
+ if isinstance(value, np.ndarray):
92
+ return value.tolist()
93
+ if isinstance(value, np.generic):
94
+ return value.item()
95
+ if isinstance(value, Path):
96
+ return str(value)
97
+ return str(value)
98
+
99
+
100
+ def find_robocasa_inner(env: Any) -> Any | None:
101
+ cur = env
102
+ for _ in range(80):
103
+ if cur is None:
104
+ return None
105
+ nxt = getattr(cur, "env", None)
106
+ if nxt is None or nxt is cur:
107
+ break
108
+ cur = nxt
109
+ if hasattr(cur, "sim") and hasattr(cur, "get_ep_meta"):
110
+ return cur
111
+ return None
112
+
113
+
114
+ def parse_mjcf_path(mjcf_path: str | None) -> dict[str, str | None]:
115
+ if not mjcf_path:
116
+ return {"registry": None, "instance_id": None, "mesh_key": None}
117
+ path = Path(str(mjcf_path))
118
+ parts = path.parts
119
+ registry = None
120
+ for candidate in ("objaverse", "aigen_objs", "lightwheel"):
121
+ if candidate in parts:
122
+ registry = candidate
123
+ break
124
+ instance_id = path.parent.name if path.name.endswith(".xml") else None
125
+ mesh_key = None
126
+ if registry and registry in parts and instance_id:
127
+ idx = parts.index(registry)
128
+ if registry == "lightwheel" and idx + 1 < len(parts):
129
+ mesh_key = "/".join((parts[idx + 1], instance_id))
130
+ elif idx + 1 < len(parts):
131
+ mesh_key = "/".join(parts[idx + 1 : -1])
132
+ return {"registry": registry, "instance_id": instance_id, "mesh_key": mesh_key}
133
+
134
+
135
+ def summarize_ep_meta(ep_meta: dict[str, Any]) -> dict[str, Any]:
136
+ summary: dict[str, Any] = {
137
+ "layout_id": ep_meta.get("layout_id"),
138
+ "style_id": ep_meta.get("style_id"),
139
+ "language": ep_meta.get("lang", ""),
140
+ "objects": [],
141
+ }
142
+ for cfg in ep_meta.get("object_cfgs", []) or []:
143
+ if not isinstance(cfg, dict):
144
+ continue
145
+ info = cfg.get("info", {}) if isinstance(cfg.get("info", {}), dict) else {}
146
+ mjcf_path = info.get("mjcf_path")
147
+ parsed = parse_mjcf_path(mjcf_path)
148
+ summary["objects"].append(
149
+ {
150
+ "name": cfg.get("name"),
151
+ "category": info.get("cat"),
152
+ "mjcf_path": mjcf_path,
153
+ "registry": parsed["registry"],
154
+ "instance_id": parsed["instance_id"],
155
+ "mesh_key": parsed["mesh_key"],
156
+ }
157
+ )
158
+ fixtures = ep_meta.get("fixtures")
159
+ if isinstance(fixtures, dict):
160
+ summary["fixture_names"] = list(fixtures.keys())
161
+ return summary
162
+
163
+
164
+ def capture_ep_meta(env: Any, pickle_path: Path) -> tuple[dict[str, Any], dict[str, Any]]:
165
+ inner = find_robocasa_inner(env)
166
+ if inner is None:
167
+ return {}, {"error": "could_not_find_robocasa_inner_env"}
168
+ try:
169
+ ep_meta = inner.get_ep_meta() or {}
170
+ except Exception as exc:
171
+ return {}, {"error": f"get_ep_meta_failed: {type(exc).__name__}: {exc}"}
172
+ with pickle_path.open("wb") as f:
173
+ pickle.dump(ep_meta, f)
174
+ return ep_meta, summarize_ep_meta(ep_meta)
175
+
176
+
177
+ def write_jsonl(path: Path, row: dict[str, Any]) -> None:
178
+ with path.open("a") as f:
179
+ f.write(json.dumps(row, default=json_default, sort_keys=True) + "\n")
180
+ f.flush()
181
+
182
+
183
+ def make_policy_action_seed(policy_seed: int, episode_index: int, policy_step: int) -> int:
184
+ seed = int(policy_seed) + int(episode_index) * ACTION_SEED_EPISODE_STRIDE + int(policy_step)
185
+ return seed % TORCH_SEED_MODULUS
186
+
187
+
188
+ def resolve_artifact_path(value: str, state_json_path: Path) -> Path:
189
+ path = Path(value)
190
+ if path.is_file():
191
+ return path
192
+ candidate = state_json_path.parent / path.name
193
+ if candidate.is_file():
194
+ return candidate
195
+ raise FileNotFoundError(f"missing replay artifact: {value}")
196
+
197
+
198
+ def load_replay_state_index(root: Path) -> dict[int, dict[str, Any]]:
199
+ if not root.is_dir():
200
+ raise FileNotFoundError(f"missing replay_state_root: {root}")
201
+ paths = sorted(root.glob("workers/worker_*/replay_states/episode_*_replay.json"))
202
+ paths.extend(sorted(root.glob("replay_states/episode_*_replay.json")))
203
+ index: dict[int, dict[str, Any]] = {}
204
+ for path in paths:
205
+ row = json.loads(path.read_text())
206
+ episode_index = int(row["episode_index"])
207
+ row["replay_json"] = str(path)
208
+ row["model_xml_gz"] = str(resolve_artifact_path(str(row["model_xml_gz"]), path))
209
+ row["state_npz"] = str(resolve_artifact_path(str(row["state_npz"]), path))
210
+ index[episode_index] = row
211
+ if not index:
212
+ raise FileNotFoundError(f"no replay state jsons found under {root}")
213
+ return index
214
+
215
+
216
+ def load_ep_meta_from_state_info(state_info: dict[str, Any]) -> dict[str, Any] | None:
217
+ for key in ("regenerated_ep_meta_pickle", "source_ep_meta_pickle", "ep_meta_pickle"):
218
+ path_value = state_info.get(key)
219
+ if not path_value:
220
+ continue
221
+ path = Path(str(path_value))
222
+ if not path.is_file():
223
+ continue
224
+ with path.open("rb") as f:
225
+ return pickle.load(f)
226
+ return None
227
+
228
+
229
+ def build_fresh_groot_obs(robo_wrapper: RoboCasaWrapper, inner: Any) -> dict[str, Any]:
230
+ gym_env = robo_wrapper.env
231
+ if getattr(inner, "viewer_get_obs", False) and getattr(inner, "viewer", None) is not None:
232
+ raw_obs = inner.viewer._get_observations(force_update=True)
233
+ else:
234
+ raw_obs = inner._get_observations(force_update=True)
235
+ if getattr(gym_env, "flatten_obs", False):
236
+ obs = gym_env._flatten_obs(raw_obs)
237
+ else:
238
+ obs = gym_env._filter_obs(raw_obs)
239
+ new_obs: dict[str, Any] = {}
240
+ for key, value in obs.items():
241
+ if key in robo_wrapper._robocasa_keys_to_gr00t_keys:
242
+ new_obs[robo_wrapper._robocasa_keys_to_gr00t_keys[key]] = value
243
+ new_obs["annotation.human.action.task_description"] = [robo_wrapper.language_instruction]
244
+ return new_obs
245
+
246
+
247
+ class ExactStateReplayWrapper(gym.Wrapper):
248
+ def __init__(self, env: RoboCasaWrapper, replay_state_index: dict[int, dict[str, Any]]):
249
+ super().__init__(env)
250
+ self.replay_state_index = replay_state_index
251
+ self.current_episode_index: int | None = None
252
+ self.last_state_info: dict[str, Any] = {}
253
+
254
+ def set_episode_index(self, episode_index: int) -> None:
255
+ self.current_episode_index = int(episode_index)
256
+
257
+ def reset(self, seed=None, options=None):
258
+ if self.current_episode_index is None:
259
+ raise RuntimeError("set_episode_index must be called before reset")
260
+ if self.current_episode_index not in self.replay_state_index:
261
+ raise KeyError(f"missing replay state for episode_index={self.current_episode_index}")
262
+ state_info = self.replay_state_index[self.current_episode_index]
263
+ inner = find_robocasa_inner(self.env)
264
+ if inner is None or not hasattr(inner, "reset_from_xml_string"):
265
+ raise RuntimeError("could not find RoboCasa inner env with reset_from_xml_string")
266
+
267
+ ep_meta = load_ep_meta_from_state_info(state_info)
268
+ if ep_meta is not None and hasattr(inner, "set_ep_meta"):
269
+ inner.set_ep_meta(ep_meta)
270
+
271
+ with gzip.open(Path(state_info["model_xml_gz"]), "rt") as f:
272
+ model_xml = f.read()
273
+ saved = np.load(Path(state_info["state_npz"]))
274
+ saved_state = saved["states"]
275
+ saved_ctrl = saved["ctrl"] if "ctrl" in saved.files else None
276
+
277
+ inner.reset()
278
+ if ep_meta is not None and hasattr(inner, "set_ep_meta"):
279
+ inner.set_ep_meta(ep_meta)
280
+ inner.reset_from_xml_string(model_xml)
281
+ inner.sim.reset()
282
+ inner.sim.set_state_from_flattened(saved_state)
283
+ if saved_ctrl is not None and saved_ctrl.shape == inner.sim.data.ctrl.shape:
284
+ inner.sim.data.ctrl[:] = saved_ctrl
285
+ inner.sim.forward()
286
+
287
+ # --- FIX B: re-sync each part controller to the RESTORED state -------------
288
+ # reset_from_xml_string() above triggers a throwaway env reset, during which
289
+ # every controller latches initial_joint (used for the OSC nullspace bias) and
290
+ # goal_pos from the robot's freshly *randomised* init pose (robot.py: init_qpos
291
+ # += gaussian noise, magnitude 0.02). set_state_from_flattened() then teleports
292
+ # the robot to the episode's saved pose, but nothing re-syncs the controllers,
293
+ # so they keep biasing toward a pose this episode never had -- and that stale
294
+ # pose depends on the client --seed, which leaks worker sharding into results.
295
+ # Re-seed each controller from the restored qpos instead.
296
+ for _robot in getattr(inner, "robots", []) or []:
297
+ _cc = getattr(_robot, "composite_controller", None)
298
+ if _cc is None:
299
+ continue
300
+ for _ctrl in getattr(_cc, "part_controllers", {}).values():
301
+ _qidx = getattr(_ctrl, "qpos_index", None)
302
+ if _qidx is None or getattr(_ctrl, "ref_name", None) is None:
303
+ continue # e.g. MobileBaseJointVelocityController has no ref_pos
304
+ try:
305
+ _ctrl.update_initial_joints(inner.sim.data.qpos[_qidx])
306
+ except Exception as _exc:
307
+ print(f"[fixB] update_initial_joints skipped: {_exc}", flush=True)
308
+ try:
309
+ _cc.reset() # reset_goal() on every part controller
310
+ except Exception as _exc:
311
+ print(f"[fixB] composite reset skipped: {_exc}", flush=True)
312
+ inner.sim.forward()
313
+ # --- end FIX B -------------------------------------------------------------
314
+
315
+ obs = build_fresh_groot_obs(self.env, inner)
316
+ info = {
317
+ "is_success": self.env.is_success()["task"],
318
+ "replay_state_npz": state_info["state_npz"],
319
+ "replay_model_xml_gz": state_info["model_xml_gz"],
320
+ "replay_state_json": state_info["replay_json"],
321
+ "replay_state_flat_sha256": state_info.get("state_flat_sha256"),
322
+ "source_episode_index": state_info.get("source_episode_index"),
323
+ "source_episode_seed": state_info.get("source_episode_seed"),
324
+ "source_worker_id": state_info.get("source_worker_id"),
325
+ "source_local_episode_index": state_info.get("source_local_episode_index"),
326
+ "source_ep_meta_pickle": state_info.get("source_ep_meta_pickle"),
327
+ "regenerated_ep_meta_pickle": state_info.get("regenerated_ep_meta_pickle"),
328
+ }
329
+ self.last_state_info = state_info
330
+ return obs, info
331
+
332
+
333
+ def main() -> None:
334
+ parser = argparse.ArgumentParser()
335
+ parser.add_argument("--host", default="127.0.0.1")
336
+ parser.add_argument("--port", type=int, required=True)
337
+ parser.add_argument("--model_path", required=True)
338
+ parser.add_argument("--output_dir", required=True)
339
+ parser.add_argument("--env_name", default="PickPlaceCounterToCabinet")
340
+ parser.add_argument("--eval_task", default="PnPCounterToCab")
341
+ parser.add_argument("--finetuned_task", required=True)
342
+ parser.add_argument("--n_episodes", type=int, default=200)
343
+ parser.add_argument("--episode_offset", type=int, default=0)
344
+ parser.add_argument("--worker_id", default="0")
345
+ parser.add_argument("--seed", type=int, default=42)
346
+ parser.add_argument("--max_episode_steps", type=int, default=750)
347
+ parser.add_argument("--action_horizon", type=int, default=16)
348
+ parser.add_argument("--robots", default="PandaOmron")
349
+ parser.add_argument("--generative_textures", action="store_true")
350
+ parser.add_argument("--save_videos", action="store_true")
351
+ parser.add_argument("--replay_state_root", required=True)
352
+ parser.add_argument("--policy_seed", type=int, default=None)
353
+ args = parser.parse_args()
354
+
355
+ out_dir = Path(args.output_dir)
356
+ video_dir = out_dir / "videos"
357
+ meta_pickle_dir = out_dir / "ep_meta_pickles"
358
+ out_dir.mkdir(parents=True, exist_ok=True)
359
+ if args.save_videos:
360
+ video_dir.mkdir(parents=True, exist_ok=True)
361
+ meta_pickle_dir.mkdir(parents=True, exist_ok=True)
362
+
363
+ replay_state_root = Path(args.replay_state_root)
364
+ replay_state_index = load_replay_state_index(replay_state_root)
365
+
366
+ run_info = {
367
+ "model_path": args.model_path,
368
+ "finetuned_task": args.finetuned_task,
369
+ "eval_task": args.eval_task,
370
+ "env_name": args.env_name,
371
+ "n_episodes_requested": args.n_episodes,
372
+ "episode_offset": args.episode_offset,
373
+ "worker_id": args.worker_id,
374
+ "seed": args.seed,
375
+ "max_episode_steps": args.max_episode_steps,
376
+ "action_horizon": args.action_horizon,
377
+ "robots": args.robots,
378
+ "layout": None,
379
+ "style": None,
380
+ "layout_and_style_ids": TARGET_LAYOUT_AND_STYLE_IDS,
381
+ "obj_instance_split": TARGET_OBJ_INSTANCE_SPLIT,
382
+ "generative_textures": bool(args.generative_textures),
383
+ "save_videos": bool(args.save_videos),
384
+ "policy_seed": args.policy_seed,
385
+ "policy_action_seed_key": ACTION_SEED_KEY if args.policy_seed is not None else None,
386
+ "policy_action_seed_formula": (
387
+ f"(policy_seed + episode_index * {ACTION_SEED_EPISODE_STRIDE} + policy_step)"
388
+ f" % {TORCH_SEED_MODULUS}"
389
+ if args.policy_seed is not None
390
+ else None
391
+ ),
392
+ "codebase": "myGR00T",
393
+ "data_config": "single_panda_gripper",
394
+ "embodiment_tag": "new_embodiment",
395
+ "replay_state_root": str(replay_state_root),
396
+ "replay_state_count": len(replay_state_index),
397
+ "replay_mode": "exact_model_xml_plus_mujoco_state",
398
+ "canonical_episode_definition": "episode_index restores model XML and flattened MuJoCo state from replay_state_root",
399
+ "started_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
400
+ }
401
+ (out_dir / "run_info.json").write_text(json.dumps(run_info, indent=2, sort_keys=True) + "\n")
402
+
403
+ client = PolicyClient(host=args.host, port=args.port)
404
+ env = load_robocasa_gym_env(
405
+ args.env_name,
406
+ seed=args.seed,
407
+ robots=args.robots,
408
+ camera_widths=256,
409
+ camera_heights=256,
410
+ render_onscreen=False,
411
+ obj_instance_split=TARGET_OBJ_INSTANCE_SPLIT,
412
+ generative_textures="100p" if args.generative_textures else None,
413
+ randomize_cameras=False,
414
+ layout_and_style_ids=TARGET_LAYOUT_AND_STYLE_IDS,
415
+ layout_ids=None,
416
+ style_ids=None,
417
+ )
418
+ env = ExactStateReplayWrapper(RoboCasaWrapper(env), replay_state_index)
419
+ replay_wrapper = env
420
+ if args.save_videos:
421
+ env = RecordVideo(
422
+ env,
423
+ str(video_dir),
424
+ disable_logger=True,
425
+ episode_trigger=lambda episode_id: True,
426
+ fps=20,
427
+ name_prefix=args.eval_task,
428
+ )
429
+ env = MultiStepWrapper(
430
+ env,
431
+ video_delta_indices=np.arange(1),
432
+ state_delta_indices=np.arange(1),
433
+ n_action_steps=args.action_horizon,
434
+ max_episode_steps=args.max_episode_steps,
435
+ )
436
+
437
+ episodes_jsonl = out_dir / "episodes.jsonl"
438
+ full_meta_jsonl = out_dir / "episodes_full_meta.jsonl"
439
+ csv_path = out_dir / "simulation_results.csv"
440
+ fieldnames = [
441
+ "episode_index",
442
+ "local_episode_index",
443
+ "worker_id",
444
+ "episode_seed",
445
+ "success",
446
+ "reward",
447
+ "policy_steps",
448
+ "sim_step_budget",
449
+ "video_path",
450
+ "layout_id",
451
+ "style_id",
452
+ "language",
453
+ "target_mesh_keys",
454
+ ]
455
+ with csv_path.open("w", newline="") as f:
456
+ csv.DictWriter(f, fieldnames=fieldnames).writeheader()
457
+
458
+ successes: list[bool] = []
459
+ try:
460
+ for local_episode_index in trange(args.n_episodes, desc=f"{args.eval_task}/w{args.worker_id}"):
461
+ episode_index = args.episode_offset + local_episode_index
462
+ replay_wrapper.set_episode_index(episode_index)
463
+ obs, info = env.reset()
464
+ reset_info = dict(info)
465
+ ep_meta_pickle = meta_pickle_dir / f"episode_{episode_index:06d}.pkl"
466
+ ep_meta, ep_summary = capture_ep_meta(env, ep_meta_pickle)
467
+ done = False
468
+ policy_steps = 0
469
+ reward_total = 0.0
470
+ last_info: dict[str, Any] = info
471
+
472
+ view_frames = {"wrist_view": []} # center is already covered by RecordVideo
473
+
474
+ while not done:
475
+ for _vk in view_frames:
476
+ _f = obs.get(f"video.{_vk}")
477
+ if _f is not None:
478
+ _f = np.asarray(_f)
479
+ view_frames[_vk].append(_f[0] if _f.ndim == 4 else _f)
480
+ policy_obs = dict(obs)
481
+ policy_obs["video.left_view"] = np.flip(obs["video.left_view"], axis=1)
482
+ policy_obs["video.right_view"] = np.flip(obs["video.right_view"], axis=1)
483
+ policy_obs["video.wrist_view"] = np.flip(obs["video.wrist_view"], axis=1)
484
+ if args.policy_seed is not None:
485
+ policy_obs[ACTION_SEED_KEY] = make_policy_action_seed(
486
+ args.policy_seed,
487
+ episode_index,
488
+ policy_steps,
489
+ )
490
+ action = client.get_action(policy_obs)
491
+ post_action = {}
492
+ for key, value in action.items():
493
+ post_action[key] = value[..., None] if getattr(value, "ndim", 0) == 1 else value
494
+ obs, reward, terminated, truncated, last_info = env.step(post_action)
495
+ reward_total += float(np.max(reward) if isinstance(reward, np.ndarray) else reward)
496
+ policy_steps += 1
497
+ done = bool(terminated or truncated)
498
+
499
+ if args.save_videos:
500
+ import imageio
501
+ ep_dir = video_dir / f"episode_{episode_index:06d}"
502
+ for _vk, _frames in view_frames.items():
503
+ if not _frames:
504
+ continue
505
+ ep_dir.mkdir(parents=True, exist_ok=True)
506
+ imageio.mimwrite(str(ep_dir / "wrist.mp4"), _frames, fps=20,
507
+ codec="libx264", macro_block_size=1,
508
+ ffmpeg_params=["-pix_fmt", "yuv420p"])
509
+
510
+ success = as_bool(last_info.get("is_success", False))
511
+ successes.append(success)
512
+ video_path = (
513
+ str(video_dir / f"{args.eval_task}-episode_{local_episode_index}.mp4")
514
+ if args.save_videos
515
+ else ""
516
+ )
517
+ row = {
518
+ "episode_index": episode_index,
519
+ "local_episode_index": local_episode_index,
520
+ "worker_id": args.worker_id,
521
+ "episode_seed": args.seed,
522
+ "canonical_episode_id": f"state_replay_episode_{episode_index:06d}",
523
+ "success": success,
524
+ "reward": reward_total,
525
+ "policy_steps": policy_steps,
526
+ "sim_step_budget": policy_steps * args.action_horizon,
527
+ "video_path": video_path,
528
+ "ep_meta_pickle": str(ep_meta_pickle),
529
+ "replay_state_npz": reset_info.get("replay_state_npz"),
530
+ "replay_model_xml_gz": reset_info.get("replay_model_xml_gz"),
531
+ "replay_state_json": reset_info.get("replay_state_json"),
532
+ "replay_state_flat_sha256": reset_info.get("replay_state_flat_sha256"),
533
+ "source_ep_meta_pickle": reset_info.get("source_ep_meta_pickle"),
534
+ "regenerated_ep_meta_pickle": reset_info.get("regenerated_ep_meta_pickle"),
535
+ "model_path": args.model_path,
536
+ "finetuned_task": args.finetuned_task,
537
+ "eval_task": args.eval_task,
538
+ "env_name": args.env_name,
539
+ "seed": args.seed,
540
+ **ep_summary,
541
+ }
542
+ write_jsonl(episodes_jsonl, row)
543
+ write_jsonl(
544
+ full_meta_jsonl,
545
+ {
546
+ "episode_index": episode_index,
547
+ "ep_meta_pickle": str(ep_meta_pickle),
548
+ "ep_meta": ep_meta,
549
+ },
550
+ )
551
+ with csv_path.open("a", newline="") as f:
552
+ writer = csv.DictWriter(f, fieldnames=fieldnames)
553
+ writer.writerow(
554
+ {
555
+ "episode_index": row["episode_index"],
556
+ "local_episode_index": row["local_episode_index"],
557
+ "worker_id": row["worker_id"],
558
+ "episode_seed": row["episode_seed"],
559
+ "success": int(success),
560
+ "reward": reward_total,
561
+ "policy_steps": policy_steps,
562
+ "sim_step_budget": policy_steps * args.action_horizon,
563
+ "video_path": video_path,
564
+ "layout_id": ep_summary.get("layout_id"),
565
+ "style_id": ep_summary.get("style_id"),
566
+ "language": ep_summary.get("language"),
567
+ "target_mesh_keys": ";".join(
568
+ str(obj.get("mesh_key"))
569
+ for obj in ep_summary.get("objects", [])
570
+ if obj.get("mesh_key")
571
+ ),
572
+ }
573
+ )
574
+ finally:
575
+ env.close()
576
+ client.close()
577
+
578
+ summary = {
579
+ **run_info,
580
+ "finished_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
581
+ "n_episodes": len(successes),
582
+ "n_success": int(sum(successes)),
583
+ "success_rate": float(sum(successes) / max(len(successes), 1)),
584
+ }
585
+ (out_dir / "summary.json").write_text(json.dumps(summary, indent=2, sort_keys=True) + "\n")
586
+ print(json.dumps(summary, indent=2, sort_keys=True))
587
+
588
+
589
+ if __name__ == "__main__":
590
+ main()
eval/eval_robocasa_replay_state_grasp.py ADDED
@@ -0,0 +1,633 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """Run RoboCasa eval from saved model XML + MuJoCo state artifacts.
3
+
4
+ This is the strict replay path for apples-to-apples evaluation after a
5
+ reconstructed episode bank has been created by `eval_robocasa_reconstruct_xyz.py`.
6
+ Each episode is restored from:
7
+
8
+ - `replay_states/episode_XXXXXX.xml.gz`
9
+ - `replay_states/episode_XXXXXX_state.npz`
10
+ - `replay_states/episode_XXXXXX_replay.json`
11
+ """
12
+
13
+ from __future__ import annotations
14
+
15
+ import argparse
16
+ import csv
17
+ import gzip
18
+ import json
19
+ import pickle
20
+ import time
21
+ from io import BytesIO
22
+ from pathlib import Path
23
+ from typing import Any
24
+
25
+ import gymnasium as gym
26
+ import numpy as np
27
+ import robocasa # noqa: F401 - registers RoboCasa environments with robosuite
28
+ import torch
29
+ import zmq
30
+ from tqdm import trange
31
+
32
+ from gr00t.eval.wrappers.multistep_wrapper import MultiStepWrapper
33
+ from gr00t.eval.wrappers.record_video import RecordVideo
34
+ from gr00t.eval.wrappers.robocasa_wrapper import RoboCasaWrapper, load_robocasa_gym_env
35
+
36
+ TARGET_LAYOUT_AND_STYLE_IDS = [[i, i] for i in range(1, 11)]
37
+ TARGET_OBJ_INSTANCE_SPLIT = "target"
38
+ ACTION_SEED_KEY = "_policy_action_seed"
39
+ ACTION_SEED_EPISODE_STRIDE = 1_000_003
40
+ TORCH_SEED_MODULUS = 2**63 - 1
41
+
42
+
43
+ class PolicyClient:
44
+ """Minimal client for myGR00T's RobotInferenceServer."""
45
+
46
+ def __init__(self, host: str, port: int):
47
+ self.context = zmq.Context()
48
+ self.socket = self.context.socket(zmq.REQ)
49
+ self.socket.connect(f"tcp://{host}:{port}")
50
+
51
+ @staticmethod
52
+ def to_bytes(data: dict[str, Any]) -> bytes:
53
+ buffer = BytesIO()
54
+ torch.save(data, buffer)
55
+ return buffer.getvalue()
56
+
57
+ @staticmethod
58
+ def from_bytes(data: bytes) -> dict[str, Any]:
59
+ buffer = BytesIO(data)
60
+ return torch.load(buffer, weights_only=False)
61
+
62
+ def call(self, endpoint: str, data: dict[str, Any] | None = None) -> dict[str, Any]:
63
+ request: dict[str, Any] = {"endpoint": endpoint}
64
+ if data is not None:
65
+ request["data"] = data
66
+ self.socket.send(self.to_bytes(request))
67
+ message = self.socket.recv()
68
+ if message == b"ERROR":
69
+ raise RuntimeError("policy server returned ERROR")
70
+ return self.from_bytes(message)
71
+
72
+ def get_action(self, observations: dict[str, Any]) -> dict[str, Any]:
73
+ return self.call("get_action", observations)
74
+
75
+ def close(self) -> None:
76
+ self.socket.close()
77
+ self.context.term()
78
+
79
+
80
+ def as_bool(value: Any) -> bool:
81
+ if isinstance(value, np.ndarray):
82
+ return bool(np.any(value))
83
+ if isinstance(value, (list, tuple)):
84
+ return any(as_bool(v) for v in value)
85
+ if isinstance(value, dict):
86
+ return any(as_bool(v) for v in value.values())
87
+ return bool(value)
88
+
89
+
90
+ def json_default(value: Any) -> Any:
91
+ if isinstance(value, np.ndarray):
92
+ return value.tolist()
93
+ if isinstance(value, np.generic):
94
+ return value.item()
95
+ if isinstance(value, Path):
96
+ return str(value)
97
+ return str(value)
98
+
99
+
100
+ def find_robocasa_inner(env: Any) -> Any | None:
101
+ cur = env
102
+ for _ in range(80):
103
+ if cur is None:
104
+ return None
105
+ nxt = getattr(cur, "env", None)
106
+ if nxt is None or nxt is cur:
107
+ break
108
+ cur = nxt
109
+ if hasattr(cur, "sim") and hasattr(cur, "get_ep_meta"):
110
+ return cur
111
+ return None
112
+
113
+
114
+ def parse_mjcf_path(mjcf_path: str | None) -> dict[str, str | None]:
115
+ if not mjcf_path:
116
+ return {"registry": None, "instance_id": None, "mesh_key": None}
117
+ path = Path(str(mjcf_path))
118
+ parts = path.parts
119
+ registry = None
120
+ for candidate in ("objaverse", "aigen_objs", "lightwheel"):
121
+ if candidate in parts:
122
+ registry = candidate
123
+ break
124
+ instance_id = path.parent.name if path.name.endswith(".xml") else None
125
+ mesh_key = None
126
+ if registry and registry in parts and instance_id:
127
+ idx = parts.index(registry)
128
+ if registry == "lightwheel" and idx + 1 < len(parts):
129
+ mesh_key = "/".join((parts[idx + 1], instance_id))
130
+ elif idx + 1 < len(parts):
131
+ mesh_key = "/".join(parts[idx + 1 : -1])
132
+ return {"registry": registry, "instance_id": instance_id, "mesh_key": mesh_key}
133
+
134
+
135
+ def summarize_ep_meta(ep_meta: dict[str, Any]) -> dict[str, Any]:
136
+ summary: dict[str, Any] = {
137
+ "layout_id": ep_meta.get("layout_id"),
138
+ "style_id": ep_meta.get("style_id"),
139
+ "language": ep_meta.get("lang", ""),
140
+ "objects": [],
141
+ }
142
+ for cfg in ep_meta.get("object_cfgs", []) or []:
143
+ if not isinstance(cfg, dict):
144
+ continue
145
+ info = cfg.get("info", {}) if isinstance(cfg.get("info", {}), dict) else {}
146
+ mjcf_path = info.get("mjcf_path")
147
+ parsed = parse_mjcf_path(mjcf_path)
148
+ summary["objects"].append(
149
+ {
150
+ "name": cfg.get("name"),
151
+ "category": info.get("cat"),
152
+ "mjcf_path": mjcf_path,
153
+ "registry": parsed["registry"],
154
+ "instance_id": parsed["instance_id"],
155
+ "mesh_key": parsed["mesh_key"],
156
+ }
157
+ )
158
+ fixtures = ep_meta.get("fixtures")
159
+ if isinstance(fixtures, dict):
160
+ summary["fixture_names"] = list(fixtures.keys())
161
+ return summary
162
+
163
+
164
+ def capture_ep_meta(env: Any, pickle_path: Path) -> tuple[dict[str, Any], dict[str, Any]]:
165
+ inner = find_robocasa_inner(env)
166
+ if inner is None:
167
+ return {}, {"error": "could_not_find_robocasa_inner_env"}
168
+ try:
169
+ ep_meta = inner.get_ep_meta() or {}
170
+ except Exception as exc:
171
+ return {}, {"error": f"get_ep_meta_failed: {type(exc).__name__}: {exc}"}
172
+ with pickle_path.open("wb") as f:
173
+ pickle.dump(ep_meta, f)
174
+ return ep_meta, summarize_ep_meta(ep_meta)
175
+
176
+
177
+ def write_jsonl(path: Path, row: dict[str, Any]) -> None:
178
+ with path.open("a") as f:
179
+ f.write(json.dumps(row, default=json_default, sort_keys=True) + "\n")
180
+ f.flush()
181
+
182
+
183
+ def make_policy_action_seed(policy_seed: int, episode_index: int, policy_step: int) -> int:
184
+ seed = int(policy_seed) + int(episode_index) * ACTION_SEED_EPISODE_STRIDE + int(policy_step)
185
+ return seed % TORCH_SEED_MODULUS
186
+
187
+
188
+ def resolve_artifact_path(value: str, state_json_path: Path) -> Path:
189
+ path = Path(value)
190
+ if path.is_file():
191
+ return path
192
+ candidate = state_json_path.parent / path.name
193
+ if candidate.is_file():
194
+ return candidate
195
+ raise FileNotFoundError(f"missing replay artifact: {value}")
196
+
197
+
198
+ def load_replay_state_index(root: Path) -> dict[int, dict[str, Any]]:
199
+ if not root.is_dir():
200
+ raise FileNotFoundError(f"missing replay_state_root: {root}")
201
+ paths = sorted(root.glob("workers/worker_*/replay_states/episode_*_replay.json"))
202
+ paths.extend(sorted(root.glob("replay_states/episode_*_replay.json")))
203
+ index: dict[int, dict[str, Any]] = {}
204
+ for path in paths:
205
+ row = json.loads(path.read_text())
206
+ episode_index = int(row["episode_index"])
207
+ row["replay_json"] = str(path)
208
+ row["model_xml_gz"] = str(resolve_artifact_path(str(row["model_xml_gz"]), path))
209
+ row["state_npz"] = str(resolve_artifact_path(str(row["state_npz"]), path))
210
+ index[episode_index] = row
211
+ if not index:
212
+ raise FileNotFoundError(f"no replay state jsons found under {root}")
213
+ return index
214
+
215
+
216
+ def load_ep_meta_from_state_info(state_info: dict[str, Any]) -> dict[str, Any] | None:
217
+ for key in ("regenerated_ep_meta_pickle", "source_ep_meta_pickle", "ep_meta_pickle"):
218
+ path_value = state_info.get(key)
219
+ if not path_value:
220
+ continue
221
+ path = Path(str(path_value))
222
+ if not path.is_file():
223
+ continue
224
+ with path.open("rb") as f:
225
+ return pickle.load(f)
226
+ return None
227
+
228
+
229
+ def build_fresh_groot_obs(robo_wrapper: RoboCasaWrapper, inner: Any) -> dict[str, Any]:
230
+ gym_env = robo_wrapper.env
231
+ if getattr(inner, "viewer_get_obs", False) and getattr(inner, "viewer", None) is not None:
232
+ raw_obs = inner.viewer._get_observations(force_update=True)
233
+ else:
234
+ raw_obs = inner._get_observations(force_update=True)
235
+ if getattr(gym_env, "flatten_obs", False):
236
+ obs = gym_env._flatten_obs(raw_obs)
237
+ else:
238
+ obs = gym_env._filter_obs(raw_obs)
239
+ new_obs: dict[str, Any] = {}
240
+ for key, value in obs.items():
241
+ if key in robo_wrapper._robocasa_keys_to_gr00t_keys:
242
+ new_obs[robo_wrapper._robocasa_keys_to_gr00t_keys[key]] = value
243
+ new_obs["annotation.human.action.task_description"] = [robo_wrapper.language_instruction]
244
+ return new_obs
245
+
246
+
247
+ class ExactStateReplayWrapper(gym.Wrapper):
248
+ def __init__(self, env: RoboCasaWrapper, replay_state_index: dict[int, dict[str, Any]]):
249
+ super().__init__(env)
250
+ self.replay_state_index = replay_state_index
251
+ self.current_episode_index: int | None = None
252
+ self.last_state_info: dict[str, Any] = {}
253
+
254
+ def set_episode_index(self, episode_index: int) -> None:
255
+ self.current_episode_index = int(episode_index)
256
+
257
+ def reset(self, seed=None, options=None):
258
+ if self.current_episode_index is None:
259
+ raise RuntimeError("set_episode_index must be called before reset")
260
+ if self.current_episode_index not in self.replay_state_index:
261
+ raise KeyError(f"missing replay state for episode_index={self.current_episode_index}")
262
+ state_info = self.replay_state_index[self.current_episode_index]
263
+ inner = find_robocasa_inner(self.env)
264
+ if inner is None or not hasattr(inner, "reset_from_xml_string"):
265
+ raise RuntimeError("could not find RoboCasa inner env with reset_from_xml_string")
266
+
267
+ ep_meta = load_ep_meta_from_state_info(state_info)
268
+ if ep_meta is not None and hasattr(inner, "set_ep_meta"):
269
+ inner.set_ep_meta(ep_meta)
270
+
271
+ with gzip.open(Path(state_info["model_xml_gz"]), "rt") as f:
272
+ model_xml = f.read()
273
+ saved = np.load(Path(state_info["state_npz"]))
274
+ saved_state = saved["states"]
275
+ saved_ctrl = saved["ctrl"] if "ctrl" in saved.files else None
276
+
277
+ inner.reset()
278
+ if ep_meta is not None and hasattr(inner, "set_ep_meta"):
279
+ inner.set_ep_meta(ep_meta)
280
+ inner.reset_from_xml_string(model_xml)
281
+ inner.sim.reset()
282
+ inner.sim.set_state_from_flattened(saved_state)
283
+ if saved_ctrl is not None and saved_ctrl.shape == inner.sim.data.ctrl.shape:
284
+ inner.sim.data.ctrl[:] = saved_ctrl
285
+ inner.sim.forward()
286
+
287
+ # --- FIX B: re-sync each part controller to the RESTORED state -------------
288
+ # reset_from_xml_string() above triggers a throwaway env reset, during which
289
+ # every controller latches initial_joint (used for the OSC nullspace bias) and
290
+ # goal_pos from the robot's freshly *randomised* init pose (robot.py: init_qpos
291
+ # += gaussian noise, magnitude 0.02). set_state_from_flattened() then teleports
292
+ # the robot to the episode's saved pose, but nothing re-syncs the controllers,
293
+ # so they keep biasing toward a pose this episode never had -- and that stale
294
+ # pose depends on the client --seed, which leaks worker sharding into results.
295
+ # Re-seed each controller from the restored qpos instead.
296
+ for _robot in getattr(inner, "robots", []) or []:
297
+ _cc = getattr(_robot, "composite_controller", None)
298
+ if _cc is None:
299
+ continue
300
+ for _ctrl in getattr(_cc, "part_controllers", {}).values():
301
+ _qidx = getattr(_ctrl, "qpos_index", None)
302
+ if _qidx is None or getattr(_ctrl, "ref_name", None) is None:
303
+ continue # e.g. MobileBaseJointVelocityController has no ref_pos
304
+ try:
305
+ _ctrl.update_initial_joints(inner.sim.data.qpos[_qidx])
306
+ except Exception as _exc:
307
+ print(f"[fixB] update_initial_joints skipped: {_exc}", flush=True)
308
+ try:
309
+ _cc.reset() # reset_goal() on every part controller
310
+ except Exception as _exc:
311
+ print(f"[fixB] composite reset skipped: {_exc}", flush=True)
312
+ inner.sim.forward()
313
+ # --- end FIX B -------------------------------------------------------------
314
+
315
+ obs = build_fresh_groot_obs(self.env, inner)
316
+ info = {
317
+ "is_success": self.env.is_success()["task"],
318
+ "replay_state_npz": state_info["state_npz"],
319
+ "replay_model_xml_gz": state_info["model_xml_gz"],
320
+ "replay_state_json": state_info["replay_json"],
321
+ "replay_state_flat_sha256": state_info.get("state_flat_sha256"),
322
+ "source_episode_index": state_info.get("source_episode_index"),
323
+ "source_episode_seed": state_info.get("source_episode_seed"),
324
+ "source_worker_id": state_info.get("source_worker_id"),
325
+ "source_local_episode_index": state_info.get("source_local_episode_index"),
326
+ "source_ep_meta_pickle": state_info.get("source_ep_meta_pickle"),
327
+ "regenerated_ep_meta_pickle": state_info.get("regenerated_ep_meta_pickle"),
328
+ }
329
+ self.last_state_info = state_info
330
+ return obs, info
331
+
332
+
333
+ def main() -> None:
334
+ parser = argparse.ArgumentParser()
335
+ parser.add_argument("--host", default="127.0.0.1")
336
+ parser.add_argument("--port", type=int, required=True)
337
+ parser.add_argument("--model_path", required=True)
338
+ parser.add_argument("--output_dir", required=True)
339
+ parser.add_argument("--env_name", default="PickPlaceCounterToCabinet")
340
+ parser.add_argument("--eval_task", default="PnPCounterToCab")
341
+ parser.add_argument("--finetuned_task", required=True)
342
+ parser.add_argument("--n_episodes", type=int, default=200)
343
+ parser.add_argument("--episode_offset", type=int, default=0)
344
+ parser.add_argument("--episode_indices", default=None,
345
+ help="comma-separated global episode indices; overrides offset/n_episodes")
346
+ parser.add_argument("--worker_id", default="0")
347
+ parser.add_argument("--seed", type=int, default=42)
348
+ parser.add_argument("--max_episode_steps", type=int, default=750)
349
+ parser.add_argument("--action_horizon", type=int, default=16)
350
+ parser.add_argument("--robots", default="PandaOmron")
351
+ parser.add_argument("--generative_textures", action="store_true")
352
+ parser.add_argument("--save_videos", action="store_true")
353
+ parser.add_argument("--replay_state_root", required=True)
354
+ parser.add_argument("--policy_seed", type=int, default=None)
355
+ args = parser.parse_args()
356
+
357
+ out_dir = Path(args.output_dir)
358
+ video_dir = out_dir / "videos"
359
+ meta_pickle_dir = out_dir / "ep_meta_pickles"
360
+ out_dir.mkdir(parents=True, exist_ok=True)
361
+ if args.save_videos:
362
+ video_dir.mkdir(parents=True, exist_ok=True)
363
+ meta_pickle_dir.mkdir(parents=True, exist_ok=True)
364
+
365
+ replay_state_root = Path(args.replay_state_root)
366
+ replay_state_index = load_replay_state_index(replay_state_root)
367
+
368
+ run_info = {
369
+ "model_path": args.model_path,
370
+ "finetuned_task": args.finetuned_task,
371
+ "eval_task": args.eval_task,
372
+ "env_name": args.env_name,
373
+ "n_episodes_requested": args.n_episodes,
374
+ "episode_offset": args.episode_offset,
375
+ "worker_id": args.worker_id,
376
+ "seed": args.seed,
377
+ "max_episode_steps": args.max_episode_steps,
378
+ "action_horizon": args.action_horizon,
379
+ "robots": args.robots,
380
+ "layout": None,
381
+ "style": None,
382
+ "layout_and_style_ids": TARGET_LAYOUT_AND_STYLE_IDS,
383
+ "obj_instance_split": TARGET_OBJ_INSTANCE_SPLIT,
384
+ "generative_textures": bool(args.generative_textures),
385
+ "save_videos": bool(args.save_videos),
386
+ "policy_seed": args.policy_seed,
387
+ "policy_action_seed_key": ACTION_SEED_KEY if args.policy_seed is not None else None,
388
+ "policy_action_seed_formula": (
389
+ f"(policy_seed + episode_index * {ACTION_SEED_EPISODE_STRIDE} + policy_step)"
390
+ f" % {TORCH_SEED_MODULUS}"
391
+ if args.policy_seed is not None
392
+ else None
393
+ ),
394
+ "codebase": "myGR00T",
395
+ "data_config": "single_panda_gripper",
396
+ "embodiment_tag": "new_embodiment",
397
+ "replay_state_root": str(replay_state_root),
398
+ "replay_state_count": len(replay_state_index),
399
+ "replay_mode": "exact_model_xml_plus_mujoco_state",
400
+ "canonical_episode_definition": "episode_index restores model XML and flattened MuJoCo state from replay_state_root",
401
+ "started_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
402
+ }
403
+ (out_dir / "run_info.json").write_text(json.dumps(run_info, indent=2, sort_keys=True) + "\n")
404
+
405
+ client = PolicyClient(host=args.host, port=args.port)
406
+ env = load_robocasa_gym_env(
407
+ args.env_name,
408
+ seed=args.seed,
409
+ robots=args.robots,
410
+ camera_widths=256,
411
+ camera_heights=256,
412
+ render_onscreen=False,
413
+ obj_instance_split=TARGET_OBJ_INSTANCE_SPLIT,
414
+ generative_textures="100p" if args.generative_textures else None,
415
+ randomize_cameras=False,
416
+ layout_and_style_ids=TARGET_LAYOUT_AND_STYLE_IDS,
417
+ layout_ids=None,
418
+ style_ids=None,
419
+ )
420
+ env = ExactStateReplayWrapper(RoboCasaWrapper(env), replay_state_index)
421
+ replay_wrapper = env
422
+ if args.save_videos:
423
+ env = RecordVideo(
424
+ env,
425
+ str(video_dir),
426
+ disable_logger=True,
427
+ episode_trigger=lambda episode_id: True,
428
+ fps=20,
429
+ name_prefix=args.eval_task,
430
+ )
431
+ env = MultiStepWrapper(
432
+ env,
433
+ video_delta_indices=np.arange(1),
434
+ state_delta_indices=np.arange(1),
435
+ n_action_steps=args.action_horizon,
436
+ max_episode_steps=args.max_episode_steps,
437
+ )
438
+
439
+ episodes_jsonl = out_dir / "episodes.jsonl"
440
+ full_meta_jsonl = out_dir / "episodes_full_meta.jsonl"
441
+ csv_path = out_dir / "simulation_results.csv"
442
+ fieldnames = [
443
+ "episode_index",
444
+ "local_episode_index",
445
+ "worker_id",
446
+ "episode_seed",
447
+ "success",
448
+ "reward",
449
+ "policy_steps",
450
+ "sim_step_budget",
451
+ "video_path",
452
+ "layout_id",
453
+ "style_id",
454
+ "language",
455
+ "target_mesh_keys",
456
+ "grasped",
457
+ "grasped_strict",
458
+ "lifted",
459
+ "in_cab",
460
+ "max_lift_m",
461
+ ]
462
+ with csv_path.open("w", newline="") as f:
463
+ csv.DictWriter(f, fieldnames=fieldnames).writeheader()
464
+
465
+ successes: list[bool] = []
466
+ try:
467
+ _ep_list = ([int(x) for x in args.episode_indices.split(',') if x.strip()]
468
+ if args.episode_indices else None)
469
+ _n = len(_ep_list) if _ep_list else args.n_episodes
470
+ for local_episode_index in trange(_n, desc=f"{args.eval_task}/w{args.worker_id}"):
471
+ episode_index = (_ep_list[local_episode_index] if _ep_list
472
+ else args.episode_offset + local_episode_index)
473
+ replay_wrapper.set_episode_index(episode_index)
474
+ obs, info = env.reset()
475
+ reset_info = dict(info)
476
+ ep_meta_pickle = meta_pickle_dir / f"episode_{episode_index:06d}.pkl"
477
+ ep_meta, ep_summary = capture_ep_meta(env, ep_meta_pickle)
478
+ done = False
479
+ policy_steps = 0
480
+ reward_total = 0.0
481
+ last_info: dict[str, Any] = info
482
+
483
+ view_frames = {"wrist_view": []} # center is already covered by RecordVideo
484
+ stage = {"grasped": False, "grasped_strict": False, "lifted": False, "in_cab": False}
485
+ _max_dz = 0.0
486
+ _obj_z0 = None
487
+
488
+ while not done:
489
+ for _vk in view_frames:
490
+ _f = obs.get(f"video.{_vk}")
491
+ if _f is not None:
492
+ _f = np.asarray(_f)
493
+ view_frames[_vk].append(_f[0] if _f.ndim == 4 else _f)
494
+ policy_obs = dict(obs)
495
+ policy_obs["video.left_view"] = np.flip(obs["video.left_view"], axis=1)
496
+ policy_obs["video.right_view"] = np.flip(obs["video.right_view"], axis=1)
497
+ policy_obs["video.wrist_view"] = np.flip(obs["video.wrist_view"], axis=1)
498
+ if args.policy_seed is not None:
499
+ policy_obs[ACTION_SEED_KEY] = make_policy_action_seed(
500
+ args.policy_seed,
501
+ episode_index,
502
+ policy_steps,
503
+ )
504
+ action = client.get_action(policy_obs)
505
+ post_action = {}
506
+ for key, value in action.items():
507
+ post_action[key] = value[..., None] if getattr(value, "ndim", 0) == 1 else value
508
+ obs, reward, terminated, truncated, last_info = env.step(post_action)
509
+ try:
510
+ _inner = replay_wrapper.env.unwrapped
511
+ import robocasa.utils.object_utils as _OU
512
+ _z = _inner.sim.data.body_xpos[
513
+ _inner.sim.model.body_name2id(_inner.objects["obj"].root_body)][2]
514
+ if _obj_z0 is None:
515
+ _obj_z0 = _z
516
+ # RoboCasa's own definition: any gripper/object contact AND fingers closed.
517
+ # robosuite's _check_grasp additionally demands contact on BOTH fingerpads,
518
+ # which missed a grasp on an episode that completed the full task, so it is
519
+ # kept only as a strict secondary flag.
520
+ if _OU.check_obj_grasped(_inner, "obj"):
521
+ stage["grasped"] = True
522
+ if _inner._check_grasp(gripper=_inner.robots[0].gripper["right"],
523
+ object_geoms=_inner.objects["obj"].contact_geoms):
524
+ stage["grasped_strict"] = True
525
+ _max_dz = max(_max_dz, _z - _obj_z0)
526
+ if _z - _obj_z0 > 0.03:
527
+ stage["lifted"] = True
528
+ if _OU.obj_inside_of(_inner, "obj", _inner.cab):
529
+ stage["in_cab"] = True
530
+ except Exception as _e:
531
+ if policy_steps == 0:
532
+ print(f" stage logging unavailable: {_e}", flush=True)
533
+ reward_total += float(np.max(reward) if isinstance(reward, np.ndarray) else reward)
534
+ policy_steps += 1
535
+ done = bool(terminated or truncated)
536
+
537
+ if args.save_videos:
538
+ import imageio
539
+ ep_dir = video_dir / f"episode_{episode_index:06d}"
540
+ for _vk, _frames in view_frames.items():
541
+ if not _frames:
542
+ continue
543
+ ep_dir.mkdir(parents=True, exist_ok=True)
544
+ imageio.mimwrite(str(ep_dir / "wrist.mp4"), _frames, fps=20,
545
+ codec="libx264", macro_block_size=1,
546
+ ffmpeg_params=["-pix_fmt", "yuv420p"])
547
+
548
+ success = as_bool(last_info.get("is_success", False))
549
+ successes.append(success)
550
+ video_path = (
551
+ str(video_dir / f"{args.eval_task}-episode_{local_episode_index}.mp4")
552
+ if args.save_videos
553
+ else ""
554
+ )
555
+ row = {
556
+ "episode_index": episode_index,
557
+ "local_episode_index": local_episode_index,
558
+ "worker_id": args.worker_id,
559
+ "episode_seed": args.seed,
560
+ "canonical_episode_id": f"state_replay_episode_{episode_index:06d}",
561
+ "success": success,
562
+ "grasped": bool(stage["grasped"]),
563
+ "grasped_strict": bool(stage["grasped_strict"]),
564
+ "max_lift_m": round(float(_max_dz), 4),
565
+ "lifted": bool(stage["lifted"]),
566
+ "in_cab": bool(stage["in_cab"]),
567
+ "reward": reward_total,
568
+ "policy_steps": policy_steps,
569
+ "sim_step_budget": policy_steps * args.action_horizon,
570
+ "video_path": video_path,
571
+ "ep_meta_pickle": str(ep_meta_pickle),
572
+ "replay_state_npz": reset_info.get("replay_state_npz"),
573
+ "replay_model_xml_gz": reset_info.get("replay_model_xml_gz"),
574
+ "replay_state_json": reset_info.get("replay_state_json"),
575
+ "replay_state_flat_sha256": reset_info.get("replay_state_flat_sha256"),
576
+ "source_ep_meta_pickle": reset_info.get("source_ep_meta_pickle"),
577
+ "regenerated_ep_meta_pickle": reset_info.get("regenerated_ep_meta_pickle"),
578
+ "model_path": args.model_path,
579
+ "finetuned_task": args.finetuned_task,
580
+ "eval_task": args.eval_task,
581
+ "env_name": args.env_name,
582
+ "seed": args.seed,
583
+ **ep_summary,
584
+ }
585
+ write_jsonl(episodes_jsonl, row)
586
+ write_jsonl(
587
+ full_meta_jsonl,
588
+ {
589
+ "episode_index": episode_index,
590
+ "ep_meta_pickle": str(ep_meta_pickle),
591
+ "ep_meta": ep_meta,
592
+ },
593
+ )
594
+ with csv_path.open("a", newline="") as f:
595
+ writer = csv.DictWriter(f, fieldnames=fieldnames)
596
+ writer.writerow(
597
+ {
598
+ "episode_index": row["episode_index"],
599
+ "local_episode_index": row["local_episode_index"],
600
+ "worker_id": row["worker_id"],
601
+ "episode_seed": row["episode_seed"],
602
+ "success": int(success),
603
+ "reward": reward_total,
604
+ "policy_steps": policy_steps,
605
+ "sim_step_budget": policy_steps * args.action_horizon,
606
+ "video_path": video_path,
607
+ "layout_id": ep_summary.get("layout_id"),
608
+ "style_id": ep_summary.get("style_id"),
609
+ "language": ep_summary.get("language"),
610
+ "target_mesh_keys": ";".join(
611
+ str(obj.get("mesh_key"))
612
+ for obj in ep_summary.get("objects", [])
613
+ if obj.get("mesh_key")
614
+ ),
615
+ }
616
+ )
617
+ finally:
618
+ env.close()
619
+ client.close()
620
+
621
+ summary = {
622
+ **run_info,
623
+ "finished_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
624
+ "n_episodes": len(successes),
625
+ "n_success": int(sum(successes)),
626
+ "success_rate": float(sum(successes) / max(len(successes), 1)),
627
+ }
628
+ (out_dir / "summary.json").write_text(json.dumps(summary, indent=2, sort_keys=True) + "\n")
629
+ print(json.dumps(summary, indent=2, sort_keys=True))
630
+
631
+
632
+ if __name__ == "__main__":
633
+ main()
eval/organize_videos.py ADDED
@@ -0,0 +1,47 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """Put center and wrist side by side under one folder per global episode.
3
+
4
+ RecordVideo names its file by the worker-local index (agentview_center); the patched client
5
+ writes the wrist view under the global index. simulation_results.csv carries the mapping.
6
+ """
7
+ import argparse, csv, os, shutil, sys
8
+
9
+ ap = argparse.ArgumentParser()
10
+ ap.add_argument("--root", required=True)
11
+ a = ap.parse_args()
12
+
13
+ csv_path = os.path.join(a.root, "simulation_results.csv")
14
+ if not os.path.exists(csv_path):
15
+ sys.exit(f"missing {csv_path}")
16
+
17
+ out = os.path.join(a.root, "rollouts")
18
+ os.makedirs(out, exist_ok=True)
19
+ moved = missing = 0
20
+ for r in csv.DictReader(open(csv_path)):
21
+ ep = int(r["episode_index"])
22
+ d = os.path.join(out, f"episode_{ep:06d}")
23
+ os.makedirs(d, exist_ok=True)
24
+
25
+ center = r.get("video_path", "")
26
+ if center and os.path.exists(center):
27
+ shutil.copy2(center, os.path.join(d, "center.mp4"))
28
+ moved += 1
29
+ else:
30
+ missing += 1
31
+
32
+ # the client wrote videos/episode_NNNNNN/wrist.mp4 inside that episode's worker dir
33
+ wd = os.path.dirname(os.path.dirname(center)) if center else ""
34
+ wrist = os.path.join(wd, "videos", f"episode_{ep:06d}", "wrist.mp4")
35
+ if os.path.exists(wrist):
36
+ shutil.copy2(wrist, os.path.join(d, "wrist.mp4"))
37
+ else:
38
+ missing += 1
39
+
40
+ with open(os.path.join(d, "info.txt"), "w") as f:
41
+ f.write(f"episode_index {ep}\n"
42
+ f"success {r['success']}\n"
43
+ f"target {r['target_mesh_keys'].split(';')[0]}\n"
44
+ f"language {r['language']}\n"
45
+ f"policy_steps {r['policy_steps']}\n"
46
+ f"layout/style {r['layout_id']}/{r['style_id']}\n")
47
+ print(f"{moved} episodes organised under {out} (missing files: {missing})")
eval/run_grasp_sample.sh ADDED
@@ -0,0 +1,32 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env bash
2
+ # 40-episode sample (5 per object) with stage logging. Two models in parallel, 2 GPUs each,
3
+ # staying off GPU 0/1 and leaving 2,3 to the running training.
4
+ set -u
5
+ M=/data/minha/pose6daug/train_robocasa/myGR00T
6
+ PY=/data/nvidia/gripper_augmentator/conda-envs/mygr00t/bin/python
7
+ RPY=/lp-dev/jonghoon/robocasa_calib/envs/robocasa/bin/python
8
+ PP=/lp-dev/jonghoon/robocasa_calib/repos/robosuite_robocasa_dev:/lp-dev/jonghoon/robocasa_calib/repos/robocasa:$M
9
+ RS=/lp-dev/jonghoon/isaac-gr00t/eval_results/pnpcountertocab_mimicgen8_exact160_replay_policyseed12345_20260506
10
+ OUT=/home/nvidia/minha/pose6daug/evals/results/grasp_sample40
11
+ EPS="0,1,2,3,4,20,21,22,23,24,40,41,42,43,44,60,61,62,63,64,80,81,82,83,84,100,101,102,103,104,120,121,122,123,124,140,141,142,143,144"
12
+
13
+ one () { # tag, model, gpu, port
14
+ local tag=$1 model=$2 gpu=$3 port=$4
15
+ local d=$OUT/$tag; mkdir -p $d
16
+ ( cd $M; PYTHONPATH=$M CUDA_VISIBLE_DEVICES=$gpu $PY scripts/inference_service.py --server --port $port --model_path $model --data_config single_panda_gripper --embodiment_tag new_embodiment --seed 12345 ) > $d/server.log 2>&1 &
17
+ echo $! > $d/server.pid
18
+ for i in $(seq 1 120); do
19
+ python3 -c "import socket,sys;s=socket.socket();sys.exit(0 if s.connect_ex(('127.0.0.1',$port))==0 else 1)" && break
20
+ sleep 5
21
+ done
22
+ echo "[grasp] $tag server ready $(date -u +%H:%M:%S)"
23
+ PYTHONPATH=$PP CUDA_VISIBLE_DEVICES=$gpu $RPY $M/scripts/eval_robocasa_replay_state_grasp.py --host 127.0.0.1 --port $port --model_path $model --output_dir $d --env_name PickPlaceCounterToCabinet --eval_task PnPCounterToCab --finetuned_task grasp-sample-$tag --replay_state_root $RS --episode_indices "$EPS" --worker_id 00 --seed 42 --max_episode_steps 750 --action_horizon 16 --robots PandaOmron --generative_textures --policy_seed 12345 > $d/client.log 2>&1
24
+ echo "[grasp] $tag done $(date -u +%H:%M:%S)"
25
+ kill $(cat $d/server.pid) 2>/dev/null
26
+ }
27
+
28
+ one mimicgen30k /data/minha/pose6daug/train_robocasa/runs/pnpcountertocab_mimicgen_natural_256_fixed_newemb_from60k_gbs64_30000steps_20260920_133919/checkpoint-30000 4 8851 &
29
+ sleep 10
30
+ one vace30k /data/minha/pose6daug/train_robocasa/runs/vace_batch64_30k_from_hf/checkpoint-30000 6 8861 &
31
+ wait
32
+ echo "[grasp] ALL DONE $(date -u +%H:%M:%S)"
tools/ACTION_LAYOUT.md ADDED
@@ -0,0 +1,103 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Action-layout check for gr00t_views datasets
2
+
3
+ Found 2026-09-20 while investigating a 0/160 eval. Use it on any gr00t_views dataset before
4
+ training on it.
5
+
6
+ ## The bug
7
+
8
+ RoboCasa's simulator emits a 12-d action as
9
+
10
+ ```
11
+ [eef_pos(3), eef_rot(3), gripper(1), base(3), torso(1), base_mode(1)] robosuite order
12
+ ```
13
+
14
+ A gr00t_views dataset's `meta/modality.json` declares
15
+
16
+ ```
17
+ base_motion[0:4] control_mode[4:5] eef_pos[5:8] eef_rot[8:11] gripper_close[11:12]
18
+ ```
19
+
20
+ which is a **different order**. A builder that copies simulator actions straight into the
21
+ parquet writes correct numbers under the wrong column names. Nothing errors, the loss
22
+ converges to a small value, and the resulting policy is useless: the slices land as
23
+
24
+ | modality.json reads | what is actually there |
25
+ |---|---|
26
+ | `base_motion[0:4]` | eef x, y, z and rot x |
27
+ | `control_mode[4:5]` | rot y |
28
+ | `eef_pos[5:8]` | rot z, **gripper**, 0 |
29
+ | `eef_rot[8:11]` | 0, 0, 0 |
30
+ | `gripper_close[11:12]` | base_mode, a constant |
31
+
32
+ So the model is trained to emit arm motion on the base-motion channel and a constant on the
33
+ gripper channel. At eval the robot drives its base away from the counter and never closes the
34
+ gripper. Measured: 0/160 on `pnpcountertocab_mimicgen8_exact160` for a checkpoint whose base
35
+ scored 15/160 on the same episodes.
36
+
37
+ ## How the check works
38
+
39
+ No reference dataset needed. In this task the mobile base never moves, so the two layouts are
40
+ distinguishable by which dimensions are constant:
41
+
42
+ ```
43
+ LeRobot dims 0-3 all zero, dim 4 constant, dim 11 two-valued (the gripper)
44
+ robosuite dims 0-5 continuous, dim 6 two-valued (the gripper), dims 7-10 zero, dim 11 constant
45
+ ```
46
+
47
+ ## Use
48
+
49
+ ```bash
50
+ python check_action_layout.py --dataset <gr00t_views dataset> [--episodes 20]
51
+ ```
52
+
53
+ Exit codes: `0` PASS (LeRobot), `1` FAIL (robosuite order), `2` not 12-d, `3` neither matched.
54
+
55
+ A `3` on a small sample can just mean the sampled episodes are degenerate -- re-run with a
56
+ larger `--episodes` before concluding anything.
57
+
58
+ ```
59
+ $ python check_action_layout.py --dataset .../mimicgen_natural_256
60
+ dim 6 gripper -1.000 .. 1.000 uniq 2 <- two-valued
61
+ dim11 base_mode -1.000 .. -1.000 uniq 1 <- constant
62
+ FAIL action column is in RoboCasa/robosuite order but modality.json declares LeRobot order.
63
+ Repair: actions = actions[:, [7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6]]
64
+
65
+ $ python check_action_layout.py --dataset .../pickplace_target_human/PickPlaceCounterToCabinet
66
+ PASS action column is in LeRobot order, matching modality.json.
67
+ ```
68
+
69
+ ## Fix
70
+
71
+ In the builder, reorder before writing the parquet:
72
+
73
+ ```python
74
+ # robosuite [eef_pos3, eef_rot3, gripper, base3, torso, base_mode]
75
+ # -> LeRobot [base3+torso, base_mode, eef_pos3, eef_rot3, gripper]
76
+ actions = all_actions[:, [7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6]]
77
+ ```
78
+
79
+ Rebuild the dataset rather than patching parquet in place, unless you have confirmed nothing
80
+ else already consumed it.
81
+
82
+ ## Which builders are affected
83
+
84
+ | builder | source of `action` | affected |
85
+ |---|---|---|
86
+ | `baseline/mimicgen/gr00t_build/render_to_gr00t.py` (line 180) | `demo_grp["actions"]` from the MimicGen HDF5, simulator order | **yes** |
87
+ | `train_robocasa/scripts/dataset_build/build_vace_objwise_gr00t_dataset.py` | `pd.read_parquet(src_parquet)` from the original dataset, already LeRobot order | no |
88
+
89
+ The rule of thumb: a builder that **re-derives** actions from a simulator rollout needs the
90
+ reorder; one that **copies** rows from an existing LeRobot dataset does not.
91
+
92
+ Note that `observation.state` is not affected in either builder -- `extract_obs_state()`
93
+ assembles it field by field (`base_pos(3) + base_quat(4) + eef_pos(3) + eef_quat(4) +
94
+ gripper_qpos(2)`) in the declared order, so only `action` was ever passed through raw.
95
+
96
+ ## Datasets checked
97
+
98
+ | dataset | result |
99
+ |---|---|
100
+ | `baseline/mimicgen/gr00t_views/mimicgen_natural_256` | FAIL |
101
+ | `baseline/mimicgen/gr00t_views/mimicgen_per_target_32_256eps` | FAIL (same builder) |
102
+ | `robocasa_full/pickplace_target_human/PickPlaceCounterToCabinet` | PASS |
103
+ | any VACE gr00t_views set | expected PASS -- run the checker to confirm on that machine |
tools/check_action_layout.py ADDED
@@ -0,0 +1,122 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """Check that a gr00t_views dataset's `action` column really is in the LeRobot order its
3
+ meta/modality.json claims.
4
+
5
+ RoboCasa's simulator emits actions as
6
+ [eef_pos(3), eef_rot(3), gripper(1), base(3), torso(1), base_mode(1)]
7
+ while a gr00t_views dataset declares
8
+ [base_motion(4), control_mode(1), eef_pos(3), eef_rot(3), gripper_close(1)]
9
+
10
+ Any builder that copies simulator actions through without reordering produces a dataset whose
11
+ numbers are right but whose columns are mislabelled. Training then puts arm motion on the
12
+ base-motion channel and a constant on the gripper channel, so the policy drives the robot away
13
+ and never closes the gripper. The loss still converges -- nothing errors -- which is why this
14
+ has to be checked explicitly.
15
+
16
+ The two layouts are told apart by which dimensions are constant, no reference dataset needed:
17
+
18
+ LeRobot dims 0-3 all zero, dim 4 constant, dim 11 two-valued (the gripper)
19
+ robosuite dims 0-5 continuous, dim 6 two-valued (the gripper), dims 7-10 zero, dim 11 constant
20
+
21
+ python check_action_layout.py --dataset <gr00t_views dataset> [--episodes 20]
22
+ """
23
+ from __future__ import annotations
24
+
25
+ import argparse
26
+ import glob
27
+ import json
28
+ import os
29
+ import sys
30
+
31
+ import numpy as np
32
+ import pyarrow.parquet as pq
33
+
34
+ LEROBOT_NAMES = ["base_x", "base_y", "base_th", "torso", "ctrl_mode",
35
+ "eef_x", "eef_y", "eef_z", "rot_x", "rot_y", "rot_z", "gripper"]
36
+ ROBOSUITE_NAMES = ["eef_x", "eef_y", "eef_z", "rot_x", "rot_y", "rot_z",
37
+ "gripper", "base_x", "base_y", "base_th", "torso", "base_mode"]
38
+ # robosuite index -> LeRobot position; apply as actions[:, FIX] to repair a dataset.
39
+ FIX = [7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6]
40
+
41
+
42
+ def load_actions(dataset: str, n_episodes: int) -> np.ndarray:
43
+ paths = sorted(glob.glob(os.path.join(dataset, "data", "chunk-*", "*.parquet")))
44
+ if not paths:
45
+ sys.exit(f"no parquet under {dataset}/data/chunk-*/")
46
+ rows = []
47
+ for p in paths[:n_episodes]:
48
+ rows.append(np.stack(pq.read_table(p, columns=["action"]).column("action").to_pylist()))
49
+ return np.vstack(rows), len(paths), min(n_episodes, len(paths))
50
+
51
+
52
+ def classify(a: np.ndarray) -> tuple[str, list[str]]:
53
+ """Which layout do the constant/continuous dimensions match?"""
54
+ const = [i for i in range(a.shape[1]) if len(np.unique(np.round(a[:, i], 4))) == 1]
55
+ zero = [i for i in const if abs(a[:, i]).max() < 1e-6]
56
+ twoval = [i for i in range(a.shape[1]) if len(np.unique(np.round(a[:, i], 4))) == 2]
57
+ notes = [f" all-zero dims : {zero}", f" constant dims : {const}",
58
+ f" two-valued : {twoval} (the gripper is two-valued)"]
59
+ lerobot = set([0, 1, 2, 3]).issubset(zero) and 4 in const and 11 in twoval
60
+ robosuite = set([7, 8, 9, 10]).issubset(zero) and 11 in const and 6 in twoval
61
+ if lerobot and not robosuite:
62
+ return "lerobot", notes
63
+ if robosuite and not lerobot:
64
+ return "robosuite", notes
65
+ return "unknown", notes
66
+
67
+
68
+ def main() -> int:
69
+ ap = argparse.ArgumentParser(description=__doc__,
70
+ formatter_class=argparse.RawDescriptionHelpFormatter)
71
+ ap.add_argument("--dataset", required=True, help="gr00t_views dataset directory")
72
+ ap.add_argument("--episodes", type=int, default=20, help="episodes to sample (default 20)")
73
+ a = ap.parse_args()
74
+
75
+ actions, n_total, n_read = load_actions(a.dataset, a.episodes)
76
+ print(f"dataset : {a.dataset}")
77
+ print(f"sampled : {n_read} of {n_total} episodes, {actions.shape[0]} frames x {actions.shape[1]} dims\n")
78
+
79
+ if actions.shape[1] != 12:
80
+ print(f"FAIL expected 12 action dims, found {actions.shape[1]}")
81
+ return 2
82
+
83
+ verdict, notes = classify(actions)
84
+ names = LEROBOT_NAMES if verdict == "lerobot" else ROBOSUITE_NAMES
85
+ for i in range(12):
86
+ col = actions[:, i]
87
+ uniq = len(np.unique(np.round(col, 4)))
88
+ tag = " <- constant" if uniq == 1 else (" <- two-valued" if uniq == 2 else "")
89
+ label = names[i] if verdict != "unknown" else "?"
90
+ print(f" dim{i:2d} {label:10s} {col.min():8.3f} .. {col.max():7.3f} "
91
+ f"|mean| {abs(col).mean():6.3f} uniq {uniq:5d}{tag}")
92
+ print()
93
+ print("\n".join(notes))
94
+ print()
95
+
96
+ modality = os.path.join(a.dataset, "meta", "modality.json")
97
+ if os.path.exists(modality):
98
+ act = json.load(open(modality)).get("action", {})
99
+ print(" modality.json action slices: "
100
+ + ", ".join(f"{k} [{v['start']}:{v['end']}]" for k, v in act.items()))
101
+ print()
102
+
103
+ if verdict == "lerobot":
104
+ print("PASS action column is in LeRobot order, matching modality.json.")
105
+ return 0
106
+ if verdict == "robosuite":
107
+ print("FAIL action column is in RoboCasa/robosuite order but modality.json declares")
108
+ print(" LeRobot order. Training would read arm motion as base motion and a")
109
+ print(" constant as the gripper.")
110
+ print(f" Repair: actions = actions[:, {FIX}]")
111
+ print(" Fix the builder and rebuild; do not patch the parquet in place unless you")
112
+ print(" also confirm nothing else consumed it.")
113
+ return 1
114
+ print("UNKNOWN neither layout matched. Inspect the dimensions above by hand:")
115
+ print(" LeRobot expects dims 0-3 zero, dim 4 constant, dim 11 two-valued.")
116
+ print(" An all-constant or degenerate episode sample can also land here --")
117
+ print(" re-run with a larger --episodes before concluding.")
118
+ return 3
119
+
120
+
121
+ if __name__ == "__main__":
122
+ raise SystemExit(main())
train/smoke_test.sh ADDED
@@ -0,0 +1,38 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env bash
2
+ # Verifies that GR00T 1.5 fine-tuning runs from THIS bundle: imports, dataset load,
3
+ # then a REAL 20-step single-GPU fine-tune on the 150-episode augmented dataset
4
+ # (per-GPU batch 2, no LoRA, same code path as the full run) writing a checkpoint
5
+ # into runs/smoke_<timestamp>/. Needs one GPU with ~25 GB free.
6
+ # ./smoke_test.sh <gpu_index> e.g. ./smoke_test.sh 3
7
+ set -euo pipefail
8
+ HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"; B="$(dirname "$HERE")"
9
+ GPU="${1:?usage: smoke_test.sh <gpu_index>}"
10
+ PY=/data/nvidia/gripper_augmentator/conda-envs/mygr00t/bin/python
11
+ DS="${DATASET_PATH:-/lp-dev/jonghoon/robocasa_full/gr00t_views/PickPlaceCounterToCabinet_150eps_augmented_groot15_v3_100_clean}"
12
+ CK="${BASE_CKPT:-/lp-dev/jonghoon/myGR00T_outputs/pnpcountertocab_all502_gbs64_wandb_60k_save5k_20260427_193901/checkpoint-60000}"
13
+ pass(){ echo " PASS $*"; }; fail(){ echo " FAIL $*"; exit 1; }
14
+ export PYTHONPATH="$B/myGR00T" HF_HOME=/lp-dev/jonghoon/.cache/huggingface HF_HUB_OFFLINE=1 NO_ALBUMENTATIONS_UPDATE=1
15
+
16
+ echo "== static =="
17
+ [ -x "$PY" ] && pass "env python $PY" || fail "mygr00t env missing"
18
+ (cd "$B/myGR00T" && "$PY" -c "import gr00t, os; assert gr00t.__file__.startswith('$B/'), gr00t.__file__; from gr00t.experiment.data_config import DATA_CONFIG_MAP; assert 'single_panda_gripper' in DATA_CONFIG_MAP") && pass "gr00t imports from the bundle, data config single_panda_gripper registered" || fail "gr00t import"
19
+ [ -f "$DS/meta/modality.json" ] && pass "dataset $DS" || fail "dataset missing"
20
+ [ -f "$CK/model.safetensors.index.json" ] && pass "base checkpoint $CK" || fail "checkpoint missing"
21
+ (cd "$B/myGR00T" && "$PY" - <<PYEOF
22
+ from gr00t.data.dataset import LeRobotSingleDataset
23
+ from gr00t.data.schema import EmbodimentTag
24
+ from gr00t.experiment.data_config import DATA_CONFIG_MAP
25
+ dc = DATA_CONFIG_MAP["single_panda_gripper"]
26
+ ds = LeRobotSingleDataset("$DS", dc.modality_config(), transforms=dc.transform(), embodiment_tag=EmbodimentTag("new_embodiment"), video_backend="decord")
27
+ s = ds[0]; print({k: tuple(v.shape) for k, v in s.items() if hasattr(v, "shape")}); assert len(ds) > 1000
28
+ PYEOF
29
+ ) && pass "LeRobotSingleDataset loads a sample (videos decoded)" || fail "dataset load"
30
+
31
+ echo "== 20-step fine-tune on GPU $GPU =="
32
+ RUN=smoke_$(date -u +%Y%m%d_%H%M%S); OUT="$B/runs/$RUN"
33
+ CUDA_VISIBLE_DEVICES="$GPU" GPUS="$GPU" NUM_GPUS=1 PER_GPU_BATCH=2 MAX_STEPS=20 SAVE_STEPS=20 DATALOADER_NUM_WORKERS=4 \
34
+ REPORT_TO=tensorboard RUN_NAME="$RUN" OUTPUT_DIR="$OUT" DATASET_PATH="$DS" BASE_CKPT="$CK" \
35
+ bash "$HERE/train_groot15_single_dataset.sh" > "$B/logs/$RUN.log" 2>&1 || { tail -30 "$B/logs/$RUN.log"; fail "fine-tune exited non-zero (log: $B/logs/$RUN.log)"; }
36
+ [ -f "$OUT/checkpoint-20/model.safetensors.index.json" ] || [ -f "$OUT/checkpoint-20/model.safetensors" ] && pass "checkpoint-20 written to $OUT" || fail "no checkpoint-20 in $OUT"
37
+ grep -q "train_loss\|'loss'" "$B/logs/$RUN.log" && pass "loss logged: $(grep -o "'loss': [0-9.]*" "$B/logs/$RUN.log" | tail -1)" || pass "run finished (no loss line found in log, check $B/logs/$RUN.log)"
38
+ echo "ALL PASSED (log: $B/logs/$RUN.log)"
train/train_groot15_single_dataset.sh ADDED
@@ -0,0 +1,116 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env bash
2
+ set -euo pipefail
3
+
4
+ # Self-contained bundle version (2026-09-19): code = ../myGR00T (this bundle), outputs = ../runs.
5
+ # Only the conda env, the LeRobot dataset and the base checkpoint are referenced by absolute path.
6
+ HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
7
+ BUNDLE="$(dirname "$HERE")"
8
+
9
+ # Set this to the dataset directory name under TRAIN_DATASET_BASE.
10
+ DATASETNAME="${DATASETNAME:-PickPlaceCounterToCabinet_150eps_augmented_groot15_v3_100_clean}"
11
+
12
+ MYGROOT_ROOT="${MYGROOT_ROOT:-$BUNDLE/myGR00T}"
13
+ PYTHON_BIN="${PYTHON_BIN:-/data/nvidia/gripper_augmentator/conda-envs/mygr00t/bin/python}"
14
+ CONDA_BIN="${CONDA_BIN:-/data/nvidia/gripper_augmentator/conda-envs/mygr00t/bin}"
15
+ TRAIN_DATASET_BASE="${TRAIN_DATASET_BASE:-/lp-dev/jonghoon/robocasa_full/gr00t_views}"
16
+ OUTPUT_BASE="${OUTPUT_BASE:-$BUNDLE/runs}"
17
+ BASE_CKPT="${BASE_CKPT:-/lp-dev/jonghoon/myGR00T_outputs/pnpcountertocab_all502_gbs64_wandb_60k_save5k_20260427_193901/checkpoint-60000}"
18
+
19
+ TASK_SLUG="${TASK_SLUG:-pnpcountertocab}"
20
+ RUN_ID="${RUN_ID:-$(date -u +%Y%m%d_%H%M%S)}"
21
+ MAX_STEPS="${MAX_STEPS:-30000}"
22
+ SAVE_STEPS="${SAVE_STEPS:-5000}"
23
+ RESUME="${RESUME:-0}"
24
+ GPUS="${GPUS:-0,1,2,3}"
25
+ PER_GPU_BATCH="${PER_GPU_BATCH:-16}"
26
+ DATALOADER_NUM_WORKERS="${DATALOADER_NUM_WORKERS:-8}"
27
+ DATA_CONFIG="${DATA_CONFIG:-single_panda_gripper}"
28
+ EMBODIMENT_TAG="${EMBODIMENT_TAG:-new_embodiment}"
29
+ BACKBONE_MODEL_TYPE="${BACKBONE_MODEL_TYPE:-eagle}"
30
+ BACKBONE_SELECT_LAYER="${BACKBONE_SELECT_LAYER:-12}"
31
+ REPORT_TO="${REPORT_TO:-tensorboard}" # wandb needs WANDB_API_KEY (or WANDB_API_KEY_FILE)
32
+
33
+ DATASET_PATH="${DATASET_PATH:-${TRAIN_DATASET_BASE}/${DATASETNAME}}"
34
+ RUN_NAME="${RUN_NAME:-${TASK_SLUG}_${DATASETNAME}_newemb_from60k_gbs64_${MAX_STEPS}steps_${RUN_ID}}"
35
+ OUTPUT_DIR="${OUTPUT_DIR:-${OUTPUT_BASE}/${RUN_NAME}}"
36
+
37
+ export CUDA_VISIBLE_DEVICES="${CUDA_VISIBLE_DEVICES:-$GPUS}"
38
+ export PATH="${CONDA_BIN}:${PATH}"
39
+ export WANDB_PROJECT="${WANDB_PROJECT:-gr00t-robocasa}"
40
+ export WANDB_ENTITY="${WANDB_ENTITY:-alphabet1}"
41
+ export WANDB_DIR="${WANDB_DIR:-$BUNDLE/logs/wandb}"
42
+ export WANDB_CACHE_DIR="${WANDB_CACHE_DIR:-$BUNDLE/logs/wandb/cache}"
43
+ export WANDB_CONFIG_DIR="${WANDB_CONFIG_DIR:-$BUNDLE/logs/wandb/config}"
44
+ export NO_ALBUMENTATIONS_UPDATE=1
45
+ # The mygr00t env's editable install of `gr00t` points at a deleted checkout; import the bundle copy instead.
46
+ export PYTHONPATH="$MYGROOT_ROOT${PYTHONPATH:+:$PYTHONPATH}"
47
+ export HF_HOME="${HF_HOME:-/lp-dev/jonghoon/.cache/huggingface}"
48
+ export HF_HUB_OFFLINE="${HF_HUB_OFFLINE:-1}"
49
+ export TOKENIZERS_PARALLELISM=false
50
+
51
+ # Optional, safer than hard-coding a secret in this script.
52
+ WANDB_API_KEY_FILE="${WANDB_API_KEY_FILE:-}"
53
+ if [[ -z "${WANDB_API_KEY:-}" && -n "$WANDB_API_KEY_FILE" && -f "$WANDB_API_KEY_FILE" ]]; then
54
+ WANDB_API_KEY="$(tr -d '[:space:]' < "$WANDB_API_KEY_FILE")"
55
+ export WANDB_API_KEY
56
+ fi
57
+
58
+ IFS=',' read -r -a GPU_LIST <<< "$CUDA_VISIBLE_DEVICES"
59
+ NUM_GPUS="${NUM_GPUS:-${#GPU_LIST[@]}}"
60
+
61
+ if [[ ! -d "$DATASET_PATH" ]]; then
62
+ echo "FATAL: dataset path does not exist: $DATASET_PATH" >&2
63
+ exit 1
64
+ fi
65
+ if [[ ! -f "$BASE_CKPT/model.safetensors.index.json" ]]; then
66
+ echo "FATAL: checkpoint is missing or incomplete: $BASE_CKPT" >&2
67
+ exit 1
68
+ fi
69
+
70
+ mkdir -p "$OUTPUT_DIR" "$WANDB_DIR" "$WANDB_CACHE_DIR" "$WANDB_CONFIG_DIR"
71
+ cd "$MYGROOT_ROOT"
72
+
73
+ exec > >(tee -a "$OUTPUT_DIR/train.log") 2>&1
74
+
75
+ echo "run_name=$RUN_NAME"
76
+ echo "output_dir=$OUTPUT_DIR"
77
+ echo "dataset_name=$DATASETNAME"
78
+ echo "dataset_path=$DATASET_PATH"
79
+ echo "base_ckpt=$BASE_CKPT"
80
+ echo "cuda_visible_devices=$CUDA_VISIBLE_DEVICES"
81
+ echo "num_gpus=$NUM_GPUS"
82
+ echo "per_gpu_batch=$PER_GPU_BATCH"
83
+ echo "global_batch=$((NUM_GPUS * PER_GPU_BATCH))"
84
+ echo "max_steps=$MAX_STEPS"
85
+ echo "save_steps=$SAVE_STEPS"
86
+ echo "resume=$RESUME"
87
+ date -u
88
+
89
+ RESUME_ARGS=()
90
+ if [[ "$RESUME" == "1" ]]; then
91
+ RESUME_ARGS+=(--resume)
92
+ fi
93
+
94
+ SEED_ARGS=()
95
+ if [[ -n "${DATA_SEED:-}" ]]; then
96
+ SEED_ARGS+=(--data-seed "$DATA_SEED")
97
+ fi
98
+
99
+ exec "$PYTHON_BIN" "$MYGROOT_ROOT/scripts/gr00t_finetune.py" \
100
+ --run-name "$RUN_NAME" \
101
+ --dataset-path "$DATASET_PATH" \
102
+ --output-dir "$OUTPUT_DIR" \
103
+ --data-config "$DATA_CONFIG" \
104
+ --dataloader-num-workers "$DATALOADER_NUM_WORKERS" \
105
+ --embodiment-tag "$EMBODIMENT_TAG" \
106
+ --backbone-model-type "$BACKBONE_MODEL_TYPE" \
107
+ --backbone-select-layer "$BACKBONE_SELECT_LAYER" \
108
+ --base-model-path "$BASE_CKPT" \
109
+ --num-gpus "$NUM_GPUS" \
110
+ --batch-size "$PER_GPU_BATCH" \
111
+ --max-steps "$MAX_STEPS" \
112
+ --save-steps "$SAVE_STEPS" \
113
+ --report-to "$REPORT_TO" \
114
+ "${SEED_ARGS[@]}" \
115
+ "${RESUME_ARGS[@]}" \
116
+ "$@"