pose6daug augmentation / dataset / training / eval scripts
Browse files- README.md +163 -0
- augment/mimicgen/config_template.json +90 -0
- augment/mimicgen/render_from_plan.py +132 -0
- augment/mimicgen/run_generation.sh +13 -0
- augment/mimicgen/run_render.sh +23 -0
- augment/mimicgen/select_by_generation_time.py +143 -0
- augment/mimicgen/select_by_generation_time_pi05.py +80 -0
- augment/mimicgen/snapshot_episode_times.sh +16 -0
- augment/vace/DATASET_CONVERSION.md +182 -0
- augment/vace/README.md +211 -0
- augment/vace/SETUP.md +138 -0
- augment/vace/build_aug32_assignments.py +148 -0
- augment/vace/collect_outputs.py +78 -0
- augment/vace/make_gt_masks.py +179 -0
- augment/vace/preflight.sh +60 -0
- augment/vace/run_aug32.sh +67 -0
- dataset/build_vace_objwise_gr00t_dataset.py +509 -0
- dataset/merge_gr00t_view_datasets.py +339 -0
- dataset/render_to_gr00t.py +353 -0
- dataset/repair_action_order.py +71 -0
- dataset/repair_mimicgen_task_ids.py +208 -0
- dataset/run_convert.sh +22 -0
- dataset/verify_geom_objwise_gr00t_dataset.py +233 -0
- eval/eval_groot15_exact_replay.sh +389 -0
- eval/eval_robocasa_replay_state_3view.py +590 -0
- eval/eval_robocasa_replay_state_grasp.py +633 -0
- eval/organize_videos.py +47 -0
- eval/run_grasp_sample.sh +32 -0
- tools/ACTION_LAYOUT.md +103 -0
- tools/check_action_layout.py +122 -0
- train/smoke_test.sh +38 -0
- train/train_groot15_single_dataset.sh +116 -0
README.md
ADDED
|
@@ -0,0 +1,163 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# pose6daug — augmentation, dataset build, training and evaluation scripts
|
| 2 |
+
|
| 3 |
+
The scripts behind the MimicGen and VACE augmentation baselines for RoboCasa
|
| 4 |
+
PickPlaceCounterToCabinet, and the GR00T 1.5 fine-tuning and exact-replay evaluation run on
|
| 5 |
+
top of them. Archived as-run: paths are absolute and point at one particular machine, so treat
|
| 6 |
+
this as a record of the procedure rather than a turnkey package. See **Paths to change** below.
|
| 7 |
+
|
| 8 |
+
```
|
| 9 |
+
augment/mimicgen/ generate episodes with MimicGen, render preview videos
|
| 10 |
+
augment/vace/ VACE object-swap augmentation (assignment builder, runner, GT masks)
|
| 11 |
+
dataset/ convert generated episodes into a gr00t_views (LeRobot v2.1) dataset
|
| 12 |
+
train/ GR00T 1.5 fine-tuning launcher
|
| 13 |
+
eval/ exact-state replay evaluation
|
| 14 |
+
tools/ dataset sanity checks
|
| 15 |
+
```
|
| 16 |
+
|
| 17 |
+
## Pipeline
|
| 18 |
+
|
| 19 |
+
```
|
| 20 |
+
MimicGen generation ──► demo.hdf5 (MuJoCo states + actions, no pixels)
|
| 21 |
+
│
|
| 22 |
+
┌─────────┴──────────┐
|
| 23 |
+
▼ ▼
|
| 24 |
+
preview mp4 (3 views) gr00t_views dataset ──► fine-tune ──► exact-replay eval
|
| 25 |
+
```
|
| 26 |
+
|
| 27 |
+
VACE skips the simulator: it repaints pixels and keeps the source trajectory, so its builder
|
| 28 |
+
copies the source parquet instead of replaying.
|
| 29 |
+
|
| 30 |
+
### 1. Augment
|
| 31 |
+
|
| 32 |
+
`augment/mimicgen/run_generation.sh` launches one MimicGen process per worker from a JSON
|
| 33 |
+
config per (object, worker); `config_template.json` is the template. Two knobs matter:
|
| 34 |
+
|
| 35 |
+
- `guarantee=false` makes `num_trials` mean **attempts**, not successes. Giving every object
|
| 36 |
+
the same attempt budget is what produces the generator's natural yield instead of a
|
| 37 |
+
per-object quota.
|
| 38 |
+
- `obj_registries` must include `aigen` for objects under `aigen_objs/` (e.g. `wine_5`), or
|
| 39 |
+
`sample_kitchen_object_helper` raises a bare `ValueError`.
|
| 40 |
+
|
| 41 |
+
`snapshot_episode_times.sh` records the per-episode temporary filenames while they exist:
|
| 42 |
+
MimicGen writes each success to `tmp/date_..._time_HH_MM_SS.hdf5`, then `merge_all_hdf5`
|
| 43 |
+
sorts by timestamp and deletes the folder. That snapshot is the only record of when each
|
| 44 |
+
episode was produced, and `select_by_generation_time.py` joins it back to order episodes
|
| 45 |
+
globally across workers. The poller can only ever miss a worker's **last** file, which the
|
| 46 |
+
selector pads with the merged file's mtime.
|
| 47 |
+
|
| 48 |
+
### 2. Build the dataset
|
| 49 |
+
|
| 50 |
+
`dataset/render_to_gr00t.py` replays each episode in MuJoCo, renders three cameras and writes
|
| 51 |
+
parquet + videos + meta. `dataset/run_convert.sh` shards it (each shard needs its own
|
| 52 |
+
`GEN_DIR` and `DATASET_OUT`, or the glob picks up the others), then
|
| 53 |
+
`merge_gr00t_view_datasets.py` merges and `repair_mimicgen_task_ids.py` fixes task ids.
|
| 54 |
+
|
| 55 |
+
`repair_mimicgen_task_ids.py` is not optional: the writer stores every parquet task column as
|
| 56 |
+
0 while `episodes.jsonl` holds the intended language, so without it every episode trains as
|
| 57 |
+
task 0 and the language conditioning silently collapses.
|
| 58 |
+
|
| 59 |
+
### 3. Train
|
| 60 |
+
|
| 61 |
+
```bash
|
| 62 |
+
DATASET_PATH=<gr00t_views dataset> DATASETNAME=<name> \
|
| 63 |
+
GPUS=2,3 PER_GPU_BATCH=32 MAX_STEPS=30000 SAVE_STEPS=5000 \
|
| 64 |
+
bash train/train_groot15_single_dataset.sh
|
| 65 |
+
```
|
| 66 |
+
|
| 67 |
+
Fine-tunes the action-head projector and diffusion head from a base checkpoint; the backbone
|
| 68 |
+
stays frozen. `RESUME=1` picks up the newest checkpoint in the output directory.
|
| 69 |
+
|
| 70 |
+
### 4. Evaluate
|
| 71 |
+
|
| 72 |
+
```bash
|
| 73 |
+
MODEL_PATH=<checkpoint> MYGROOT_ROOT=<myGR00T tree> \
|
| 74 |
+
EVAL_CLIENT=<eval/eval_robocasa_replay_state_grasp.py> \
|
| 75 |
+
REPLAY_STATE_ROOT=<replay set> N_EPISODES=160 GPUS=4,5,6,7 POLICY_SEED=12345 SEED_BASE=42 \
|
| 76 |
+
bash eval/eval_groot15_exact_replay.sh
|
| 77 |
+
```
|
| 78 |
+
|
| 79 |
+
Each episode restores a saved scene XML and flattened MuJoCo state, so runs are comparable
|
| 80 |
+
across checkpoints. Two independent seeds:
|
| 81 |
+
|
| 82 |
+
- `POLICY_SEED` — the policy server's global RNG **and** the per-step action seed
|
| 83 |
+
(`policy_seed + episode_index × stride + step`). This is the one that changes the rollout.
|
| 84 |
+
- `SEED_BASE` — the env seed, `SEED_BASE + worker_id` per worker. Exact replay overwrites the
|
| 85 |
+
scene immediately after, so it should not affect the initial state.
|
| 86 |
+
|
| 87 |
+
Worker count follows the GPU list, and episodes are split evenly across workers — so the same
|
| 88 |
+
`SEED_BASE` with a different GPU count gives each episode a different worker seed.
|
| 89 |
+
|
| 90 |
+
Client variants:
|
| 91 |
+
|
| 92 |
+
| file | adds |
|
| 93 |
+
|---|---|
|
| 94 |
+
| `eval_robocasa_replay_state_3view.py` | saves the wrist (ego) view; `RecordVideo` only captures `robot0_agentview_center` |
|
| 95 |
+
| `eval_robocasa_replay_state_grasp.py` | the above, plus `--episode_indices` for an arbitrary subset, and per-episode `grasped` / `lifted` / `in_cab` stage flags |
|
| 96 |
+
|
| 97 |
+
Stock RoboCasa success is `obj_inside_of(cab) and gripper_obj_far` — a single boolean, which
|
| 98 |
+
says nothing about where a failed episode broke down. The stage flags come from
|
| 99 |
+
`_check_grasp`, a 3 cm rise in the object's body z, and `OU.obj_inside_of`.
|
| 100 |
+
|
| 101 |
+
`organize_videos.py` regroups the output into one folder per global episode
|
| 102 |
+
(`rollouts/episode_NNNNNN/{center.mp4, wrist.mp4, info.txt}`); `RecordVideo` names files by
|
| 103 |
+
worker-local index, which does not match the global index.
|
| 104 |
+
|
| 105 |
+
## The action-order bug — check any dataset before training on it
|
| 106 |
+
|
| 107 |
+
RoboCasa's simulator exports actions **arm-first**:
|
| 108 |
+
|
| 109 |
+
```
|
| 110 |
+
[eef_pos(3), eef_rot(3), gripper, base(3), torso, base_mode]
|
| 111 |
+
```
|
| 112 |
+
|
| 113 |
+
A gr00t_views dataset declares them **base-first**:
|
| 114 |
+
|
| 115 |
+
```
|
| 116 |
+
base_motion[0:4] control_mode[4:5] eef_pos[5:8] eef_rot[8:11] gripper_close[11:12]
|
| 117 |
+
```
|
| 118 |
+
|
| 119 |
+
A builder that re-derives actions from a simulator rollout has to reorder; one that copies
|
| 120 |
+
rows from an existing LeRobot dataset does not. Copying them through unchanged puts arm motion
|
| 121 |
+
on the base-motion channel and `base_mode` on the gripper. Nothing errors, the loss converges
|
| 122 |
+
to a small value, and the policy drives the base away from the counter and never closes the
|
| 123 |
+
gripper — **0/160** on an exact-replay eval whose base checkpoint scored 11/160.
|
| 124 |
+
|
| 125 |
+
```python
|
| 126 |
+
actions = raw_actions[:, [7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6]]
|
| 127 |
+
```
|
| 128 |
+
|
| 129 |
+
`tools/check_action_layout.py` tells the two layouts apart from the data alone (the mobile
|
| 130 |
+
base never moves in this task, so the constant dimensions give it away), and
|
| 131 |
+
`dataset/repair_action_order.py` fixes an already-built dataset in place — parquet plus the
|
| 132 |
+
action entry of `meta/stats.json`, no re-render. It refuses to run on a dataset that does not
|
| 133 |
+
look like simulator order, so it cannot be applied twice. Read `tools/ACTION_LAYOUT.md` first.
|
| 134 |
+
|
| 135 |
+
## Paths to change
|
| 136 |
+
|
| 137 |
+
Every script hard-codes absolute paths from the machine this was run on. At minimum:
|
| 138 |
+
|
| 139 |
+
| what | appears as |
|
| 140 |
+
|---|---|
|
| 141 |
+
| RoboCasa / robosuite checkouts | `/lp-dev/jonghoon/robocasa_calib/repos/...` |
|
| 142 |
+
| MimicGen env + augmentation code | `/lp-dev/jonghoon/mimicgen_augment/...` |
|
| 143 |
+
| myGR00T tree and conda envs | `/data/minha/pose6daug/train_robocasa/myGR00T`, `/data/nvidia/gripper_augmentator/conda-envs/...` |
|
| 144 |
+
| base checkpoint | `/lp-dev/jonghoon/myGR00T_outputs/pnpcountertocab_all502_.../checkpoint-60000` |
|
| 145 |
+
| replay sets and eval output | `/lp-dev/jonghoon/isaac-gr00t/eval_results/...` |
|
| 146 |
+
| generation scratch | `/tmp/claude-.../scratchpad/...` |
|
| 147 |
+
|
| 148 |
+
`eval/eval_groot15_exact_replay.sh` defaults `MYGROOT_ROOT` to a path that no longer exists;
|
| 149 |
+
pass it explicitly. No credentials are embedded — the training launcher reads
|
| 150 |
+
`WANDB_API_KEY` from an env var or a file path you supply.
|
| 151 |
+
|
| 152 |
+
## Attribution
|
| 153 |
+
|
| 154 |
+
`eval/eval_groot15_exact_replay.sh` and the upstream of `dataset/render_to_gr00t.py` and the
|
| 155 |
+
eval clients come from the pose6daug project's shared tree; the copies here carry the fixes
|
| 156 |
+
described above (action reorder, mesh-swap bbox fix, target sharding, wrist-view capture,
|
| 157 |
+
stage logging).
|
| 158 |
+
|
| 159 |
+
## Related artifacts
|
| 160 |
+
|
| 161 |
+
- `mlnha/mimicgen-batch64-30k-ckpts` — checkpoints from the fixed-action-order run
|
| 162 |
+
- `mlnha/vace-batch64-30k-ckpts` — VACE run checkpoints
|
| 163 |
+
- `mlnha/mimicgen-pi05-aug256` — MimicGen augmentation for the pi0.5 hard-object set
|
augment/mimicgen/config_template.json
ADDED
|
@@ -0,0 +1,90 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"name": "pick_place_counter_to_cabinet",
|
| 3 |
+
"type": "robocasa",
|
| 4 |
+
"experiment": {
|
| 5 |
+
"name": "nat_MeasuringCup009_w0",
|
| 6 |
+
"source": {
|
| 7 |
+
"dataset_path": "/tmp/claude-1001/-home-nvidia/68c27ffb-706b-4cb3-8bf2-4618a200d378/scratchpad/big/src150_fix.hdf5",
|
| 8 |
+
"filter_key": "valid",
|
| 9 |
+
"n": null,
|
| 10 |
+
"start": null
|
| 11 |
+
},
|
| 12 |
+
"generation": {
|
| 13 |
+
"path": "/tmp/claude-1001/-home-nvidia/68c27ffb-706b-4cb3-8bf2-4618a200d378/scratchpad/natural256/nat_MeasuringCup009_w0",
|
| 14 |
+
"guarantee": false,
|
| 15 |
+
"keep_failed": false,
|
| 16 |
+
"num_trials": 49,
|
| 17 |
+
"target_initial_state": {
|
| 18 |
+
"enabled": false,
|
| 19 |
+
"mode": null,
|
| 20 |
+
"demo_key": null,
|
| 21 |
+
"demo_key_policy": "fixed",
|
| 22 |
+
"object_name": "obj",
|
| 23 |
+
"asset_path": null,
|
| 24 |
+
"model_xml_mode": "regenerate"
|
| 25 |
+
},
|
| 26 |
+
"select_src_per_subtask": false,
|
| 27 |
+
"transform_first_robot_pose": false,
|
| 28 |
+
"interpolate_from_last_target_pose": true
|
| 29 |
+
},
|
| 30 |
+
"task": {
|
| 31 |
+
"name": "PickPlaceCounterToCabinet",
|
| 32 |
+
"robot": null,
|
| 33 |
+
"gripper": null,
|
| 34 |
+
"env_meta_update_kwargs": {
|
| 35 |
+
"env_kwargs": {
|
| 36 |
+
"obj_instance_split": null,
|
| 37 |
+
"obj_groups": "/lp-dev/jonghoon/robocasa_calib/repos/robocasa/robocasa/models/assets/objects/lightwheel/measuring_cup/MeasuringCup009/model.xml"
|
| 38 |
+
}
|
| 39 |
+
},
|
| 40 |
+
"interface": "MG_PickPlaceCounterToCabinet",
|
| 41 |
+
"interface_type": "robocasa"
|
| 42 |
+
},
|
| 43 |
+
"max_num_failures": 999999,
|
| 44 |
+
"max_num_problematic": null,
|
| 45 |
+
"render_video": false,
|
| 46 |
+
"num_demo_to_render": 50,
|
| 47 |
+
"num_fail_demo_to_render": 50,
|
| 48 |
+
"log_every_n_attempts": 10,
|
| 49 |
+
"seed": 101
|
| 50 |
+
},
|
| 51 |
+
"obs": {
|
| 52 |
+
"collect_obs": false,
|
| 53 |
+
"camera_names": [],
|
| 54 |
+
"camera_height": 84,
|
| 55 |
+
"camera_width": 84
|
| 56 |
+
},
|
| 57 |
+
"task": {
|
| 58 |
+
"task_spec": {
|
| 59 |
+
"subtask_1": {
|
| 60 |
+
"object_ref": "obj",
|
| 61 |
+
"subtask_term_signal": "stage_contact_obj",
|
| 62 |
+
"subtask_term_offset_range": [
|
| 63 |
+
10,
|
| 64 |
+
20
|
| 65 |
+
],
|
| 66 |
+
"selection_strategy": "nearest_neighbor_object",
|
| 67 |
+
"selection_strategy_kwargs": {
|
| 68 |
+
"nn_k": 3
|
| 69 |
+
},
|
| 70 |
+
"action_noise": 0.0,
|
| 71 |
+
"num_interpolation_steps": 5,
|
| 72 |
+
"num_fixed_steps": 0,
|
| 73 |
+
"apply_noise_during_interpolation": false
|
| 74 |
+
},
|
| 75 |
+
"subtask_2": {
|
| 76 |
+
"object_ref": "cab",
|
| 77 |
+
"subtask_term_signal": null,
|
| 78 |
+
"subtask_term_offset_range": null,
|
| 79 |
+
"selection_strategy": "nearest_neighbor_object",
|
| 80 |
+
"selection_strategy_kwargs": {
|
| 81 |
+
"nn_k": 3
|
| 82 |
+
},
|
| 83 |
+
"action_noise": 0.0,
|
| 84 |
+
"num_interpolation_steps": 5,
|
| 85 |
+
"num_fixed_steps": 0,
|
| 86 |
+
"apply_noise_during_interpolation": false
|
| 87 |
+
}
|
| 88 |
+
}
|
| 89 |
+
}
|
| 90 |
+
}
|
augment/mimicgen/render_from_plan.py
ADDED
|
@@ -0,0 +1,132 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
"""Render episodes listed in a render plan, one mp4 per camera.
|
| 3 |
+
|
| 4 |
+
The plan fixes every folder name up front, in one global pass over all runs, so a name is
|
| 5 |
+
unique across the whole output rather than only within the run that produced it. Assigning
|
| 6 |
+
names per run is what silently dropped episodes: a later run's success collided with an
|
| 7 |
+
earlier one's folder and was skipped as already done.
|
| 8 |
+
|
| 9 |
+
Plan entries are `{run_dir, demo, target, name}`. Shard with `--shard i --num-shards n` to
|
| 10 |
+
render in parallel; each episode writes a `.done` marker, so re-running skips finished work.
|
| 11 |
+
|
| 12 |
+
MUJOCO_GL=egl python render_from_plan.py --plan render_plan.json \
|
| 13 |
+
--source src150_fix.hdf5 --filter-key valid --out output_aug_256 --shard 0 --num-shards 6
|
| 14 |
+
"""
|
| 15 |
+
from __future__ import annotations
|
| 16 |
+
|
| 17 |
+
import argparse
|
| 18 |
+
import json
|
| 19 |
+
import os
|
| 20 |
+
import time
|
| 21 |
+
|
| 22 |
+
os.environ.setdefault("MUJOCO_GL", "egl")
|
| 23 |
+
os.environ.setdefault("PYOPENGL_PLATFORM", "egl")
|
| 24 |
+
|
| 25 |
+
import h5py
|
| 26 |
+
import imageio
|
| 27 |
+
import mujoco
|
| 28 |
+
import numpy as np
|
| 29 |
+
import robocasa # noqa: F401
|
| 30 |
+
import robosuite
|
| 31 |
+
from robosuite.controllers import load_composite_controller_config
|
| 32 |
+
|
| 33 |
+
CAMERAS = ["robot0_agentview_left", "robot0_agentview_right", "robot0_eye_in_hand"]
|
| 34 |
+
|
| 35 |
+
|
| 36 |
+
def source_index(path, filter_key):
|
| 37 |
+
out = {}
|
| 38 |
+
with h5py.File(path, "r") as f:
|
| 39 |
+
keys = ([k.decode() for k in f[f"mask/{filter_key}"][:]] if filter_key
|
| 40 |
+
else sorted(f["data"].keys(), key=lambda x: int(x.split("_")[1])))
|
| 41 |
+
for i, k in enumerate(keys):
|
| 42 |
+
meta = json.loads(f[f"data/{k}"].attrs["ep_meta"])
|
| 43 |
+
cat = next((c["info"]["cat"] for c in meta.get("object_cfgs", [])
|
| 44 |
+
if c.get("name") == "obj"), "unknown")
|
| 45 |
+
out[i] = (k, cat)
|
| 46 |
+
return out
|
| 47 |
+
|
| 48 |
+
|
| 49 |
+
def build_env(env_args, res):
|
| 50 |
+
kw = dict(env_args["env_kwargs"])
|
| 51 |
+
kw.pop("env_name", None)
|
| 52 |
+
kw.update(has_renderer=False, has_offscreen_renderer=True, use_camera_obs=False,
|
| 53 |
+
use_object_obs=True, camera_depths=False, camera_names=CAMERAS,
|
| 54 |
+
camera_widths=res, camera_heights=res, ignore_done=True)
|
| 55 |
+
kw.setdefault("controller_configs",
|
| 56 |
+
load_composite_controller_config(controller=None, robot="PandaOmron"))
|
| 57 |
+
return robosuite.make(env_name=env_args["env_name"], **kw)
|
| 58 |
+
|
| 59 |
+
|
| 60 |
+
def main():
|
| 61 |
+
ap = argparse.ArgumentParser()
|
| 62 |
+
ap.add_argument("--plan", required=True)
|
| 63 |
+
ap.add_argument("--source", required=True)
|
| 64 |
+
ap.add_argument("--filter-key", default=None)
|
| 65 |
+
ap.add_argument("--out", required=True)
|
| 66 |
+
ap.add_argument("--shard", type=int, default=0)
|
| 67 |
+
ap.add_argument("--num-shards", type=int, default=1)
|
| 68 |
+
ap.add_argument("--res", type=int, default=256)
|
| 69 |
+
ap.add_argument("--fps", type=int, default=20)
|
| 70 |
+
a = ap.parse_args()
|
| 71 |
+
|
| 72 |
+
plan = [e for i, e in enumerate(json.load(open(a.plan))) if i % a.num_shards == a.shard]
|
| 73 |
+
src = source_index(a.source, a.filter_key)
|
| 74 |
+
os.makedirs(a.out, exist_ok=True)
|
| 75 |
+
|
| 76 |
+
env, opened, done = None, None, 0
|
| 77 |
+
for e in plan:
|
| 78 |
+
ep_dir = os.path.join(a.out, e["name"])
|
| 79 |
+
if os.path.exists(os.path.join(ep_dir, ".done")):
|
| 80 |
+
done += 1
|
| 81 |
+
continue
|
| 82 |
+
t0 = time.time()
|
| 83 |
+
with h5py.File(os.path.join(e["run_dir"], "demo.hdf5"), "r") as f:
|
| 84 |
+
g = f[f"data/{e['demo']}"]
|
| 85 |
+
states = g["states"][:]
|
| 86 |
+
meta = json.loads(g.attrs["ep_meta"])
|
| 87 |
+
model_xml = g.attrs["model_file"]
|
| 88 |
+
si = int(g["src_demo_inds"][0]) if "src_demo_inds" in g else -1
|
| 89 |
+
env_args = json.loads(f["data"].attrs["env_args"])
|
| 90 |
+
sk, scat = src.get(si, (f"idx{si}", "unknown"))
|
| 91 |
+
|
| 92 |
+
if env is None:
|
| 93 |
+
env = build_env(env_args, a.res)
|
| 94 |
+
env.set_ep_meta(meta)
|
| 95 |
+
env.reset()
|
| 96 |
+
env.reset_from_xml_string(model_xml)
|
| 97 |
+
env.sim.reset()
|
| 98 |
+
|
| 99 |
+
model, data = env.sim.model._model, env.sim.data._data
|
| 100 |
+
opt = mujoco.MjvOption()
|
| 101 |
+
opt.geomgroup[:] = 0
|
| 102 |
+
opt.geomgroup[1] = 1 # visual geoms, as the dataset's own videos are drawn
|
| 103 |
+
opt.sitegroup[:] = 0
|
| 104 |
+
r = mujoco.Renderer(model, height=a.res, width=a.res)
|
| 105 |
+
cam_ids = {c: mujoco.mj_name2id(model, mujoco.mjtObj.mjOBJ_CAMERA, c) for c in CAMERAS}
|
| 106 |
+
frames = {c: [] for c in CAMERAS}
|
| 107 |
+
for t in range(len(states)):
|
| 108 |
+
env.sim.set_state_from_flattened(states[t])
|
| 109 |
+
env.sim.forward()
|
| 110 |
+
for c in CAMERAS:
|
| 111 |
+
r.update_scene(data, camera=cam_ids[c], scene_option=opt)
|
| 112 |
+
frames[c].append(r.render().copy())
|
| 113 |
+
r.close()
|
| 114 |
+
|
| 115 |
+
for c in CAMERAS:
|
| 116 |
+
d = os.path.join(ep_dir, c)
|
| 117 |
+
os.makedirs(d, exist_ok=True)
|
| 118 |
+
imageio.mimwrite(os.path.join(d, f"{c}.mp4"), frames[c], fps=a.fps,
|
| 119 |
+
codec="libx264", macro_block_size=1,
|
| 120 |
+
ffmpeg_params=["-pix_fmt", "yuv420p"])
|
| 121 |
+
with open(os.path.join(ep_dir, ".done"), "w") as fh:
|
| 122 |
+
json.dump(dict(generated_demo=e["demo"], run_dir=e["run_dir"],
|
| 123 |
+
source_demo=sk, source_object=scat, target_object=e["target"],
|
| 124 |
+
frames=len(states), layout=meta.get("layout_id"),
|
| 125 |
+
style=meta.get("style_id"), lang=meta.get("lang")), fh, indent=1)
|
| 126 |
+
done += 1
|
| 127 |
+
print(f" [{a.shard}] {e['name']}: {len(states)}f {time.time()-t0:.0f}s", flush=True)
|
| 128 |
+
print(f"shard {a.shard}: {done}/{len(plan)}", flush=True)
|
| 129 |
+
|
| 130 |
+
|
| 131 |
+
if __name__ == "__main__":
|
| 132 |
+
raise SystemExit(main())
|
augment/mimicgen/run_generation.sh
ADDED
|
@@ -0,0 +1,13 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env bash
|
| 2 |
+
# Launch all 32 equal-attempt generation jobs at once (one core each, no GPU).
|
| 3 |
+
set -u
|
| 4 |
+
MG=/lp-dev/jonghoon/mimicgen_augment/envs/mimicgen/bin/python
|
| 5 |
+
cd /home/nvidia/minha/pose6daug/baseline/mimicgen/mimicgen_mh || exit 1
|
| 6 |
+
OUT=/tmp/claude-1001/-home-nvidia/68c27ffb-706b-4cb3-8bf2-4618a200d378/scratchpad/natural256
|
| 7 |
+
date -u +"[nat] start %FT%TZ" | tee "$OUT/chain.log"
|
| 8 |
+
while IFS=$'\t' read -r name cfg log; do
|
| 9 |
+
MUJOCO_GL=egl nohup "$MG" mimicgen/scripts/generate_dataset.py \
|
| 10 |
+
--config "$cfg" --auto-remove-exp >"$log" 2>&1 &
|
| 11 |
+
done < "$OUT/jobs.txt"
|
| 12 |
+
wait
|
| 13 |
+
date -u +"[nat] generation done %FT%TZ" | tee -a "$OUT/chain.log"
|
augment/mimicgen/run_render.sh
ADDED
|
@@ -0,0 +1,23 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env bash
|
| 2 |
+
# Preview mp4s, original RoboCasa asset, 3 cameras -- same layout as output_aug_256.
|
| 3 |
+
# taskset escapes this shell's 64-CPU affinity mask; EGL device round-robins over the
|
| 4 |
+
# three free GPUs (4-7 belong to another user's training).
|
| 5 |
+
set -u
|
| 6 |
+
N=/tmp/claude-1001/-home-nvidia/68c27ffb-706b-4cb3-8bf2-4618a200d378/scratchpad/natural256
|
| 7 |
+
S=/tmp/claude-1001/-home-nvidia/68c27ffb-706b-4cb3-8bf2-4618a200d378/scratchpad
|
| 8 |
+
MG=/lp-dev/jonghoon/mimicgen_augment/envs/mimicgen/bin/python
|
| 9 |
+
OUT=/home/nvidia/minha/pose6daug/baseline/mimicgen/output_natural_256
|
| 10 |
+
SHARDS=${SHARDS:-16}
|
| 11 |
+
GPUS=(0 2 3)
|
| 12 |
+
mkdir -p "$OUT" "$N/logs"
|
| 13 |
+
cd /home/nvidia/minha/pose6daug/baseline/mimicgen/mimicgen_augment || exit 1
|
| 14 |
+
for i in $(seq 0 $((SHARDS-1))); do
|
| 15 |
+
g=${GPUS[$((i % ${#GPUS[@]}))]}
|
| 16 |
+
MUJOCO_GL=egl PYOPENGL_PLATFORM=egl MUJOCO_EGL_DEVICE_ID=$g \
|
| 17 |
+
taskset -c 0-254 nohup "$MG" -u render_from_plan.py \
|
| 18 |
+
--plan "$N/render_plan.json" --source "$S/big/src150_fix.hdf5" \
|
| 19 |
+
--filter-key valid --out "$OUT" --shard "$i" --num-shards "$SHARDS" \
|
| 20 |
+
> "$N/logs/render_$i.log" 2>&1 &
|
| 21 |
+
done
|
| 22 |
+
wait
|
| 23 |
+
echo "[nat] render done $(date -u +%FT%TZ)"
|
augment/mimicgen/select_by_generation_time.py
ADDED
|
@@ -0,0 +1,143 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
"""Pick the first 256 episodes in true generation order and lay out the work for both
|
| 3 |
+
downstream stages.
|
| 4 |
+
|
| 5 |
+
Every worker writes each success to tmp/date_..._time_HH_MM_SS.hdf5 before merge_all_hdf5
|
| 6 |
+
sorts those files by timestamp into demo_0..demo_N and deletes the folder. snapshot_times.sh
|
| 7 |
+
captured the filenames while they existed, so joining them back gives each merged demo its
|
| 8 |
+
wall-clock time -- and since all 32 workers share one machine clock, a global ordering.
|
| 9 |
+
|
| 10 |
+
Writes:
|
| 11 |
+
selection.json the ordered 256 with their timestamps
|
| 12 |
+
render_plan.json for render_from_plan.py (preview mp4s, original asset)
|
| 13 |
+
gen/shard_K/... per-shard gen trees for render_to_gr00t.py (dataset, swapped mesh)
|
| 14 |
+
"""
|
| 15 |
+
import json, os, re, glob, sys, collections, datetime
|
| 16 |
+
import h5py
|
| 17 |
+
|
| 18 |
+
S = "/tmp/claude-1001/-home-nvidia/68c27ffb-706b-4cb3-8bf2-4618a200d378/scratchpad"
|
| 19 |
+
NAT = f"{S}/natural256"
|
| 20 |
+
SRC = f"{S}/big/src150_fix.hdf5"
|
| 21 |
+
N_KEEP = int(os.environ.get("N_KEEP", 256))
|
| 22 |
+
N_CONV_SHARDS = int(os.environ.get("N_CONV_SHARDS", 24))
|
| 23 |
+
|
| 24 |
+
CATEGORY = {
|
| 25 |
+
"donut_5": "donut", "Jar023": "jar", "MeasuringCup009": "measuring_cup",
|
| 26 |
+
"SoapDispenser010": "soap_dispenser", "steak_8": "steak",
|
| 27 |
+
"SyrupBottle005": "syrup_bottle", "teapot_6": "teapot", "teapot_7": "teapot",
|
| 28 |
+
}
|
| 29 |
+
|
| 30 |
+
# ---- source demo index -> (demo key, the object that source episode had) ----
|
| 31 |
+
src = {}
|
| 32 |
+
with h5py.File(SRC, "r") as f:
|
| 33 |
+
keys = [k.decode() for k in f["mask/valid"][:]]
|
| 34 |
+
for i, k in enumerate(keys):
|
| 35 |
+
meta = json.loads(f[f"data/{k}"].attrs["ep_meta"])
|
| 36 |
+
cat = next((c["info"]["cat"] for c in meta.get("object_cfgs", [])
|
| 37 |
+
if c.get("name") == "obj"), "unknown")
|
| 38 |
+
src[i] = (k, cat)
|
| 39 |
+
|
| 40 |
+
# ---- timestamps captured before the tmp folders were deleted ----
|
| 41 |
+
# One line per poll, so the same file appears many times and its mtime drifts while it is
|
| 42 |
+
# still being written -- dedupe on the name and order by the timestamp the name itself
|
| 43 |
+
# carries, which is fixed at creation.
|
| 44 |
+
seen = collections.defaultdict(dict)
|
| 45 |
+
with open(f"{NAT}/episode_times.tsv") as f:
|
| 46 |
+
for line in f:
|
| 47 |
+
parts = line.rstrip("\n").split("\t")
|
| 48 |
+
if len(parts) != 3:
|
| 49 |
+
continue
|
| 50 |
+
tmpdir, fname, _ = parts
|
| 51 |
+
m = re.match(r"date_(\d+)_(\d+)_(\d+)_time_(\d+)_(\d+)_(\d+)\.hdf5", fname)
|
| 52 |
+
if not m:
|
| 53 |
+
continue
|
| 54 |
+
worker = os.path.basename(os.path.dirname(os.path.dirname(tmpdir)))
|
| 55 |
+
mo, dy, yr, hh, mm, ss = (int(x) for x in m.groups())
|
| 56 |
+
seen[worker][fname] = datetime.datetime(
|
| 57 |
+
yr, mo, dy, hh, mm, ss, tzinfo=datetime.timezone.utc).timestamp()
|
| 58 |
+
times = {w: sorted((v, k) for k, v in d.items()) for w, d in seen.items()}
|
| 59 |
+
|
| 60 |
+
rows, missing = [], []
|
| 61 |
+
for wdir in sorted(glob.glob(f"{NAT}/nat_*")) + sorted(glob.glob(f"{NAT}/r2_*")):
|
| 62 |
+
worker = os.path.basename(wdir)
|
| 63 |
+
model = re.sub(r"^(nat|r2)_|_w\d+$", "", worker)
|
| 64 |
+
hp = f"{wdir}/{worker}/demo.hdf5"
|
| 65 |
+
if not os.path.isdir(wdir) or not os.path.exists(hp):
|
| 66 |
+
continue
|
| 67 |
+
with h5py.File(hp, "r") as f:
|
| 68 |
+
demos = sorted(f["data"].keys(), key=lambda x: int(x.split("_")[1]))
|
| 69 |
+
stamps = list(times.get(worker, []))
|
| 70 |
+
# The poller can miss the very last success: the worker merges and deletes tmp/
|
| 71 |
+
# immediately after writing it. Only the tail can ever be missing (earlier files sat
|
| 72 |
+
# there for the whole run), so pad it with the merged file's own mtime -- the moment
|
| 73 |
+
# the worker finished, which bounds that episode from above.
|
| 74 |
+
if len(stamps) < len(demos):
|
| 75 |
+
end = os.path.getmtime(hp)
|
| 76 |
+
pad = len(demos) - len(stamps)
|
| 77 |
+
missing.append(f"{worker}: padded {pad} with run end")
|
| 78 |
+
stamps += [(end, "<run end>")] * pad
|
| 79 |
+
for i, dk in enumerate(demos):
|
| 80 |
+
g = f[f"data/{dk}"]
|
| 81 |
+
si = int(g["src_demo_inds"][0]) if "src_demo_inds" in g else -1
|
| 82 |
+
sk, scat = src.get(si, (f"idx{si}", "unknown"))
|
| 83 |
+
t = stamps[i][0] if i < len(stamps) else float("inf")
|
| 84 |
+
rows.append(dict(t=t, worker=worker, run_dir=f"{wdir}/{worker}", demo=dk,
|
| 85 |
+
target=model, source_demo=sk, source_object=scat))
|
| 86 |
+
|
| 87 |
+
rows.sort(key=lambda r: (r["t"], r["worker"], r["demo"]))
|
| 88 |
+
print(f"총 성공 {len(rows)}개" + (f" (경고: {missing})" if missing else ""))
|
| 89 |
+
for i, r in enumerate(rows):
|
| 90 |
+
r["order"] = i
|
| 91 |
+
kept = rows[:N_KEEP]
|
| 92 |
+
|
| 93 |
+
taken = collections.Counter()
|
| 94 |
+
for r in kept:
|
| 95 |
+
base = f"{r['source_demo'].replace('demo_', 'ep')}_{r['source_object']}_{r['target']}"
|
| 96 |
+
taken[base] += 1
|
| 97 |
+
r["name"] = base if taken[base] == 1 else f"{base}_{taken[base]}"
|
| 98 |
+
|
| 99 |
+
json.dump(kept, open(f"{NAT}/selection.json", "w"), indent=1)
|
| 100 |
+
json.dump([{k: r[k] for k in ("run_dir", "demo", "target", "name")} for r in kept],
|
| 101 |
+
open(f"{NAT}/render_plan.json", "w"), indent=1)
|
| 102 |
+
|
| 103 |
+
# ---- conversion shards: contiguous chunks, each its own GEN_DIR tree ----
|
| 104 |
+
by_target = collections.defaultdict(list)
|
| 105 |
+
for r in kept:
|
| 106 |
+
by_target[r["target"]].append(r)
|
| 107 |
+
chunks = []
|
| 108 |
+
per = max(1, -(-len(kept) // N_CONV_SHARDS))
|
| 109 |
+
for model, rs in by_target.items():
|
| 110 |
+
for i in range(0, len(rs), per):
|
| 111 |
+
chunks.append((model, rs[i:i + per]))
|
| 112 |
+
|
| 113 |
+
for k, (model, rs) in enumerate(chunks):
|
| 114 |
+
name = f"{CATEGORY[model]}_{model}"
|
| 115 |
+
d = f"{NAT}/gen/shard_{k:02d}/{name}/{name}_gen"
|
| 116 |
+
os.makedirs(d, exist_ok=True)
|
| 117 |
+
out = f"{d}/demo.hdf5"
|
| 118 |
+
if os.path.exists(out):
|
| 119 |
+
os.remove(out)
|
| 120 |
+
with h5py.File(out, "w") as fo:
|
| 121 |
+
grp = fo.create_group("data")
|
| 122 |
+
total, env_args = 0, None
|
| 123 |
+
by_run = collections.defaultdict(list)
|
| 124 |
+
for r in rs:
|
| 125 |
+
by_run[r["run_dir"]].append(r)
|
| 126 |
+
j = 0
|
| 127 |
+
for run_dir, items in by_run.items():
|
| 128 |
+
with h5py.File(f"{run_dir}/demo.hdf5", "r") as fi:
|
| 129 |
+
if env_args is None:
|
| 130 |
+
env_args = fi["data"].attrs["env_args"]
|
| 131 |
+
for r in items:
|
| 132 |
+
fi.copy(f"data/{r['demo']}", grp, name=f"demo_{j}")
|
| 133 |
+
total += int(fi[f"data/{r['demo']}"].attrs["num_samples"])
|
| 134 |
+
j += 1
|
| 135 |
+
grp.attrs["total"] = total
|
| 136 |
+
grp.attrs["env_args"] = env_args
|
| 137 |
+
|
| 138 |
+
print(f"선택 {len(kept)}개 -> render_plan.json, 변환 샤드 {len(chunks)}개 "
|
| 139 |
+
f"(샤드당 최대 {per})")
|
| 140 |
+
c = collections.Counter(r["target"] for r in kept)
|
| 141 |
+
print("\n물체별 분포:")
|
| 142 |
+
for k, v in c.most_common():
|
| 143 |
+
print(f" {k:18s} {v}")
|
augment/mimicgen/select_by_generation_time_pi05.py
ADDED
|
@@ -0,0 +1,80 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
"""Order every success by real generation time and keep the first 256.
|
| 3 |
+
|
| 4 |
+
Each worker writes a success to tmp/date_..._time_HH_MM_SS.hdf5 before merge_all_hdf5 sorts
|
| 5 |
+
those by timestamp into demo_0..demo_N and deletes the folder; snapshot.sh captured the names
|
| 6 |
+
while they existed. The poller can only ever miss a worker's LAST file (written immediately
|
| 7 |
+
before the merge), so a short tail is padded with the merged file's mtime.
|
| 8 |
+
"""
|
| 9 |
+
import json, os, re, glob, collections, datetime
|
| 10 |
+
import h5py
|
| 11 |
+
|
| 12 |
+
S = "/tmp/claude-1001/-home-nvidia/68c27ffb-706b-4cb3-8bf2-4618a200d378/scratchpad"
|
| 13 |
+
Q = f"{S}/pi05_256"
|
| 14 |
+
SRC = f"{S}/big/src150_fix.hdf5"
|
| 15 |
+
N_KEEP = int(os.environ.get("N_KEEP", 256))
|
| 16 |
+
|
| 17 |
+
src = {}
|
| 18 |
+
with h5py.File(SRC, "r") as f:
|
| 19 |
+
for i, k in enumerate([k.decode() for k in f["mask/valid"][:]]):
|
| 20 |
+
meta = json.loads(f[f"data/{k}"].attrs["ep_meta"])
|
| 21 |
+
src[i] = (k, next((c["info"]["cat"] for c in meta.get("object_cfgs", [])
|
| 22 |
+
if c.get("name") == "obj"), "unknown"))
|
| 23 |
+
|
| 24 |
+
seen = collections.defaultdict(dict)
|
| 25 |
+
with open(f"{Q}/episode_times.tsv") as f:
|
| 26 |
+
for line in f:
|
| 27 |
+
p = line.rstrip("\n").split("\t")
|
| 28 |
+
if len(p) != 3:
|
| 29 |
+
continue
|
| 30 |
+
m = re.match(r"date_(\d+)_(\d+)_(\d+)_time_(\d+)_(\d+)_(\d+)\.hdf5", p[1])
|
| 31 |
+
if not m:
|
| 32 |
+
continue
|
| 33 |
+
w = os.path.basename(os.path.dirname(os.path.dirname(p[0])))
|
| 34 |
+
mo, dy, yr, hh, mm, ss = (int(x) for x in m.groups())
|
| 35 |
+
seen[w][p[1]] = datetime.datetime(yr, mo, dy, hh, mm, ss,
|
| 36 |
+
tzinfo=datetime.timezone.utc).timestamp()
|
| 37 |
+
times = {w: sorted(v for v in d.values()) for w, d in seen.items()}
|
| 38 |
+
|
| 39 |
+
rows, notes = [], []
|
| 40 |
+
for wdir in sorted(glob.glob(f"{Q}/q_*")):
|
| 41 |
+
if not os.path.isdir(wdir):
|
| 42 |
+
continue
|
| 43 |
+
worker = os.path.basename(wdir)
|
| 44 |
+
model = re.sub(r"^q_|_w\d+$", "", worker)
|
| 45 |
+
hp = f"{wdir}/{worker}/demo.hdf5"
|
| 46 |
+
if not os.path.exists(hp):
|
| 47 |
+
continue
|
| 48 |
+
with h5py.File(hp, "r") as f:
|
| 49 |
+
demos = sorted(f["data"].keys(), key=lambda x: int(x.split("_")[1]))
|
| 50 |
+
st = list(times.get(worker, []))
|
| 51 |
+
if len(st) < len(demos):
|
| 52 |
+
st += [os.path.getmtime(hp)] * (len(demos) - len(st))
|
| 53 |
+
notes.append(f"{worker}+{len(demos)-len(times.get(worker, []))}")
|
| 54 |
+
for i, dk in enumerate(demos):
|
| 55 |
+
g = f[f"data/{dk}"]
|
| 56 |
+
si = int(g["src_demo_inds"][0]) if "src_demo_inds" in g else -1
|
| 57 |
+
sk, scat = src.get(si, (f"idx{si}", "unknown"))
|
| 58 |
+
rows.append(dict(t=st[i], worker=worker, run_dir=f"{wdir}/{worker}",
|
| 59 |
+
demo=dk, target=model, source_demo=sk, source_object=scat))
|
| 60 |
+
|
| 61 |
+
rows.sort(key=lambda r: (r["t"], r["worker"], r["demo"]))
|
| 62 |
+
for i, r in enumerate(rows):
|
| 63 |
+
r["order"] = i
|
| 64 |
+
kept = rows[:N_KEEP]
|
| 65 |
+
taken = collections.Counter()
|
| 66 |
+
for r in kept:
|
| 67 |
+
base = f"{r['source_demo'].replace('demo_', 'ep')}_{r['source_object']}_{r['target']}"
|
| 68 |
+
taken[base] += 1
|
| 69 |
+
r["name"] = base if taken[base] == 1 else f"{base}_{taken[base]}"
|
| 70 |
+
|
| 71 |
+
json.dump(kept, open(f"{Q}/selection.json", "w"), indent=1)
|
| 72 |
+
json.dump([{k: r[k] for k in ("run_dir", "demo", "target", "name")} for r in kept],
|
| 73 |
+
open(f"{Q}/render_plan.json", "w"), indent=1)
|
| 74 |
+
|
| 75 |
+
print(f"총 성공 {len(rows)}개 -> 선택 {len(kept)}개" + (f" (tail 패딩: {len(notes)}워커)" if notes else ""))
|
| 76 |
+
tot = collections.Counter(r["target"] for r in rows)
|
| 77 |
+
sel = collections.Counter(r["target"] for r in kept)
|
| 78 |
+
print(f"\n{'물체':20s} {'전체':>6s} {'선택256':>8s}")
|
| 79 |
+
for m, _ in tot.most_common():
|
| 80 |
+
print(f" {m:20s} {tot[m]:6d} {sel[m]:8d}")
|
augment/mimicgen/snapshot_episode_times.sh
ADDED
|
@@ -0,0 +1,16 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env bash
|
| 2 |
+
# Per-episode hdf5s are named with their completion timestamp, but merge_all_hdf5 deletes
|
| 3 |
+
# the tmp folder once a worker finishes. Snapshot the names while they still exist -- this
|
| 4 |
+
# is the only record of when each episode was actually produced.
|
| 5 |
+
S=/tmp/claude-1001/-home-nvidia/68c27ffb-706b-4cb3-8bf2-4618a200d378/scratchpad/natural256
|
| 6 |
+
M=$S/episode_times.tsv
|
| 7 |
+
touch "$M"
|
| 8 |
+
while pgrep -f "generate_dataset.py --config $S" >/dev/null; do
|
| 9 |
+
find "$S" -path "*/tmp/*.hdf5" -printf '%h\t%f\t%T@\n' 2>/dev/null >> "$M.new"
|
| 10 |
+
sort -u "$M" "$M.new" 2>/dev/null > "$M.tmp" && mv "$M.tmp" "$M"
|
| 11 |
+
rm -f "$M.new"
|
| 12 |
+
sleep 15
|
| 13 |
+
done
|
| 14 |
+
find "$S" -path "*/tmp/*.hdf5" -printf '%h\t%f\t%T@\n' 2>/dev/null >> "$M"
|
| 15 |
+
sort -u "$M" -o "$M"
|
| 16 |
+
echo "snapshot done: $(wc -l < "$M") episodes"
|
augment/vace/DATASET_CONVERSION.md
ADDED
|
@@ -0,0 +1,182 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Turning VACE output into a trainable dataset — everything needed
|
| 2 |
+
|
| 3 |
+
After `run_aug32.sh` finishes you have per-episode videos. Training needs a LeRobot v2.1
|
| 4 |
+
"gr00t_views" dataset. This is the full inventory for doing that conversion on a machine
|
| 5 |
+
that has nothing but this repo.
|
| 6 |
+
|
| 7 |
+
Written 2026-09-19. The MimicGen side of the same job is documented in
|
| 8 |
+
`baseline/mimicgen/TRAINING_DATASET_PREP.md`; **the two are not the same procedure** — see
|
| 9 |
+
"Why VACE is the easy case" below.
|
| 10 |
+
|
| 11 |
+
---
|
| 12 |
+
|
| 13 |
+
## Why VACE is the easy case
|
| 14 |
+
|
| 15 |
+
VACE repaints pixels; it never changes the trajectory. So the augmented episode's
|
| 16 |
+
state/action rows are **the source episode's own rows, unchanged**. The builder copies that
|
| 17 |
+
parquet, rewrites indices and task ids, and links the new videos in. No simulator, no
|
| 18 |
+
re-render, no MuJoCo.
|
| 19 |
+
|
| 20 |
+
MimicGen is the opposite: every episode is a new rollout, so its parquet has to be rebuilt
|
| 21 |
+
from the generation HDF5 by replaying it in the simulator (~2 h for 256 episodes).
|
| 22 |
+
|
| 23 |
+
| | VACE | MimicGen |
|
| 24 |
+
|---|---|---|
|
| 25 |
+
| builder | `build_vace_objwise_gr00t_dataset.py` | `render_to_gr00t.py` |
|
| 26 |
+
| parquet | copied from the source episode | rebuilt from HDF5 |
|
| 27 |
+
| video | symlinked from the VACE output | re-rendered in MuJoCo |
|
| 28 |
+
| needs robosuite/robocasa/mujoco | **no** | yes |
|
| 29 |
+
| runtime for 256 eps | minutes | ~2 h |
|
| 30 |
+
|
| 31 |
+
---
|
| 32 |
+
|
| 33 |
+
## What to bring
|
| 34 |
+
|
| 35 |
+
### 1. The builder and its checker
|
| 36 |
+
|
| 37 |
+
```
|
| 38 |
+
train_robocasa/scripts/dataset_build/build_vace_objwise_gr00t_dataset.py
|
| 39 |
+
train_robocasa/scripts/dataset_build/repair_mimicgen_task_ids.py # also needed here
|
| 40 |
+
train_robocasa/scripts/dataset_build/verify_geom_objwise_gr00t_dataset.py
|
| 41 |
+
train_robocasa/scripts/dataset_build/merge_gr00t_view_datasets.py # only if merging with the 502 base
|
| 42 |
+
train_robocasa/scripts/dataset_build/verify_merged_gr00t_dataset.py
|
| 43 |
+
```
|
| 44 |
+
|
| 45 |
+
Only `numpy` and `pandas` beyond the standard library, plus `ffmpeg` on `PATH` for the
|
| 46 |
+
optional three-view preview (`--skip-three-view` avoids even that).
|
| 47 |
+
|
| 48 |
+
### 2. Inputs
|
| 49 |
+
|
| 50 |
+
| `--flag` | what | where it comes from |
|
| 51 |
+
|---|---|---|
|
| 52 |
+
| `--vace-root` | the VACE output tree, one directory per assignment holding `work_<view>/<inference_dir>/out_video.mp4` | `run_aug32.sh`'s work dir |
|
| 53 |
+
| `--manifest` | `assignments.jsonl` (repeatable) | `aug32/build_aug32_assignments.py` |
|
| 54 |
+
| `--source-root` | the original RoboCasa dataset — **the parquet is taken from here** | `datasets_pack/source_episodes.tar` |
|
| 55 |
+
| `--canonical-meta-source` | a dataset whose `info.json` / `modality.json` / `view_manifest.json` contract is copied | **`dataset_build/canonical_meta/`**, bundled here |
|
| 56 |
+
| `--output` | where the dataset is written | your choice |
|
| 57 |
+
|
| 58 |
+
The defaults all point at the original machine, so **pass all five explicitly**.
|
| 59 |
+
|
| 60 |
+
### The 502-episode gr00t_views set is NOT needed
|
| 61 |
+
|
| 62 |
+
Neither for training (we do not merge) nor for conversion:
|
| 63 |
+
|
| 64 |
+
- its `meta/` was the only thing `--canonical-meta-source` reads, and those six files are
|
| 65 |
+
bundled at `dataset_build/canonical_meta/` (96 KB);
|
| 66 |
+
- the state/action rows come from `--source-root`, i.e. `pickplace_target_human`, whose
|
| 67 |
+
parquet already carries **state 16-d and action 12-d** — the same schema as gr00t_views.
|
| 68 |
+
|
| 69 |
+
`--canonical-meta-source` expects a *dataset directory* and reads `<dir>/meta/`, so
|
| 70 |
+
`dataset_build/canonical_dataset/` is set up for exactly that — its `meta` is a symlink to
|
| 71 |
+
`canonical_meta`. Pass `--canonical-meta-source dataset_build/canonical_dataset`.
|
| 72 |
+
|
| 73 |
+
### 3. Manifest fields the builder reads
|
| 74 |
+
|
| 75 |
+
`case_dir`, `episode_index`, `provenance`, and the whole `source` block (episode, task,
|
| 76 |
+
`lang`, `source_mesh`, `source_category`, per-camera `videos`). `build_aug32_assignments.py`
|
| 77 |
+
writes all of them.
|
| 78 |
+
|
| 79 |
+
`provenance` decides what gets in:
|
| 80 |
+
|
| 81 |
+
```python
|
| 82 |
+
row.setdefault("provenance", "existing")
|
| 83 |
+
if row["provenance"] == "random_fill" and not include_random_fill:
|
| 84 |
+
continue # dropped
|
| 85 |
+
```
|
| 86 |
+
|
| 87 |
+
Our builder marks same-category assignments `existing` and the random remainder
|
| 88 |
+
`random_fill`. With 32 per target that is **59 same-category and 197 random**, so the
|
| 89 |
+
default keeps only 59. **Pass `--include-random-fill` to get all 256.**
|
| 90 |
+
|
| 91 |
+
---
|
| 92 |
+
|
| 93 |
+
## Run it
|
| 94 |
+
|
| 95 |
+
```bash
|
| 96 |
+
python build_vace_objwise_gr00t_dataset.py \
|
| 97 |
+
--vace-root /path/to/vace_aug32_work \
|
| 98 |
+
--manifest /path/to/vace_aug32_work/assignments.jsonl \
|
| 99 |
+
--source-root $PWD/data/pickplace_target_human \
|
| 100 |
+
--canonical-meta-source $PWD/dataset_build/canonical_dataset \
|
| 101 |
+
--output /path/to/gr00t_views/vace_per_target_32_256eps \
|
| 102 |
+
--include-random-fill \
|
| 103 |
+
--skip-three-view
|
| 104 |
+
|
| 105 |
+
python repair_mimicgen_task_ids.py --dataset /path/to/gr00t_views/vace_per_target_32_256eps
|
| 106 |
+
python verify_geom_objwise_gr00t_dataset.py --dataset /path/to/gr00t_views/vace_per_target_32_256eps
|
| 107 |
+
```
|
| 108 |
+
|
| 109 |
+
`--copy-videos` materialises the videos instead of symlinking — needed if the dataset will
|
| 110 |
+
be moved away from the VACE output tree.
|
| 111 |
+
|
| 112 |
+
### `repair_mimicgen_task_ids.py` is not optional
|
| 113 |
+
|
| 114 |
+
Its docstring: the writer "stores all parquet task columns as 0 while episodes.jsonl stores
|
| 115 |
+
the intended language per episode", and training reads the task id from the parquet — so
|
| 116 |
+
without this pass **every augmented episode trains as task 0** and the language conditioning
|
| 117 |
+
silently collapses. It also renames the modality keys to the canonical
|
| 118 |
+
`left_view` / `right_view` / `wrist_view` that `single_panda_gripper` expects.
|
| 119 |
+
|
| 120 |
+
---
|
| 121 |
+
|
| 122 |
+
## What the result must look like
|
| 123 |
+
|
| 124 |
+
```
|
| 125 |
+
<output>/
|
| 126 |
+
meta/{info,modality,episodes,tasks,stats}.json all five, asserted by the loader
|
| 127 |
+
data/chunk-000/episode_NNNNNN.parquet
|
| 128 |
+
videos/chunk-000/observation.images.robot0_agentview_left/episode_NNNNNN.mp4
|
| 129 |
+
.../robot0_agentview_right/... .../robot0_eye_in_hand/...
|
| 130 |
+
extras/episode_NNNNNN/... eval only, training never reads it
|
| 131 |
+
```
|
| 132 |
+
|
| 133 |
+
parquet columns (11): `observation.state` (list, **16**), `action` (list, **12**),
|
| 134 |
+
`next.reward`, `next.done`, `timestamp`, `frame_index`, `episode_index`, `index`,
|
| 135 |
+
`task_index`, `annotation.human.task_description` (**int index into tasks.jsonl**, not the
|
| 136 |
+
string), `annotation.human.task_name`.
|
| 137 |
+
|
| 138 |
+
`modality.json` slicing:
|
| 139 |
+
|
| 140 |
+
```
|
| 141 |
+
state base_position 0:3 | base_rotation 3:7 (quat) | EE_pos_rel 7:10
|
| 142 |
+
| EE_rot_rel 10:14 (quat) | gripper_qpos 14:16
|
| 143 |
+
action base_motion 0:4 | control_mode 4:5 | EE_pos 5:8 | EE_rot 8:11 | gripper_close 11:12
|
| 144 |
+
video left_view / right_view / wrist_view -> observation.images.robot0_agentview_{left,right}
|
| 145 |
+
and observation.images.robot0_eye_in_hand
|
| 146 |
+
```
|
| 147 |
+
|
| 148 |
+
---
|
| 149 |
+
|
| 150 |
+
## Then training
|
| 151 |
+
|
| 152 |
+
`train_robocasa/` is a separate self-contained bundle (49 MB + the `mygr00t` conda env and a
|
| 153 |
+
7.1 GB base checkpoint, both referenced by absolute path). Point it at the dataset by
|
| 154 |
+
absolute path rather than by name:
|
| 155 |
+
|
| 156 |
+
```bash
|
| 157 |
+
cd train_robocasa
|
| 158 |
+
DATASET_PATH=/path/to/gr00t_views/vace_per_target_32_256eps \
|
| 159 |
+
GPUS=0,1,2,3 MAX_STEPS=30000 SAVE_STEPS=5000 PER_GPU_BATCH=16 \
|
| 160 |
+
bash scripts/train_groot15_single_dataset.sh
|
| 161 |
+
```
|
| 162 |
+
|
| 163 |
+
`DATASETNAME` (the alternative) resolves against
|
| 164 |
+
`/lp-dev/jonghoon/robocasa_full/gr00t_views/`, which will not exist elsewhere.
|
| 165 |
+
|
| 166 |
+
Check first with `bash scripts/smoke_test.sh <gpu>` — imports, dataset load, and a real
|
| 167 |
+
20-step fine-tune.
|
| 168 |
+
|
| 169 |
+
---
|
| 170 |
+
|
| 171 |
+
## Checklist for the other machine
|
| 172 |
+
|
| 173 |
+
- [ ] `run_aug32.sh` finished; every assignment has three `out_video.mp4`
|
| 174 |
+
- [ ] `assignments.jsonl` carries `provenance` (re-run `build_aug32_assignments.py` if it predates 2026-09-19)
|
| 175 |
+
- [ ] `source_episodes.tar` unpacked — **the parquet (`data/`) is read from it**; the archive
|
| 176 |
+
was repacked 2026-09-19 to include it, an earlier copy has only `videos/` and `extras/`
|
| 177 |
+
- [ ] `dataset_build/canonical_dataset/` present (its `meta` symlink must resolve; use
|
| 178 |
+
`cp -rL` when moving)
|
| 179 |
+
- [ ] `build_vace_objwise_gr00t_dataset.py` + `repair_mimicgen_task_ids.py` + a verifier copied over
|
| 180 |
+
- [ ] python with `numpy`, `pandas`; `ffmpeg` unless `--skip-three-view`
|
| 181 |
+
- [ ] decided: `--include-random-fill` (256) or not (59)
|
| 182 |
+
- [ ] decided: merge with the 502 base or train on the augmented set alone
|
augment/vace/README.md
ADDED
|
@@ -0,0 +1,211 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# VACE augmentation — 8 objects × 32 episodes, 3 cameras
|
| 2 |
+
|
| 3 |
+
Swaps a target mesh into RoboCasa episodes with the VACE video-diffusion model, producing
|
| 4 |
+
the same folder layout as the MimicGen augmentation run so the two baselines can be
|
| 5 |
+
compared file-for-file.
|
| 6 |
+
|
| 7 |
+
**Same-category first, random for the remainder.** For `donut_5`, episodes that already
|
| 8 |
+
contain a donut (`donut_8`, `donut_9`, …) are used first — those are the swaps VACE was
|
| 9 |
+
designed for. Only after that pool runs out does the rest of the 32 come from randomly
|
| 10 |
+
drawn episodes of any category (apple → donut_5, milk → donut_5, …). The split is recorded
|
| 11 |
+
per assignment as `fill: same_category | random`.
|
| 12 |
+
|
| 13 |
+
How far the same-category pool goes, after `make_gt_masks.py` filled in the 58 missing
|
| 14 |
+
same-category masks (episodes with a complete set of GT masks, excluding the target mesh):
|
| 15 |
+
|
| 16 |
+
| target | same-category pool | same-category used | random fill | reaches 32 same-category? |
|
| 17 |
+
|---|---|---|---|---|
|
| 18 |
+
| MeasuringCup009 | 57 | **32** | 0 | **yes** |
|
| 19 |
+
| steak_8 | 18 | 18 | 14 | no |
|
| 20 |
+
| SoapDispenser010 | 5 | 5 | 27 | no |
|
| 21 |
+
| teapot_6 | 5 | 5 | 27 | no |
|
| 22 |
+
| teapot_7 | 5 | 5 | 27 | no |
|
| 23 |
+
| donut_5 | 3 | 3 | 29 | no |
|
| 24 |
+
| Jar023 | 2 | 2 | 30 | no |
|
| 25 |
+
| SyrupBottle005 | 2 | 2 | 30 | no |
|
| 26 |
+
| **total** | | **72** | **184** | |
|
| 27 |
+
|
| 28 |
+
Only MeasuringCup009 can be filled from its own category. This is a property of the
|
| 29 |
+
dataset, not of the masks: across all 2,517 episodes there are only 5 teapot episodes,
|
| 30 |
+
3 donut, 2 jar, 2 syrup_bottle. Generating the other 1,830 missing masks would not move
|
| 31 |
+
these numbers — only 58 of the 1,888 were same-category for these eight targets (55
|
| 32 |
+
measuring_cup, 3 steak), and those are already done.
|
| 33 |
+
|
| 34 |
+
## Run
|
| 35 |
+
|
| 36 |
+
```bash
|
| 37 |
+
./run_aug32.sh <work_dir> <out_dir> <gpu> [num_wan_workers]
|
| 38 |
+
|
| 39 |
+
# one GPU
|
| 40 |
+
./run_aug32.sh /scratch/vace_aug32 \
|
| 41 |
+
/home/nvidia/minha/pose6daug/baseline/mimicgen/output_aug_256 2
|
| 42 |
+
|
| 43 |
+
# four GPUs: split-wan writes four manifests, one worker each
|
| 44 |
+
./run_aug32.sh /scratch/vace_aug32 /path/out 2 4 # then move the workers onto 4 GPUs by hand
|
| 45 |
+
```
|
| 46 |
+
|
| 47 |
+
Every stage is idempotent — re-running after an interruption resumes rather than redoing.
|
| 48 |
+
|
| 49 |
+
| stage | what it does | cost |
|
| 50 |
+
|---|---|---|
|
| 51 |
+
| 1 `build` | pick sources, same-category first | seconds |
|
| 52 |
+
| 2 `prep` | render the target mesh's reference views, cut frames, per-view manifests | ~4 s / assignment |
|
| 53 |
+
| 3 `split` | chop into 81-frame WAN jobs | seconds |
|
| 54 |
+
| 4 `wan` | the diffusion pass | **everything** |
|
| 55 |
+
| 5 `stitch` | glue each camera's chunks back together | seconds |
|
| 56 |
+
| 6 `collect` | copy into the output layout | seconds |
|
| 57 |
+
|
| 58 |
+
**Runtime.** 256 assignments × 3 cameras ≈ **3,084 WAN jobs**. At the measured 51.4 s/job
|
| 59 |
+
that is **~44 h on one GPU**, ~11 h on four. Levers, in order of how much they buy and how
|
| 60 |
+
little they cost:
|
| 61 |
+
|
| 62 |
+
- `--offload_model False` is already passed by the script. The upstream default is `True`
|
| 63 |
+
whenever `world_size == 1`, which moves the DiT to CPU and back around every job: 81.8 s
|
| 64 |
+
vs 51.4 s measured. Never drop this flag.
|
| 65 |
+
- `--sample_steps 25` (upstream default 50) → 32.1 s/job, ~27 h on one GPU. Quality
|
| 66 |
+
trade-off; look at the plates before committing.
|
| 67 |
+
- Dropping the two exo cameras cuts the job count to a third. At 256×256 the target-object
|
| 68 |
+
mask is ~0.2 % of an exo frame and VACE tends to erase the object instead of replacing it;
|
| 69 |
+
only `robot0_eye_in_hand` (~9 %) reliably gets a real swap. If the exo views are not
|
| 70 |
+
needed, edit `CAMS` in `../vace_object_insert_batch.py`.
|
| 71 |
+
|
| 72 |
+
## Output
|
| 73 |
+
|
| 74 |
+
```
|
| 75 |
+
<out_dir>/<task>-ep<N>_<object the episode had>_<object it becomes>/
|
| 76 |
+
robot0_agentview_left/robot0_agentview_left.mp4
|
| 77 |
+
robot0_agentview_right/robot0_agentview_right.mp4
|
| 78 |
+
robot0_eye_in_hand/robot0_eye_in_hand.mp4
|
| 79 |
+
.done
|
| 80 |
+
```
|
| 81 |
+
|
| 82 |
+
e.g. `c2cab-ep000151_donut_donut_5` (same-category) and
|
| 83 |
+
`snk2c-ep000004_egg_teapot_7` (random fill). Task tags are `c2cab`, `c2sto`, `drw2c`,
|
| 84 |
+
`snk2c`, `tst2c`; the hyphen keeps the name to three underscore-separated parts, matching
|
| 85 |
+
MimicGen's `ep098_apple_teapot_6`. `.done` carries the real task, episode, source mesh,
|
| 86 |
+
target, language instruction, and which fill it came from.
|
| 87 |
+
|
| 88 |
+
## Files
|
| 89 |
+
|
| 90 |
+
| file | |
|
| 91 |
+
|---|---|
|
| 92 |
+
| `build_aug32_assignments.py` | source selection — the only new logic |
|
| 93 |
+
| `collect_outputs.py` | VACE plates → the output layout |
|
| 94 |
+
| `run_aug32.sh` | all six stages |
|
| 95 |
+
| `SETUP.md` | **start here on a new machine** — the setup path, in order |
|
| 96 |
+
| `preflight.sh` | checks environments, weights, datasets, GT-mask coverage, GPU |
|
| 97 |
+
| `make_gt_masks.py` | re-renders the missing GT masks from the simulator |
|
| 98 |
+
| `vace_wan_inference_batch.patch` | the upstream diff, for porting onto a fresh clone |
|
| 99 |
+
| `vendor/render_mesh_views_side.py` | mesh reference-view renderer, vendored from the original machine |
|
| 100 |
+
|
| 101 |
+
Stages 2–5 are `../vace_object_insert_batch.py` and `../VACE/vace/vace_wan_inference.py`
|
| 102 |
+
unmodified; this directory adds selection and collection around them.
|
| 103 |
+
|
| 104 |
+
## Prerequisites
|
| 105 |
+
|
| 106 |
+
Run `./preflight.sh` — it checks every item below and prints GT-mask coverage and GPU
|
| 107 |
+
state. Nothing here needs to be installed or cloned; it all already exists on this box and
|
| 108 |
+
is referenced by absolute path, the same convention the other baselines use.
|
| 109 |
+
|
| 110 |
+
### Environments (do NOT create a venv)
|
| 111 |
+
|
| 112 |
+
| what | python |
|
| 113 |
+
|---|---|
|
| 114 |
+
| batch driver (`vace_object_insert_batch.py`, the builder, the collector) | `/lp-dev/jonghoon/robocasa_calib/envs/robocasa/bin/python` (3.10, mujoco + robocasa) |
|
| 115 |
+
| WAN inference | `/lp-dev/jonghoon/VACE/venv/bin/python` (3.12, torch 2.5.1+cu124, flash_attn 2.8.3) |
|
| 116 |
+
| mesh reference-view rendering (called by `prep-worker`) | `/lp-dev/jonghoon/isaac-gr00t/venv/bin/python` — override with `VACE_RENDER_PYTHON`; it only needs `trimesh`, `pyrender`, `numpy`, `pillow` (measured: 4.12.1 / 0.1.45 / 1.26.4 / 12.1.1), so any env with those will do |
|
| 117 |
+
|
| 118 |
+
`prep-worker` renders the target mesh's reference views by importing `render_views` from
|
| 119 |
+
`render_mesh_views_side.py`, which lived outside this tree on the original machine. A copy
|
| 120 |
+
ships in **`vendor/render_mesh_views_side.py`**; point at it with
|
| 121 |
+
|
| 122 |
+
```bash
|
| 123 |
+
export VACE_RENDER_SCRIPT_DIR=$PWD/aug32/vendor
|
| 124 |
+
export VACE_RENDER_PYTHON=/path/to/a/python/with/trimesh+pyrender
|
| 125 |
+
```
|
| 126 |
+
|
| 127 |
+
### VACE code and weights (already cloned — do NOT clone again)
|
| 128 |
+
|
| 129 |
+
`../VACE/` is a fork of `https://github.com/ali-vilab/VACE.git` carrying the batch flags
|
| 130 |
+
(`--batch_manifest / --skip_existing / --batch_continue_on_error`) this pipeline depends on;
|
| 131 |
+
a fresh upstream clone will not have them. Weights come in through
|
| 132 |
+
`../VACE/models -> /lp-dev/jonghoon/VACE-home-models` (31 GB: `Wan2.1-VACE-1.3B`,
|
| 133 |
+
`VACE-Annotators`). ~27 GB of GPU memory is needed with `--offload_model False`.
|
| 134 |
+
|
| 135 |
+
### Datasets
|
| 136 |
+
|
| 137 |
+
| path | what |
|
| 138 |
+
|---|---|
|
| 139 |
+
| `/home/nvidia/jonghoon/robocasa_full/pickplace_target_human/` | source episodes — `videos/chunk-000/observation.images.<camera>/episode_NNNNNN.mp4` and `extras/episode_NNNNNN/ep_meta.json` (symlink to `/lp-dev/...`) |
|
| 140 |
+
| `/home/nvidia/jonghoon/robocasa_full/aug_materials/robocasa/<task>/ep<N>/` | `mask_gt_target_object_<camera>.mp4` — the GT masks |
|
| 141 |
+
| `/lp-dev/jonghoon/robocasa_calib/repos/robocasa/robocasa/models/assets/objects/{objaverse,lightwheel}/` | target meshes |
|
| 142 |
+
|
| 143 |
+
GT-mask coverage, measured: `PickPlaceCounterToCabinet` 502, `CounterToStove` 50,
|
| 144 |
+
`DrawerToCounter` 43, `SinkToCounter` 34, `ToasterToCounter` 0 — 629 usable episodes, and
|
| 145 |
+
the builder draws only from those.
|
| 146 |
+
|
| 147 |
+
## Porting to another machine
|
| 148 |
+
|
| 149 |
+
Do **not** clone `ali-vilab/VACE` fresh — upstream has no `--batch_manifest`, which every
|
| 150 |
+
stage past `split-wan` depends on.
|
| 151 |
+
|
| 152 |
+
**Copy the whole `baseline/vace/` directory**, not just `aug32/`. Nothing outside it is
|
| 153 |
+
needed except the 31 GB of WAN weights. To rebuild from a clone instead:
|
| 154 |
+
`git clone https://github.com/ali-vilab/VACE.git && git checkout 48eb44f`, then
|
| 155 |
+
`git apply vace_wan_inference_batch.patch`, and copy `build_vace_wan_batch_manifest.py`,
|
| 156 |
+
`vace_inference_full.py` and this `aug32/` directory alongside it.
|
| 157 |
+
|
| 158 |
+
### Unpack the data
|
| 159 |
+
|
| 160 |
+
```bash
|
| 161 |
+
cd baseline/vace && mkdir -p data && cd data
|
| 162 |
+
tar -xf ../datasets_pack/source_episodes.tar # 2.7 GB -> pickplace_target_human/
|
| 163 |
+
tar -xzf ../datasets_pack/aug_materials_ext.tar.gz # 121 MB -> aug_materials_ext/
|
| 164 |
+
tar -xzf ../datasets_pack/target_meshes.tar.gz # 14 MB -> objaverse/, lightwheel/
|
| 165 |
+
cd ..
|
| 166 |
+
```
|
| 167 |
+
|
| 168 |
+
### Point everything at it
|
| 169 |
+
|
| 170 |
+
Every path is an environment variable with the original machine's value as the default, so
|
| 171 |
+
this is the whole configuration:
|
| 172 |
+
|
| 173 |
+
| variable | default | set it to |
|
| 174 |
+
|---|---|---|
|
| 175 |
+
| `VACE_ROBOCASA_ROOT` | original machine | `$PWD/data/pickplace_target_human` |
|
| 176 |
+
| `VACE_AUG_MATERIALS` | original machine | `$PWD/data/aug_materials_ext` |
|
| 177 |
+
| `VACE_ASSET_ROOTS` | original machine | `$PWD/data/lightwheel:$PWD/data/objaverse` |
|
| 178 |
+
| `VACE_PY_ROBOCASA` | original machine | a python with `robocasa`, `mujoco`, `trimesh`, `pyrender` |
|
| 179 |
+
| `VACE_PY_WAN` | original machine | a python with torch 2.5.1+cu124 and `flash_attn` |
|
| 180 |
+
| `VACE_RENDER_SCRIPT_DIR` | **`aug32/vendor/`** | — already inside the tree |
|
| 181 |
+
| `VACE_RENDER_PYTHON` | **`VACE_PY_ROBOCASA`** | — only if that env lacks trimesh/pyrender |
|
| 182 |
+
|
| 183 |
+
```bash
|
| 184 |
+
export VACE_ROBOCASA_ROOT=$PWD/data/pickplace_target_human
|
| 185 |
+
export VACE_AUG_MATERIALS=$PWD/data/aug_materials_ext
|
| 186 |
+
export VACE_ASSET_ROOTS=$PWD/data/lightwheel:$PWD/data/objaverse
|
| 187 |
+
export VACE_PY_ROBOCASA=/path/to/robocasa/python
|
| 188 |
+
export VACE_PY_WAN=/path/to/wan/python
|
| 189 |
+
./aug32/preflight.sh # says MISS for anything still wrong
|
| 190 |
+
```
|
| 191 |
+
|
| 192 |
+
The mesh reference-view renderer (`render_mesh_views_side.py`, which `prep-worker` imports)
|
| 193 |
+
is vendored at `aug32/vendor/` and defaults to the driver's own interpreter, so it needs no
|
| 194 |
+
separate environment — `trimesh` and `pyrender` are its only requirements beyond numpy/PIL.
|
| 195 |
+
|
| 196 |
+
### Still needed from elsewhere
|
| 197 |
+
|
| 198 |
+
The 31 GB of WAN weights (`Wan2.1-VACE-1.3B`, `VACE-Annotators`) — link them in at
|
| 199 |
+
`VACE/models`. And, only if you want to re-run `make_gt_masks.py` there, the meshes of
|
| 200 |
+
objects other than the eight targets, since that rebuilds each episode's own scene.
|
| 201 |
+
|
| 202 |
+
### Gotchas
|
| 203 |
+
|
| 204 |
+
- An episode without GT masks makes `prep-worker` raise `FileNotFoundError`. The builder
|
| 205 |
+
filters on this, so it cannot happen through `run_aug32.sh` — but it will if you hand-edit
|
| 206 |
+
`assignments.jsonl`.
|
| 207 |
+
- The WAN stage must run with `vace/VACE` as the working directory (imports and `--ckpt_dir`
|
| 208 |
+
are relative to it). `run_aug32.sh` does the `cd`.
|
| 209 |
+
- `--save_file` in `vace_wan_inference.py` is written relative to the *current* directory,
|
| 210 |
+
not `--save_dir`. The batch path does not use it; a one-off call should pass an absolute
|
| 211 |
+
path.
|
augment/vace/SETUP.md
ADDED
|
@@ -0,0 +1,138 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# First hour on a new machine
|
| 2 |
+
|
| 3 |
+
Do these in order. Everything else — how the pipeline works, why the sources are picked the
|
| 4 |
+
way they are, runtime numbers — is in `README.md`; this file is only the setup path.
|
| 5 |
+
|
| 6 |
+
Assumes you have this repo and nothing else.
|
| 7 |
+
|
| 8 |
+
---
|
| 9 |
+
|
| 10 |
+
## 0. What you must bring yourself
|
| 11 |
+
|
| 12 |
+
| | size | why it is not here |
|
| 13 |
+
|---|---|---|
|
| 14 |
+
| **WAN weights** — `Wan2.1-VACE-1.3B`, `VACE-Annotators` | 31 GB | too large to ship; get them from the VACE release or copy from the original machine |
|
| 15 |
+
| **GPU** with ≥ 27 GB free | | `--offload_model False` keeps the DiT resident |
|
| 16 |
+
|
| 17 |
+
Everything else is in this repo.
|
| 18 |
+
|
| 19 |
+
---
|
| 20 |
+
|
| 21 |
+
## 1. Unpack the data (~3 GB, 2 min)
|
| 22 |
+
|
| 23 |
+
```bash
|
| 24 |
+
cd <repo>/ # the directory holding aug32/, VACE/, datasets_pack/
|
| 25 |
+
mkdir -p data && cd data
|
| 26 |
+
tar -xf ../datasets_pack/source_episodes.tar # 2.7 GB -> pickplace_target_human/
|
| 27 |
+
tar -xzf ../datasets_pack/aug_materials_ext.tar.gz # 121 MB -> aug_materials_ext/
|
| 28 |
+
tar -xzf ../datasets_pack/target_meshes.tar.gz # 14 MB -> objaverse/ lightwheel/
|
| 29 |
+
cd ..
|
| 30 |
+
```
|
| 31 |
+
|
| 32 |
+
Check:
|
| 33 |
+
|
| 34 |
+
```
|
| 35 |
+
data/pickplace_target_human/<5 tasks>/{videos,extras}
|
| 36 |
+
data/aug_materials_ext/<4 tasks>/ep<N>/mask_gt_target_object_<camera>.mp4
|
| 37 |
+
data/{objaverse,lightwheel}/<category>/<model>/model.xml
|
| 38 |
+
```
|
| 39 |
+
|
| 40 |
+
---
|
| 41 |
+
|
| 42 |
+
## 2. Link the WAN weights
|
| 43 |
+
|
| 44 |
+
```bash
|
| 45 |
+
ln -s /path/to/VACE-home-models VACE/models
|
| 46 |
+
ls VACE/models # must show Wan2.1-VACE-1.3B and VACE-Annotators
|
| 47 |
+
```
|
| 48 |
+
|
| 49 |
+
---
|
| 50 |
+
|
| 51 |
+
## 3. Build two Python environments
|
| 52 |
+
|
| 53 |
+
**A — batch driver.** Needs `robocasa`, `robosuite`, `mujoco`, `trimesh`, `pyrender`,
|
| 54 |
+
`opencv-python`, `numpy`. This one also renders the target meshes' reference views, so
|
| 55 |
+
`trimesh` + `pyrender` are not optional. Versions that work: mujoco 3.3.1, robosuite 1.5.2,
|
| 56 |
+
robocasa 1.0.0, trimesh 4.12.1, pyrender 0.1.45, numpy 1.26.4, Python 3.10.
|
| 57 |
+
|
| 58 |
+
**B — WAN inference.** Python 3.12, `torch 2.5.1+cu124`, `flash_attn 2.8.3`, plus
|
| 59 |
+
`VACE/requirements.txt`. `wan` must be importable (it is a site-package in the original
|
| 60 |
+
environment, installed from the VACE repo).
|
| 61 |
+
|
| 62 |
+
```bash
|
| 63 |
+
export VACE_PY_ROBOCASA=/path/to/env-A/bin/python
|
| 64 |
+
export VACE_PY_WAN=/path/to/env-B/bin/python
|
| 65 |
+
```
|
| 66 |
+
|
| 67 |
+
---
|
| 68 |
+
|
| 69 |
+
## 4. Export the data paths
|
| 70 |
+
|
| 71 |
+
```bash
|
| 72 |
+
cd <repo>
|
| 73 |
+
export VACE_ROBOCASA_ROOT=$PWD/data/pickplace_target_human
|
| 74 |
+
export VACE_AUG_MATERIALS=$PWD/data/aug_materials_ext
|
| 75 |
+
export VACE_ASSET_ROOTS=$PWD/data/lightwheel:$PWD/data/objaverse
|
| 76 |
+
```
|
| 77 |
+
|
| 78 |
+
Leave `VACE_RENDER_SCRIPT_DIR` and `VACE_RENDER_PYTHON` alone — they default to
|
| 79 |
+
`aug32/vendor/` and to `VACE_PY_ROBOCASA`, both already correct.
|
| 80 |
+
|
| 81 |
+
Put all five exports in a file you can `source`; every later step needs them.
|
| 82 |
+
|
| 83 |
+
---
|
| 84 |
+
|
| 85 |
+
## 5. Preflight
|
| 86 |
+
|
| 87 |
+
```bash
|
| 88 |
+
./aug32/preflight.sh
|
| 89 |
+
```
|
| 90 |
+
|
| 91 |
+
Every line must read `OK`. It checks both interpreters, the vendored renderer and its
|
| 92 |
+
imports, the VACE fork and weights, the three data roots, per-task GT-mask coverage, and
|
| 93 |
+
free GPU memory. A `MISS` names the exact path that is wrong.
|
| 94 |
+
|
| 95 |
+
---
|
| 96 |
+
|
| 97 |
+
## 6. Smoke test before the real batch
|
| 98 |
+
|
| 99 |
+
One assignment end to end, ~10 min on one GPU, to catch a broken path before a multi-hour
|
| 100 |
+
run:
|
| 101 |
+
|
| 102 |
+
```bash
|
| 103 |
+
mkdir -p /tmp/vace_smoke
|
| 104 |
+
$VACE_PY_ROBOCASA aug32/build_aug32_assignments.py \
|
| 105 |
+
--out-root /tmp/vace_smoke --per-mesh 1 --targets teapot/teapot_6
|
| 106 |
+
head -1 /tmp/vace_smoke/assignments.jsonl > /tmp/vace_smoke/one.jsonl
|
| 107 |
+
mv /tmp/vace_smoke/one.jsonl /tmp/vace_smoke/assignments.jsonl
|
| 108 |
+
./aug32/run_aug32.sh /tmp/vace_smoke /tmp/vace_smoke_out 0
|
| 109 |
+
```
|
| 110 |
+
|
| 111 |
+
Expect `/tmp/vace_smoke_out/<task>-ep<N>_<src>_teapot_6/` with three camera folders, each
|
| 112 |
+
holding one mp4.
|
| 113 |
+
|
| 114 |
+
---
|
| 115 |
+
|
| 116 |
+
## 7. The real run
|
| 117 |
+
|
| 118 |
+
```bash
|
| 119 |
+
./aug32/run_aug32.sh /scratch/vace_aug32 /scratch/output_aug_256 0
|
| 120 |
+
```
|
| 121 |
+
|
| 122 |
+
8 targets × 32 episodes × 3 cameras ≈ 3,084 WAN jobs, **~44 h on one GPU** at the measured
|
| 123 |
+
51.4 s/job. Split it: `run_aug32.sh ... <gpu> 4` writes four worker manifests, and the four
|
| 124 |
+
WAN processes can be moved onto four GPUs (~11 h). Every stage is idempotent, so an
|
| 125 |
+
interrupted run resumes where it stopped.
|
| 126 |
+
|
| 127 |
+
---
|
| 128 |
+
|
| 129 |
+
## Common failures
|
| 130 |
+
|
| 131 |
+
| symptom | cause |
|
| 132 |
+
|---|---|
|
| 133 |
+
| `FileNotFoundError: Missing GT target-object mask video` | `VACE_AUG_MATERIALS` not exported, or pointing above `aug_materials_ext/` |
|
| 134 |
+
| `ModuleNotFoundError: No module named 'trimesh'` during `prep-worker` | env A lacks trimesh/pyrender — install them, or point `VACE_RENDER_PYTHON` at an env that has them |
|
| 135 |
+
| `unrecognized arguments: --batch_manifest` | `VACE/` was replaced with a fresh upstream clone; apply `aug32/vace_wan_inference_batch.patch` |
|
| 136 |
+
| WAN job ~82 s instead of ~51 s | `--offload_model False` got dropped; it is in `run_aug32.sh`, do not remove it |
|
| 137 |
+
| exo cameras come out with the object *erased* rather than swapped | expected at 256×256 — the target mask is ~0.2 % of an exo frame. Only `robot0_eye_in_hand` (~9 %) reliably swaps |
|
| 138 |
+
| assignments have fewer same-category sources than expected | the dataset only holds 5 teapot / 3 donut / 2 jar / 2 syrup_bottle episodes. Only MeasuringCup009 can fill 32 from its own category |
|
augment/vace/build_aug32_assignments.py
ADDED
|
@@ -0,0 +1,148 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
"""Build VACE assignments: same-category sources first, random episodes for the remainder.
|
| 3 |
+
|
| 4 |
+
For each target mesh we want N augmentations. Episodes whose own object is the same
|
| 5 |
+
category as the target are used first (donut_8, donut_9 ... -> donut_5), because those are
|
| 6 |
+
the swaps VACE was designed for. Only once that pool is exhausted does the rest get filled
|
| 7 |
+
from randomly drawn episodes of any category (apple, milk, ... -> donut_5).
|
| 8 |
+
|
| 9 |
+
Only episodes with a complete set of GT target-object mask videos are eligible --
|
| 10 |
+
`prep-worker` fails on any other. No source episode is used twice for the same target.
|
| 11 |
+
|
| 12 |
+
The stock `vace_object_insert_batch.py build` cannot express this: it draws a target's
|
| 13 |
+
sources from its own category only, and round-robins with wraparound so a short pool comes
|
| 14 |
+
back as duplicates. This writes the same `assignments.jsonl` the stock builder writes, so
|
| 15 |
+
every later stage (`prep-worker`, `split-wan`, the WAN batch, `stitch-segments`) is the
|
| 16 |
+
unmodified upstream code.
|
| 17 |
+
|
| 18 |
+
python build_aug32_assignments.py --out-root DIR [--per-mesh 32] [--seed 0]
|
| 19 |
+
"""
|
| 20 |
+
from __future__ import annotations
|
| 21 |
+
|
| 22 |
+
import argparse
|
| 23 |
+
import importlib.util
|
| 24 |
+
import json
|
| 25 |
+
import random
|
| 26 |
+
import sys
|
| 27 |
+
from collections import Counter
|
| 28 |
+
from pathlib import Path
|
| 29 |
+
|
| 30 |
+
HERE = Path(__file__).resolve().parent
|
| 31 |
+
spec = importlib.util.spec_from_file_location(
|
| 32 |
+
"vace_batch", HERE.parent / "vace_object_insert_batch.py")
|
| 33 |
+
mg = importlib.util.module_from_spec(spec)
|
| 34 |
+
spec.loader.exec_module(mg)
|
| 35 |
+
|
| 36 |
+
TARGETS = [
|
| 37 |
+
"donut/donut_5",
|
| 38 |
+
"jar/Jar023",
|
| 39 |
+
"measuring_cup/MeasuringCup009",
|
| 40 |
+
"soap_dispenser/SoapDispenser010",
|
| 41 |
+
"steak/steak_8",
|
| 42 |
+
"syrup_bottle/SyrupBottle005",
|
| 43 |
+
"teapot/teapot_6",
|
| 44 |
+
"teapot/teapot_7",
|
| 45 |
+
]
|
| 46 |
+
|
| 47 |
+
# short, underscore-free task tags so the folder name stays three underscore-separated parts
|
| 48 |
+
TASK_TAG = {
|
| 49 |
+
"PickPlaceCounterToCabinet": "c2cab",
|
| 50 |
+
"PickPlaceCounterToStove": "c2sto",
|
| 51 |
+
"PickPlaceDrawerToCounter": "drw2c",
|
| 52 |
+
"PickPlaceSinkToCounter": "snk2c",
|
| 53 |
+
"PickPlaceToasterToCounter": "tst2c",
|
| 54 |
+
}
|
| 55 |
+
|
| 56 |
+
|
| 57 |
+
class AnyCategory:
|
| 58 |
+
"""Container that accepts every category, so the stock scan keeps all episodes."""
|
| 59 |
+
|
| 60 |
+
def __contains__(self, value):
|
| 61 |
+
return value is not None
|
| 62 |
+
|
| 63 |
+
|
| 64 |
+
def has_gt_masks(row: dict) -> bool:
|
| 65 |
+
ep_dir = mg.AUG_MATERIALS_ROOT / row["task"] / f"ep{int(row['episode']):06d}"
|
| 66 |
+
return all(
|
| 67 |
+
(ep_dir / f"mask_gt_target_object_{cam}.mp4").exists()
|
| 68 |
+
or (ep_dir / "encoded_videos" / f"mask_gt_target_object_{cam}.mp4").exists()
|
| 69 |
+
for cam in mg.CAMS
|
| 70 |
+
)
|
| 71 |
+
|
| 72 |
+
|
| 73 |
+
def run_name_for(src: dict, target_model: str) -> str:
|
| 74 |
+
"""<task tag>-ep<N>_<object that episode had>_<object it becomes>, as MimicGen names them."""
|
| 75 |
+
tag = TASK_TAG.get(src["task"], mg.sanitize(src["task"]))
|
| 76 |
+
return (f"{tag}-ep{int(src['episode']):06d}"
|
| 77 |
+
f"_{mg.sanitize(src['source_category'])}"
|
| 78 |
+
f"_{mg.sanitize(target_model)}")
|
| 79 |
+
|
| 80 |
+
|
| 81 |
+
def main() -> int:
|
| 82 |
+
ap = argparse.ArgumentParser()
|
| 83 |
+
ap.add_argument("--out-root", type=Path, required=True)
|
| 84 |
+
ap.add_argument("--per-mesh", type=int, default=32)
|
| 85 |
+
ap.add_argument("--targets", nargs="*", default=TARGETS)
|
| 86 |
+
ap.add_argument("--dataset-root", type=Path, default=mg.ROBOCASA_ROOT)
|
| 87 |
+
ap.add_argument("--seed", type=int, default=0)
|
| 88 |
+
a = ap.parse_args()
|
| 89 |
+
|
| 90 |
+
by_cat = mg.scan_sources(a.dataset_root, AnyCategory())
|
| 91 |
+
pool = [r for rows in by_cat.values() for r in rows if has_gt_masks(r)]
|
| 92 |
+
pool.sort(key=lambda r: (r["task"], r["episode"]))
|
| 93 |
+
print(f"GT 마스크 보유 소스: {len(pool)}개", file=sys.stderr, flush=True)
|
| 94 |
+
|
| 95 |
+
rng = random.Random(a.seed)
|
| 96 |
+
a.out_root.mkdir(parents=True, exist_ok=True)
|
| 97 |
+
assignments, summary = [], {}
|
| 98 |
+
idx = 0
|
| 99 |
+
for target_mesh in a.targets:
|
| 100 |
+
category, model = target_mesh.split("/", 1)
|
| 101 |
+
same = [r for r in pool
|
| 102 |
+
if r["source_category"] == category and r["source_model"] != model]
|
| 103 |
+
rng.shuffle(same)
|
| 104 |
+
chosen = same[: a.per_mesh]
|
| 105 |
+
|
| 106 |
+
if len(chosen) < a.per_mesh:
|
| 107 |
+
used = {(r["task"], r["episode"]) for r in chosen}
|
| 108 |
+
rest = [r for r in pool
|
| 109 |
+
if (r["task"], r["episode"]) not in used and r["source_model"] != model]
|
| 110 |
+
rng.shuffle(rest)
|
| 111 |
+
chosen += rest[: a.per_mesh - len(chosen)]
|
| 112 |
+
|
| 113 |
+
n_same = sum(1 for r in chosen if r["source_category"] == category)
|
| 114 |
+
summary[target_mesh] = dict(
|
| 115 |
+
requested=a.per_mesh, assigned=len(chosen),
|
| 116 |
+
same_category=n_same, random_fill=len(chosen) - n_same,
|
| 117 |
+
same_category_pool=len(same),
|
| 118 |
+
by_task=dict(sorted(Counter(r["task"] for r in chosen).items())),
|
| 119 |
+
)
|
| 120 |
+
for sample_idx, src in enumerate(chosen):
|
| 121 |
+
name = run_name_for(src, model)
|
| 122 |
+
assignments.append(dict(
|
| 123 |
+
assignment_idx=idx, sample_idx_for_target=sample_idx,
|
| 124 |
+
target_mesh=target_mesh, target_category=category, target_model=model,
|
| 125 |
+
source=src, case_dir=str(a.out_root / name), run_name=name,
|
| 126 |
+
# `provenance` is the key build_vace_objwise_gr00t_dataset.py filters on:
|
| 127 |
+
# it drops "random_fill" rows unless --include-random-fill is passed.
|
| 128 |
+
provenance=("existing" if src["source_category"] == category else "random_fill"),
|
| 129 |
+
fill=("same_category" if src["source_category"] == category else "random"),
|
| 130 |
+
))
|
| 131 |
+
idx += 1
|
| 132 |
+
|
| 133 |
+
with (a.out_root / "assignments.jsonl").open("w", encoding="utf-8") as f:
|
| 134 |
+
for row in assignments:
|
| 135 |
+
f.write(json.dumps(row, ensure_ascii=False) + "\n")
|
| 136 |
+
(a.out_root / "assignment_summary.json").write_text(
|
| 137 |
+
json.dumps(summary, indent=2, ensure_ascii=False))
|
| 138 |
+
|
| 139 |
+
print(f"\n{'타깃':34s} {'배정':>5s} {'동일카테고리':>10s} {'랜덤보충':>8s} {'동일풀':>7s}")
|
| 140 |
+
for t, s in summary.items():
|
| 141 |
+
print(f" {t:32s} {s['assigned']:5d} {s['same_category']:10d} "
|
| 142 |
+
f"{s['random_fill']:8d} {s['same_category_pool']:7d}")
|
| 143 |
+
print(f"\n배정 {len(assignments)}건 -> {a.out_root / 'assignments.jsonl'}")
|
| 144 |
+
return 0
|
| 145 |
+
|
| 146 |
+
|
| 147 |
+
if __name__ == "__main__":
|
| 148 |
+
raise SystemExit(main())
|
augment/vace/collect_outputs.py
ADDED
|
@@ -0,0 +1,78 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
"""Collect finished VACE plates into the MimicGen output layout.
|
| 3 |
+
|
| 4 |
+
VACE writes one stitched video per camera under
|
| 5 |
+
|
| 6 |
+
<case_dir>/work_<camera>/inference_gtmask_<tag>/out_video.mp4
|
| 7 |
+
|
| 8 |
+
which this copies to the same shape the MimicGen augmentation run uses:
|
| 9 |
+
|
| 10 |
+
<out>/<task>-ep<N>_<object the episode had>_<object it becomes>/<camera>/<camera>.mp4
|
| 11 |
+
|
| 12 |
+
plus a `.done` recording where the plate came from. An assignment whose three cameras are
|
| 13 |
+
not all present is skipped and listed at the end, so a partial WAN batch can be collected
|
| 14 |
+
and re-collected later without producing half-populated folders.
|
| 15 |
+
|
| 16 |
+
python collect_outputs.py --assignments DIR/assignments.jsonl --out DIR_OUT
|
| 17 |
+
"""
|
| 18 |
+
from __future__ import annotations
|
| 19 |
+
|
| 20 |
+
import argparse
|
| 21 |
+
import glob
|
| 22 |
+
import json
|
| 23 |
+
import os
|
| 24 |
+
import shutil
|
| 25 |
+
from pathlib import Path
|
| 26 |
+
|
| 27 |
+
CAMERAS = ["robot0_agentview_left", "robot0_agentview_right", "robot0_eye_in_hand"]
|
| 28 |
+
|
| 29 |
+
|
| 30 |
+
def plate_for(case_dir: str, camera: str) -> str | None:
|
| 31 |
+
hits = sorted(glob.glob(os.path.join(case_dir, f"work_{camera}", "*", "out_video.mp4")))
|
| 32 |
+
return hits[0] if hits else None
|
| 33 |
+
|
| 34 |
+
|
| 35 |
+
def main() -> int:
|
| 36 |
+
ap = argparse.ArgumentParser()
|
| 37 |
+
ap.add_argument("--assignments", type=Path, required=True)
|
| 38 |
+
ap.add_argument("--out", type=Path, required=True)
|
| 39 |
+
ap.add_argument("--force", action="store_true")
|
| 40 |
+
a = ap.parse_args()
|
| 41 |
+
|
| 42 |
+
rows = [json.loads(l) for l in a.assignments.open()]
|
| 43 |
+
a.out.mkdir(parents=True, exist_ok=True)
|
| 44 |
+
done, missing = 0, []
|
| 45 |
+
for r in rows:
|
| 46 |
+
plates = {c: plate_for(r["case_dir"], c) for c in CAMERAS}
|
| 47 |
+
if any(v is None for v in plates.values()):
|
| 48 |
+
missing.append((r["run_name"], [c for c, v in plates.items() if v is None]))
|
| 49 |
+
continue
|
| 50 |
+
ep_dir = a.out / r["run_name"]
|
| 51 |
+
if (ep_dir / ".done").exists() and not a.force:
|
| 52 |
+
done += 1
|
| 53 |
+
continue
|
| 54 |
+
for c, src in plates.items():
|
| 55 |
+
d = ep_dir / c
|
| 56 |
+
d.mkdir(parents=True, exist_ok=True)
|
| 57 |
+
shutil.copy2(src, d / f"{c}.mp4")
|
| 58 |
+
src = r["source"]
|
| 59 |
+
(ep_dir / ".done").write_text(json.dumps(dict(
|
| 60 |
+
method="vace", fill=r.get("fill"),
|
| 61 |
+
source_task=src["task"], source_episode=src["episode"],
|
| 62 |
+
source_object=src["source_mesh"], target_object=r["target_mesh"],
|
| 63 |
+
lang=src.get("lang"), case_dir=r["case_dir"]), indent=1))
|
| 64 |
+
done += 1
|
| 65 |
+
print(f" {r['run_name']}", flush=True)
|
| 66 |
+
|
| 67 |
+
print(f"\n수집 {done} / {len(rows)}")
|
| 68 |
+
if missing:
|
| 69 |
+
print(f"미완 {len(missing)}건 (카메라 누락):")
|
| 70 |
+
for name, cams in missing[:20]:
|
| 71 |
+
print(f" {name}: {', '.join(c.replace('robot0_', '') for c in cams)}")
|
| 72 |
+
if len(missing) > 20:
|
| 73 |
+
print(f" ... 외 {len(missing) - 20}건")
|
| 74 |
+
return 0
|
| 75 |
+
|
| 76 |
+
|
| 77 |
+
if __name__ == "__main__":
|
| 78 |
+
raise SystemExit(main())
|
augment/vace/make_gt_masks.py
ADDED
|
@@ -0,0 +1,179 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
"""Render the missing GT target-object mask videos straight from the simulator.
|
| 3 |
+
|
| 4 |
+
`prep-worker` needs `aug_materials/robocasa/<task>/ep<N>/mask_gt_target_object_<cam>.mp4`
|
| 5 |
+
for every source episode, and those only exist for 629 of the 2,517 episodes — which is
|
| 6 |
+
what caps the VACE batch. They are reproducible: every episode ships `model.xml.gz` and
|
| 7 |
+
`states.npz`, so restoring each frame and rendering a segmentation pass gives the same
|
| 8 |
+
mask. This writes them in the same shape as the delivered ones (256x256, 20 fps, h264,
|
| 9 |
+
white where the target object is).
|
| 10 |
+
|
| 11 |
+
The target is every geom whose name starts with `obj_`, matching
|
| 12 |
+
`vace_object_insert_batch.render_target_mask` (which only ever rendered frame 0, as a
|
| 13 |
+
visibility prefilter — this renders the whole episode).
|
| 14 |
+
|
| 15 |
+
# what is missing
|
| 16 |
+
python make_gt_masks.py --list
|
| 17 |
+
|
| 18 |
+
# render it, 8 workers, writing beside the delivered masks
|
| 19 |
+
python make_gt_masks.py --workers 8
|
| 20 |
+
python make_gt_masks.py --workers 8 --out-root /somewhere/else # or to a new tree
|
| 21 |
+
"""
|
| 22 |
+
from __future__ import annotations
|
| 23 |
+
|
| 24 |
+
import argparse
|
| 25 |
+
import json
|
| 26 |
+
import os
|
| 27 |
+
import sys
|
| 28 |
+
from pathlib import Path
|
| 29 |
+
|
| 30 |
+
os.environ.setdefault("MUJOCO_GL", "egl")
|
| 31 |
+
os.environ.setdefault("PYOPENGL_PLATFORM", "egl")
|
| 32 |
+
|
| 33 |
+
import importlib.util
|
| 34 |
+
from concurrent.futures import ProcessPoolExecutor, as_completed
|
| 35 |
+
|
| 36 |
+
HERE = Path(__file__).resolve().parent
|
| 37 |
+
spec = importlib.util.spec_from_file_location(
|
| 38 |
+
"vace_batch", HERE.parent / "vace_object_insert_batch.py")
|
| 39 |
+
mg = importlib.util.module_from_spec(spec)
|
| 40 |
+
spec.loader.exec_module(mg)
|
| 41 |
+
|
| 42 |
+
CAMS = mg.CAMS
|
| 43 |
+
FPS = 20
|
| 44 |
+
RES = 256
|
| 45 |
+
|
| 46 |
+
|
| 47 |
+
def mask_path(root: Path, task: str, episode: int, cam: str) -> Path:
|
| 48 |
+
return root / task / f"ep{episode:06d}" / f"mask_gt_target_object_{cam}.mp4"
|
| 49 |
+
|
| 50 |
+
|
| 51 |
+
def episodes(dataset_root: Path):
|
| 52 |
+
for task_dir in sorted(p for p in dataset_root.iterdir() if p.is_dir()):
|
| 53 |
+
extras = task_dir / "extras"
|
| 54 |
+
if not extras.exists():
|
| 55 |
+
continue
|
| 56 |
+
for ep_dir in sorted(extras.glob("episode_*")):
|
| 57 |
+
try:
|
| 58 |
+
yield task_dir.name, int(ep_dir.name.split("_")[-1]), ep_dir
|
| 59 |
+
except ValueError:
|
| 60 |
+
continue
|
| 61 |
+
|
| 62 |
+
|
| 63 |
+
def missing(dataset_root: Path, out_root: Path, read_root: Path, categories=None):
|
| 64 |
+
out = []
|
| 65 |
+
for task, ep, ep_dir in episodes(dataset_root):
|
| 66 |
+
if categories:
|
| 67 |
+
info = mg.target_obj_info(ep_dir)
|
| 68 |
+
if not info or info["category"] not in categories:
|
| 69 |
+
continue
|
| 70 |
+
need = [c for c in CAMS
|
| 71 |
+
if not mask_path(read_root, task, ep, c).exists()
|
| 72 |
+
and not (read_root / task / f"ep{ep:06d}" / "encoded_videos"
|
| 73 |
+
/ f"mask_gt_target_object_{c}.mp4").exists()
|
| 74 |
+
and not mask_path(out_root, task, ep, c).exists()]
|
| 75 |
+
if need:
|
| 76 |
+
out.append((task, ep, str(ep_dir), need))
|
| 77 |
+
return out
|
| 78 |
+
|
| 79 |
+
|
| 80 |
+
def render_one(job):
|
| 81 |
+
task, ep, ep_dir, cams, out_root = job
|
| 82 |
+
import cv2
|
| 83 |
+
import mujoco
|
| 84 |
+
import numpy as np
|
| 85 |
+
|
| 86 |
+
try:
|
| 87 |
+
sim, states = mg.sim_from_episode(Path(ep_dir))
|
| 88 |
+
model, data = sim.model._model, sim.data._data
|
| 89 |
+
target = {gid for gid in range(model.ngeom)
|
| 90 |
+
if (mujoco.mj_id2name(model, mujoco.mjtObj.mjOBJ_GEOM, gid) or "").startswith("obj_")}
|
| 91 |
+
if not target:
|
| 92 |
+
return dict(task=task, episode=ep, status="failed", error="no obj_ geoms")
|
| 93 |
+
opt = mujoco.MjvOption()
|
| 94 |
+
opt.sitegroup[:] = 0
|
| 95 |
+
renderer = mujoco.Renderer(model, height=RES, width=RES)
|
| 96 |
+
renderer.enable_segmentation_rendering()
|
| 97 |
+
written = {}
|
| 98 |
+
try:
|
| 99 |
+
for cam in cams:
|
| 100 |
+
cid = mujoco.mj_name2id(model, mujoco.mjtObj.mjOBJ_CAMERA, cam)
|
| 101 |
+
if cid < 0:
|
| 102 |
+
return dict(task=task, episode=ep, status="failed", error=f"no camera {cam}")
|
| 103 |
+
dst = mask_path(Path(out_root), task, ep, cam)
|
| 104 |
+
dst.parent.mkdir(parents=True, exist_ok=True)
|
| 105 |
+
vw = cv2.VideoWriter(str(dst), cv2.VideoWriter_fourcc(*"mp4v"),
|
| 106 |
+
FPS, (RES, RES), True)
|
| 107 |
+
area = 0
|
| 108 |
+
for t in range(len(states)):
|
| 109 |
+
sim.set_state_from_flattened(states[t])
|
| 110 |
+
sim.forward()
|
| 111 |
+
renderer.update_scene(data, camera=cid, scene_option=opt)
|
| 112 |
+
seg = mg.render_segmentation_safe(renderer)
|
| 113 |
+
m = (np.isin(seg[..., 0], list(target)).astype(np.uint8) * 255)
|
| 114 |
+
area = max(area, int((m > 127).sum()))
|
| 115 |
+
vw.write(np.repeat(m[:, :, None], 3, axis=2))
|
| 116 |
+
vw.release()
|
| 117 |
+
written[cam] = dict(frames=len(states), max_area=area)
|
| 118 |
+
finally:
|
| 119 |
+
renderer.close()
|
| 120 |
+
return dict(task=task, episode=ep, status="ok", frames=len(states), cams=written)
|
| 121 |
+
except Exception as e: # one bad episode must not take the batch down
|
| 122 |
+
return dict(task=task, episode=ep, status="failed", error=f"{type(e).__name__}: {e}")
|
| 123 |
+
|
| 124 |
+
|
| 125 |
+
def main() -> int:
|
| 126 |
+
ap = argparse.ArgumentParser()
|
| 127 |
+
ap.add_argument("--dataset-root", type=Path, default=mg.ROBOCASA_ROOT)
|
| 128 |
+
ap.add_argument("--read-root", type=Path, default=mg.AUG_MATERIALS_ROOT,
|
| 129 |
+
help="where the delivered masks live (checked, never written)")
|
| 130 |
+
ap.add_argument("--out-root", type=Path, default=mg.AUG_MATERIALS_ROOT,
|
| 131 |
+
help="where new masks are written; default = beside the delivered ones")
|
| 132 |
+
ap.add_argument("--workers", type=int, default=8)
|
| 133 |
+
ap.add_argument("--limit", type=int, default=None)
|
| 134 |
+
ap.add_argument("--list", action="store_true", help="only report what is missing")
|
| 135 |
+
ap.add_argument("--categories", nargs="*", default=None,
|
| 136 |
+
help="only episodes whose own object is one of these categories -- the "
|
| 137 |
+
"others can never serve as a same-category source")
|
| 138 |
+
ap.add_argument("--log", type=Path, default=None)
|
| 139 |
+
a = ap.parse_args()
|
| 140 |
+
|
| 141 |
+
todo = missing(a.dataset_root, a.out_root, a.read_root,
|
| 142 |
+
set(a.categories) if a.categories else None)
|
| 143 |
+
from collections import Counter
|
| 144 |
+
per = Counter(t for t, _, _, _ in todo)
|
| 145 |
+
total = sum(1 for _ in episodes(a.dataset_root))
|
| 146 |
+
print(f"에피소드 {total}개 중 마스크 미보유 {len(todo)}개")
|
| 147 |
+
for t, n in sorted(per.items()):
|
| 148 |
+
print(f" {t:30s} {n:5d}")
|
| 149 |
+
if a.list:
|
| 150 |
+
return 0
|
| 151 |
+
if a.limit:
|
| 152 |
+
todo = todo[: a.limit]
|
| 153 |
+
if not todo:
|
| 154 |
+
return 0
|
| 155 |
+
|
| 156 |
+
jobs = [(t, e, d, cams, str(a.out_root)) for t, e, d, cams in todo]
|
| 157 |
+
log = open(a.log, "w") if a.log else None
|
| 158 |
+
ok = fail = 0
|
| 159 |
+
with ProcessPoolExecutor(max_workers=a.workers) as ex:
|
| 160 |
+
futs = {ex.submit(render_one, j): j for j in jobs}
|
| 161 |
+
for i, fut in enumerate(as_completed(futs), 1):
|
| 162 |
+
r = fut.result()
|
| 163 |
+
ok += r["status"] == "ok"
|
| 164 |
+
fail += r["status"] == "failed"
|
| 165 |
+
if log:
|
| 166 |
+
log.write(json.dumps(r) + "\n")
|
| 167 |
+
log.flush()
|
| 168 |
+
if r["status"] == "failed":
|
| 169 |
+
print(f" FAIL {r['task']}/ep{r['episode']:06d}: {r['error'][:90]}", flush=True)
|
| 170 |
+
if i % 25 == 0 or i == len(jobs):
|
| 171 |
+
print(f" {i}/{len(jobs)} ok={ok} fail={fail}", flush=True)
|
| 172 |
+
if log:
|
| 173 |
+
log.close()
|
| 174 |
+
print(f"\n완료: ok {ok}, failed {fail}")
|
| 175 |
+
return 0 if fail == 0 else 1
|
| 176 |
+
|
| 177 |
+
|
| 178 |
+
if __name__ == "__main__":
|
| 179 |
+
raise SystemExit(main())
|
augment/vace/preflight.sh
ADDED
|
@@ -0,0 +1,60 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env bash
|
| 2 |
+
# Check everything run_aug32.sh needs before committing a multi-hour batch to a GPU.
|
| 3 |
+
# Exits non-zero if anything required is missing.
|
| 4 |
+
set -uo pipefail
|
| 5 |
+
|
| 6 |
+
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
| 7 |
+
VACE="$(cd "$HERE/.." && pwd)"
|
| 8 |
+
fail=0
|
| 9 |
+
|
| 10 |
+
ok() { printf ' \033[32mOK \033[0m %s\n' "$1"; }
|
| 11 |
+
bad() { printf ' \033[31mMISS\033[0m %s\n' "$1"; fail=1; }
|
| 12 |
+
have() { [ -e "$1" ] && ok "$2" || bad "$2 ($1)"; }
|
| 13 |
+
runs() { [ -x "$1" ] && ok "$2 ($("$1" -V 2>&1))" || bad "$2 ($1)"; }
|
| 14 |
+
|
| 15 |
+
echo "환경 (가상환경 — 새로 만들 필요 없음)"
|
| 16 |
+
runs "${VACE_PY_ROBOCASA:-/lp-dev/jonghoon/robocasa_calib/envs/robocasa/bin/python}" "batch driver (robocasa, mujoco)"
|
| 17 |
+
runs "${VACE_PY_WAN:-/lp-dev/jonghoon/VACE/venv/bin/python}" "WAN inference (torch+flash_attn)"
|
| 18 |
+
have "${VACE_RENDER_SCRIPT_DIR:-$HERE/vendor}/render_mesh_views_side.py" "render_mesh_views_side.py (vendored)"
|
| 19 |
+
"${VACE_RENDER_PYTHON:-${VACE_PY_ROBOCASA:-/lp-dev/jonghoon/robocasa_calib/envs/robocasa/bin/python}}" \
|
| 20 |
+
-c "import trimesh,pyrender" 2>/dev/null && ok "trimesh + pyrender (mesh view rendering)" \
|
| 21 |
+
|| bad "trimesh + pyrender missing from the render python"
|
| 22 |
+
|
| 23 |
+
echo
|
| 24 |
+
echo "VACE 코드와 가중치"
|
| 25 |
+
have "$VACE/VACE/vace/vace_wan_inference.py" "VACE fork (ali-vilab/VACE + batch flags)"
|
| 26 |
+
have "$VACE/VACE/models/Wan2.1-VACE-1.3B" "Wan2.1-VACE-1.3B weights (via models symlink)"
|
| 27 |
+
have "$VACE/VACE/models/VACE-Annotators" "VACE-Annotators"
|
| 28 |
+
have "$VACE/vace_object_insert_batch.py" "batch driver"
|
| 29 |
+
|
| 30 |
+
echo
|
| 31 |
+
echo "데이터셋"
|
| 32 |
+
have "${VACE_ROBOCASA_ROOT:-/home/nvidia/jonghoon/robocasa_full/pickplace_target_human}" "소스 에피소드 (영상 + extras)"
|
| 33 |
+
have "${VACE_AUG_MATERIALS:-/home/nvidia/jonghoon/robocasa_full/aug_materials/robocasa}" "GT target-object 마스크"
|
| 34 |
+
for r in $(echo "${VACE_ASSET_ROOTS:-/lp-dev/jonghoon/robocasa_calib/repos/robocasa/robocasa/models/assets/objects/lightwheel:/lp-dev/jonghoon/robocasa_calib/repos/robocasa/robocasa/models/assets/objects/objaverse}" | tr ":" " "); do
|
| 35 |
+
have "$r" "메쉬 $(basename "$r")"
|
| 36 |
+
done
|
| 37 |
+
|
| 38 |
+
echo
|
| 39 |
+
echo "GT 마스크 커버리지 (prep-worker 가 요구 — 없으면 그 에피소드는 실패)"
|
| 40 |
+
A="${VACE_AUG_MATERIALS:-/home/nvidia/jonghoon/robocasa_full/aug_materials/robocasa}"
|
| 41 |
+
for t in PickPlaceCounterToCabinet PickPlaceCounterToStove PickPlaceDrawerToCounter \
|
| 42 |
+
PickPlaceSinkToCounter PickPlaceToasterToCounter; do
|
| 43 |
+
n=0
|
| 44 |
+
for d in "$A/$t"/ep*; do
|
| 45 |
+
[ -f "$d/mask_gt_target_object_robot0_agentview_left.mp4" ] &&
|
| 46 |
+
[ -f "$d/mask_gt_target_object_robot0_agentview_right.mp4" ] &&
|
| 47 |
+
[ -f "$d/mask_gt_target_object_robot0_eye_in_hand.mp4" ] && n=$((n+1))
|
| 48 |
+
done 2>/dev/null
|
| 49 |
+
printf ' %-30s %4d episodes\n' "$t" "$n"
|
| 50 |
+
done
|
| 51 |
+
|
| 52 |
+
echo
|
| 53 |
+
echo "GPU"
|
| 54 |
+
nvidia-smi --query-gpu=index,memory.used,memory.total,utilization.gpu --format=csv,noheader 2>/dev/null |
|
| 55 |
+
awk -F', ' '{printf " gpu %s %s / %s util %s\n",$1,$2,$3,$4}'
|
| 56 |
+
echo " 필요: 약 27 GB (Wan2.1-VACE-1.3B, --offload_model False 기준)"
|
| 57 |
+
|
| 58 |
+
echo
|
| 59 |
+
[ $fail -eq 0 ] && echo "모두 준비됨." || echo "누락 항목 있음 — 위 MISS 확인."
|
| 60 |
+
exit $fail
|
augment/vace/run_aug32.sh
ADDED
|
@@ -0,0 +1,67 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env bash
|
| 2 |
+
# VACE augmentation: 8 target objects x 32 episodes, three cameras each.
|
| 3 |
+
#
|
| 4 |
+
# ./run_aug32.sh <work_dir> <out_dir> <gpu> [num_wan_workers]
|
| 5 |
+
#
|
| 6 |
+
# Stages, in order. Each is idempotent, so re-running after an interruption resumes.
|
| 7 |
+
# 1 build same-category sources first, random episodes for the remainder
|
| 8 |
+
# 2 prep render the target mesh's reference views, cut frames, build per-view manifests
|
| 9 |
+
# 3 split chop into 81-frame WAN jobs
|
| 10 |
+
# 4 wan the diffusion pass -- the only slow stage
|
| 11 |
+
# 5 stitch glue each camera's chunks back into one video
|
| 12 |
+
# 6 collect copy into <out_dir>/<ep>_<src obj>_<target>/<camera>/<camera>.mp4
|
| 13 |
+
set -euo pipefail
|
| 14 |
+
|
| 15 |
+
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
| 16 |
+
VACE="$(cd "$HERE/.." && pwd)"
|
| 17 |
+
|
| 18 |
+
WORK="${1:?usage: run_aug32.sh <work_dir> <out_dir> <gpu> [num_wan_workers]}"
|
| 19 |
+
OUT="${2:?usage: run_aug32.sh <work_dir> <out_dir> <gpu> [num_wan_workers]}"
|
| 20 |
+
GPU="${3:?usage: run_aug32.sh <work_dir> <out_dir> <gpu> [num_wan_workers]}"
|
| 21 |
+
WORKERS="${4:-1}"
|
| 22 |
+
|
| 23 |
+
RC="${VACE_PY_ROBOCASA:-/lp-dev/jonghoon/robocasa_calib/envs/robocasa/bin/python}"
|
| 24 |
+
VP="${VACE_PY_WAN:-/lp-dev/jonghoon/VACE/venv/bin/python}"
|
| 25 |
+
export MUJOCO_GL=egl PYOPENGL_PLATFORM=egl
|
| 26 |
+
|
| 27 |
+
# Data paths come from the environment; unset means "the original machine's path".
|
| 28 |
+
# On any other box unpack ../datasets_pack and `export` these first -- see README "Porting":
|
| 29 |
+
# VACE_ROBOCASA_ROOT VACE_AUG_MATERIALS VACE_ASSET_ROOTS
|
| 30 |
+
# VACE_PY_ROBOCASA VACE_PY_WAN (interpreters)
|
| 31 |
+
|
| 32 |
+
mkdir -p "$WORK"
|
| 33 |
+
|
| 34 |
+
echo "== 1/6 build =================================================="
|
| 35 |
+
"$RC" "$HERE/build_aug32_assignments.py" --out-root "$WORK" --per-mesh 32
|
| 36 |
+
|
| 37 |
+
echo "== 2/6 prep (gpu $GPU) ========================================"
|
| 38 |
+
cd "$VACE"
|
| 39 |
+
"$RC" vace_object_insert_batch.py prep-worker \
|
| 40 |
+
--assignments "$WORK/assignments.jsonl" --out-root "$WORK" \
|
| 41 |
+
--worker-id 0 --num-workers 1 --gpu "$GPU" | tee "$WORK/prep.log" | \
|
| 42 |
+
grep -c '"status": "ok"' | xargs -I{} echo " prepared views: {}"
|
| 43 |
+
|
| 44 |
+
echo "== 3/6 split =================================================="
|
| 45 |
+
"$RC" vace_object_insert_batch.py split-wan \
|
| 46 |
+
--assignments "$WORK/assignments.jsonl" --out-root "$WORK" --num-workers "$WORKERS"
|
| 47 |
+
|
| 48 |
+
echo "== 4/6 wan (gpu $GPU) ========================================="
|
| 49 |
+
# --offload_model False keeps the DiT resident; the default True costs ~30 s per job
|
| 50 |
+
cd "$VACE/VACE"
|
| 51 |
+
for w in $(seq -f "%02g" 0 $((WORKERS - 1))); do
|
| 52 |
+
CUDA_VISIBLE_DEVICES="$GPU" "$VP" vace/vace_wan_inference.py \
|
| 53 |
+
--model_name vace-1.3B --ckpt_dir models/Wan2.1-VACE-1.3B \
|
| 54 |
+
--batch_manifest "$WORK/batch_manifests/wan_worker_$w.jsonl" \
|
| 55 |
+
--offload_model False --skip_existing true --batch_continue_on_error true \
|
| 56 |
+
> "$WORK/wan_$w.log" 2>&1 &
|
| 57 |
+
done
|
| 58 |
+
wait
|
| 59 |
+
|
| 60 |
+
echo "== 5/6 stitch ================================================="
|
| 61 |
+
cd "$VACE"
|
| 62 |
+
"$RC" vace_object_insert_batch.py stitch-segments \
|
| 63 |
+
--assignments "$WORK/assignments.jsonl" --out-root "$WORK"
|
| 64 |
+
|
| 65 |
+
echo "== 6/6 collect ================================================"
|
| 66 |
+
"$RC" "$HERE/collect_outputs.py" --assignments "$WORK/assignments.jsonl" --out "$OUT"
|
| 67 |
+
echo "done -> $OUT"
|
dataset/build_vace_objwise_gr00t_dataset.py
ADDED
|
@@ -0,0 +1,509 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
"""Build a GR00T/LeRobot dataset from object-wise VACE augmentation videos.
|
| 3 |
+
|
| 4 |
+
The VACE output stores one assignment directory per source trajectory and target
|
| 5 |
+
mesh, with three final view videos under work_<view>/<inference_dir>/out_video.mp4.
|
| 6 |
+
This builder keeps the original source state/action parquet rows, rewrites only
|
| 7 |
+
episode/global indices and task ids, links the VACE videos into the LeRobot video
|
| 8 |
+
layout, and creates one side-by-side three_view.mp4 preview per augmented episode.
|
| 9 |
+
"""
|
| 10 |
+
|
| 11 |
+
from __future__ import annotations
|
| 12 |
+
|
| 13 |
+
import argparse
|
| 14 |
+
import json
|
| 15 |
+
import math
|
| 16 |
+
import os
|
| 17 |
+
import shutil
|
| 18 |
+
import statistics
|
| 19 |
+
import subprocess
|
| 20 |
+
import time
|
| 21 |
+
from collections import Counter, defaultdict
|
| 22 |
+
from datetime import datetime, timezone
|
| 23 |
+
from pathlib import Path
|
| 24 |
+
from typing import Any
|
| 25 |
+
|
| 26 |
+
import numpy as np
|
| 27 |
+
import pandas as pd
|
| 28 |
+
|
| 29 |
+
|
| 30 |
+
TASKS = (
|
| 31 |
+
"PickPlaceCounterToCabinet",
|
| 32 |
+
"PickPlaceCounterToStove",
|
| 33 |
+
"PickPlaceDrawerToCounter",
|
| 34 |
+
"PickPlaceSinkToCounter",
|
| 35 |
+
"PickPlaceToasterToCounter",
|
| 36 |
+
)
|
| 37 |
+
|
| 38 |
+
VIEW_KEYS = (
|
| 39 |
+
"robot0_agentview_left",
|
| 40 |
+
"robot0_agentview_right",
|
| 41 |
+
"robot0_eye_in_hand",
|
| 42 |
+
)
|
| 43 |
+
|
| 44 |
+
VIDEO_FEATURES = {
|
| 45 |
+
"robot0_agentview_left": "observation.images.robot0_agentview_left",
|
| 46 |
+
"robot0_agentview_right": "observation.images.robot0_agentview_right",
|
| 47 |
+
"robot0_eye_in_hand": "observation.images.robot0_eye_in_hand",
|
| 48 |
+
}
|
| 49 |
+
|
| 50 |
+
TASK_COLUMNS = (
|
| 51 |
+
"task_index",
|
| 52 |
+
"annotation.human.task_description",
|
| 53 |
+
"annotation.human.task_name",
|
| 54 |
+
)
|
| 55 |
+
|
| 56 |
+
|
| 57 |
+
def parse_args() -> argparse.Namespace:
|
| 58 |
+
parser = argparse.ArgumentParser(description=__doc__)
|
| 59 |
+
parser.add_argument(
|
| 60 |
+
"--vace-root",
|
| 61 |
+
type=Path,
|
| 62 |
+
default=Path("/home/nvidia/jonghoon/robocasa_full/augmented_video/groot1.5_objwise_vace"),
|
| 63 |
+
)
|
| 64 |
+
parser.add_argument(
|
| 65 |
+
"--source-root",
|
| 66 |
+
type=Path,
|
| 67 |
+
default=Path("/home/nvidia/jonghoon/robocasa_full/pickplace_target_human"),
|
| 68 |
+
)
|
| 69 |
+
parser.add_argument(
|
| 70 |
+
"--canonical-meta-source",
|
| 71 |
+
type=Path,
|
| 72 |
+
default=Path("/home/nvidia/jonghoon/robocasa_full/gr00t_views/PickPlaceCounterToCabinet_all_502eps"),
|
| 73 |
+
help="Dataset whose GR00T-compatible info/modality/view_manifest contract is copied.",
|
| 74 |
+
)
|
| 75 |
+
parser.add_argument(
|
| 76 |
+
"--manifest",
|
| 77 |
+
type=Path,
|
| 78 |
+
action="append",
|
| 79 |
+
default=None,
|
| 80 |
+
help="Assignment JSONL. Defaults to assignments_target16.jsonl and assignments_teapot6_extra.jsonl.",
|
| 81 |
+
)
|
| 82 |
+
parser.add_argument("--output", type=Path, required=True)
|
| 83 |
+
parser.add_argument(
|
| 84 |
+
"--include-random-fill",
|
| 85 |
+
action="store_true",
|
| 86 |
+
help="Include assignments whose provenance is random_fill. Default keeps only clean same-category assignments.",
|
| 87 |
+
)
|
| 88 |
+
parser.add_argument(
|
| 89 |
+
"--copy-videos",
|
| 90 |
+
action="store_true",
|
| 91 |
+
help="Copy per-view VACE videos instead of symlinking them.",
|
| 92 |
+
)
|
| 93 |
+
parser.add_argument("--force", action="store_true")
|
| 94 |
+
parser.add_argument(
|
| 95 |
+
"--skip-three-view",
|
| 96 |
+
action="store_true",
|
| 97 |
+
help="Do not render extras/episode_xxxxxx/three_view.mp4 previews.",
|
| 98 |
+
)
|
| 99 |
+
return parser.parse_args()
|
| 100 |
+
|
| 101 |
+
|
| 102 |
+
def load_json(path: Path) -> dict[str, Any]:
|
| 103 |
+
return json.loads(path.read_text())
|
| 104 |
+
|
| 105 |
+
|
| 106 |
+
def load_jsonl(path: Path) -> list[dict[str, Any]]:
|
| 107 |
+
with path.open() as f:
|
| 108 |
+
return [json.loads(line) for line in f if line.strip()]
|
| 109 |
+
|
| 110 |
+
|
| 111 |
+
def write_jsonl(path: Path, rows: list[dict[str, Any]]) -> None:
|
| 112 |
+
path.parent.mkdir(parents=True, exist_ok=True)
|
| 113 |
+
with path.open("w") as f:
|
| 114 |
+
for row in rows:
|
| 115 |
+
f.write(json.dumps(row, sort_keys=False) + "\n")
|
| 116 |
+
|
| 117 |
+
|
| 118 |
+
def copy_or_link(src: Path, dst: Path, copy: bool = False) -> None:
|
| 119 |
+
if not src.exists():
|
| 120 |
+
raise FileNotFoundError(src)
|
| 121 |
+
dst.parent.mkdir(parents=True, exist_ok=True)
|
| 122 |
+
if dst.exists() or dst.is_symlink():
|
| 123 |
+
if dst.is_dir() and not dst.is_symlink():
|
| 124 |
+
shutil.rmtree(dst)
|
| 125 |
+
else:
|
| 126 |
+
dst.unlink()
|
| 127 |
+
if copy:
|
| 128 |
+
shutil.copy2(src, dst)
|
| 129 |
+
else:
|
| 130 |
+
dst.symlink_to(src.resolve())
|
| 131 |
+
|
| 132 |
+
|
| 133 |
+
def source_episode_path(dataset_root: Path, episode_index: int, chunks_size: int) -> Path:
|
| 134 |
+
chunk = episode_index // chunks_size
|
| 135 |
+
return dataset_root / "data" / f"chunk-{chunk:03d}" / f"episode_{episode_index:06d}.parquet"
|
| 136 |
+
|
| 137 |
+
|
| 138 |
+
def load_source_catalog(source_root: Path) -> dict[str, dict[str, Any]]:
|
| 139 |
+
catalog: dict[str, dict[str, Any]] = {}
|
| 140 |
+
for task in TASKS:
|
| 141 |
+
root = source_root / task
|
| 142 |
+
if not root.exists():
|
| 143 |
+
continue
|
| 144 |
+
meta = root / "meta"
|
| 145 |
+
info = load_json(meta / "info.json")
|
| 146 |
+
catalog[task] = {
|
| 147 |
+
"root": root,
|
| 148 |
+
"meta": meta,
|
| 149 |
+
"info": info,
|
| 150 |
+
"chunks_size": int(info.get("chunks_size", 1000)),
|
| 151 |
+
"episodes": {int(r["episode_index"]): r for r in load_jsonl(meta / "episodes.jsonl")},
|
| 152 |
+
"episode_stats": (
|
| 153 |
+
{int(r["episode_index"]): r for r in load_jsonl(meta / "episodes_stats.jsonl")}
|
| 154 |
+
if (meta / "episodes_stats.jsonl").exists()
|
| 155 |
+
else {}
|
| 156 |
+
),
|
| 157 |
+
"tasks": {int(r["task_index"]): r["task"] for r in load_jsonl(meta / "tasks.jsonl")},
|
| 158 |
+
}
|
| 159 |
+
return catalog
|
| 160 |
+
|
| 161 |
+
|
| 162 |
+
def default_manifests(vace_root: Path) -> list[Path]:
|
| 163 |
+
manifests = [vace_root / "assignments_target16.jsonl"]
|
| 164 |
+
teapot = vace_root / "assignments_teapot6_extra.jsonl"
|
| 165 |
+
if teapot.exists():
|
| 166 |
+
manifests.append(teapot)
|
| 167 |
+
return manifests
|
| 168 |
+
|
| 169 |
+
|
| 170 |
+
def load_assignments(paths: list[Path], include_random_fill: bool) -> list[dict[str, Any]]:
|
| 171 |
+
rows: list[dict[str, Any]] = []
|
| 172 |
+
for path in paths:
|
| 173 |
+
for row in load_jsonl(path):
|
| 174 |
+
row = dict(row)
|
| 175 |
+
row.setdefault("assignment_manifest", str(path))
|
| 176 |
+
row.setdefault("provenance", "existing")
|
| 177 |
+
if row["provenance"] == "random_fill" and not include_random_fill:
|
| 178 |
+
continue
|
| 179 |
+
rows.append(row)
|
| 180 |
+
rows.sort(key=lambda r: (str(r.get("target_mesh", "")), int(r.get("assignment_idx", 10**9)), str(r.get("case_dir", ""))))
|
| 181 |
+
return rows
|
| 182 |
+
|
| 183 |
+
|
| 184 |
+
def final_view_video(case_dir: Path, view: str) -> tuple[Path, int | None, dict[str, Any]]:
|
| 185 |
+
work = case_dir / f"work_{view}"
|
| 186 |
+
manifest_path = work / "manifest.json"
|
| 187 |
+
manifest: dict[str, Any] = {}
|
| 188 |
+
if manifest_path.exists():
|
| 189 |
+
manifest = load_json(manifest_path)
|
| 190 |
+
inference_dir = manifest.get("inference_save_dir")
|
| 191 |
+
if inference_dir:
|
| 192 |
+
out = Path(inference_dir) / "out_video.mp4"
|
| 193 |
+
if out.exists():
|
| 194 |
+
return out, manifest.get("video_info", {}).get("nb_frames"), manifest
|
| 195 |
+
candidates = [p for p in work.glob("*/out_video.mp4") if "/segments/" not in str(p)]
|
| 196 |
+
if not candidates:
|
| 197 |
+
raise FileNotFoundError(f"No final out_video.mp4 for {case_dir} view={view}")
|
| 198 |
+
candidates.sort(key=lambda p: p.stat().st_mtime, reverse=True)
|
| 199 |
+
return candidates[0], manifest.get("video_info", {}).get("nb_frames"), manifest
|
| 200 |
+
|
| 201 |
+
|
| 202 |
+
def create_three_view(left: Path, right: Path, wrist: Path, out: Path) -> None:
|
| 203 |
+
out.parent.mkdir(parents=True, exist_ok=True)
|
| 204 |
+
tmp = out.with_suffix(".tmp.mp4")
|
| 205 |
+
if tmp.exists():
|
| 206 |
+
tmp.unlink()
|
| 207 |
+
cmd = [
|
| 208 |
+
"ffmpeg",
|
| 209 |
+
"-y",
|
| 210 |
+
"-loglevel",
|
| 211 |
+
"error",
|
| 212 |
+
"-i",
|
| 213 |
+
str(left),
|
| 214 |
+
"-i",
|
| 215 |
+
str(right),
|
| 216 |
+
"-i",
|
| 217 |
+
str(wrist),
|
| 218 |
+
"-filter_complex",
|
| 219 |
+
"[0:v]scale=256:256[v0];[1:v]scale=256:256[v1];[2:v]scale=256:256[v2];[v0][v1][v2]hstack=inputs=3[v]",
|
| 220 |
+
"-map",
|
| 221 |
+
"[v]",
|
| 222 |
+
"-an",
|
| 223 |
+
"-c:v",
|
| 224 |
+
"libx264",
|
| 225 |
+
"-preset",
|
| 226 |
+
"veryfast",
|
| 227 |
+
"-crf",
|
| 228 |
+
"18",
|
| 229 |
+
"-pix_fmt",
|
| 230 |
+
"yuv420p",
|
| 231 |
+
"-shortest",
|
| 232 |
+
str(tmp),
|
| 233 |
+
]
|
| 234 |
+
subprocess.run(cmd, check=True)
|
| 235 |
+
tmp.replace(out)
|
| 236 |
+
|
| 237 |
+
|
| 238 |
+
def remap_episode_stats(row: dict[str, Any], new_episode_index: int, global_start: int, length: int) -> dict[str, Any]:
|
| 239 |
+
row = json.loads(json.dumps(row))
|
| 240 |
+
row["episode_index"] = new_episode_index
|
| 241 |
+
stats = row.setdefault("stats", {})
|
| 242 |
+
stats["episode_index"] = {
|
| 243 |
+
"min": [new_episode_index],
|
| 244 |
+
"max": [new_episode_index],
|
| 245 |
+
"mean": [float(new_episode_index)],
|
| 246 |
+
"std": [0.0],
|
| 247 |
+
"count": [length],
|
| 248 |
+
}
|
| 249 |
+
stats["index"] = {
|
| 250 |
+
"min": [global_start],
|
| 251 |
+
"max": [global_start + length - 1],
|
| 252 |
+
"mean": [global_start + (length - 1) / 2.0],
|
| 253 |
+
"std": [statistics.pstdev(range(global_start, global_start + length))],
|
| 254 |
+
"count": [length],
|
| 255 |
+
}
|
| 256 |
+
return row
|
| 257 |
+
|
| 258 |
+
|
| 259 |
+
def column_values_to_matrix(values: pd.Series) -> np.ndarray:
|
| 260 |
+
return np.vstack([np.asarray(v, dtype=np.float64).reshape(1, -1) for v in values])
|
| 261 |
+
|
| 262 |
+
|
| 263 |
+
def calculate_stats(parquet_paths: list[Path]) -> dict[str, dict[str, list[float]]]:
|
| 264 |
+
accum: dict[str, list[np.ndarray]] = defaultdict(list)
|
| 265 |
+
for path in parquet_paths:
|
| 266 |
+
df = pd.read_parquet(path)
|
| 267 |
+
for column in df.columns:
|
| 268 |
+
accum[column].append(column_values_to_matrix(df[column]))
|
| 269 |
+
stats: dict[str, dict[str, list[float]]] = {}
|
| 270 |
+
for column, chunks in accum.items():
|
| 271 |
+
values = np.concatenate(chunks, axis=0)
|
| 272 |
+
stats[column] = {
|
| 273 |
+
"mean": np.mean(values, axis=0).tolist(),
|
| 274 |
+
"std": np.std(values, axis=0).tolist(),
|
| 275 |
+
"min": np.min(values, axis=0).tolist(),
|
| 276 |
+
"max": np.max(values, axis=0).tolist(),
|
| 277 |
+
"q01": np.quantile(values, 0.01, axis=0).tolist(),
|
| 278 |
+
"q99": np.quantile(values, 0.99, axis=0).tolist(),
|
| 279 |
+
}
|
| 280 |
+
return stats
|
| 281 |
+
|
| 282 |
+
|
| 283 |
+
def main() -> None:
|
| 284 |
+
args = parse_args()
|
| 285 |
+
manifests = args.manifest or default_manifests(args.vace_root)
|
| 286 |
+
assignments = load_assignments(manifests, include_random_fill=args.include_random_fill)
|
| 287 |
+
if not assignments:
|
| 288 |
+
raise RuntimeError("No assignments selected")
|
| 289 |
+
|
| 290 |
+
if args.output.exists():
|
| 291 |
+
if not args.force:
|
| 292 |
+
raise FileExistsError(f"{args.output} exists; pass --force")
|
| 293 |
+
backup = args.output.with_name(f"{args.output.name}.bak_{time.strftime('%Y%m%d_%H%M%S')}")
|
| 294 |
+
args.output.rename(backup)
|
| 295 |
+
print(f"Moved existing output to {backup}")
|
| 296 |
+
|
| 297 |
+
source_catalog = load_source_catalog(args.source_root)
|
| 298 |
+
canonical_meta = args.canonical_meta_source / "meta"
|
| 299 |
+
out_meta = args.output / "meta"
|
| 300 |
+
out_data = args.output / "data" / "chunk-000"
|
| 301 |
+
out_videos = args.output / "videos" / "chunk-000"
|
| 302 |
+
out_meta.mkdir(parents=True, exist_ok=True)
|
| 303 |
+
out_data.mkdir(parents=True, exist_ok=True)
|
| 304 |
+
|
| 305 |
+
global_task_id: dict[str, int] = {}
|
| 306 |
+
global_tasks: list[dict[str, Any]] = []
|
| 307 |
+
|
| 308 |
+
def task_id(task_text: str) -> int:
|
| 309 |
+
if task_text not in global_task_id:
|
| 310 |
+
idx = len(global_task_id)
|
| 311 |
+
global_task_id[task_text] = idx
|
| 312 |
+
global_tasks.append({"task_index": idx, "task": task_text})
|
| 313 |
+
return global_task_id[task_text]
|
| 314 |
+
|
| 315 |
+
new_episodes: list[dict[str, Any]] = []
|
| 316 |
+
new_episode_stats: list[dict[str, Any]] = []
|
| 317 |
+
dataset_manifest: list[dict[str, Any]] = []
|
| 318 |
+
parquet_paths: list[Path] = []
|
| 319 |
+
source_counts: Counter[str] = Counter()
|
| 320 |
+
provenance_counts: Counter[str] = Counter()
|
| 321 |
+
global_index = 0
|
| 322 |
+
|
| 323 |
+
for new_ep, row in enumerate(assignments):
|
| 324 |
+
source = row["source"]
|
| 325 |
+
source_task = source["task"]
|
| 326 |
+
source_episode = int(source["episode"])
|
| 327 |
+
if source_task not in source_catalog:
|
| 328 |
+
raise KeyError(f"Unknown source task {source_task} for {row.get('case_dir')}")
|
| 329 |
+
src = source_catalog[source_task]
|
| 330 |
+
src_parquet = source_episode_path(src["root"], source_episode, src["chunks_size"])
|
| 331 |
+
if not src_parquet.exists():
|
| 332 |
+
raise FileNotFoundError(src_parquet)
|
| 333 |
+
source_episode_meta = src["episodes"][source_episode]
|
| 334 |
+
df = pd.read_parquet(src_parquet).copy()
|
| 335 |
+
length = len(df)
|
| 336 |
+
if int(source_episode_meta["length"]) != length:
|
| 337 |
+
raise RuntimeError(f"Source length mismatch for {src_parquet}: parquet={length} meta={source_episode_meta['length']}")
|
| 338 |
+
|
| 339 |
+
case_dir = Path(row["case_dir"])
|
| 340 |
+
view_outputs: dict[str, str] = {}
|
| 341 |
+
view_frame_counts: dict[str, int | None] = {}
|
| 342 |
+
view_manifests: dict[str, Any] = {}
|
| 343 |
+
for view in VIEW_KEYS:
|
| 344 |
+
video, frame_count, view_manifest = final_view_video(case_dir, view)
|
| 345 |
+
if frame_count is not None and int(frame_count) != length:
|
| 346 |
+
raise RuntimeError(f"Frame mismatch for {case_dir} {view}: vace={frame_count} parquet={length}")
|
| 347 |
+
feature = VIDEO_FEATURES[view]
|
| 348 |
+
dst = out_videos / feature / f"episode_{new_ep:06d}.mp4"
|
| 349 |
+
copy_or_link(video, dst, copy=args.copy_videos)
|
| 350 |
+
view_outputs[view] = str(dst)
|
| 351 |
+
view_frame_counts[view] = int(frame_count) if frame_count is not None else None
|
| 352 |
+
view_manifests[view] = view_manifest
|
| 353 |
+
|
| 354 |
+
if "episode_index" in df.columns:
|
| 355 |
+
df["episode_index"] = np.full(length, new_ep, dtype=np.int64)
|
| 356 |
+
if "frame_index" in df.columns:
|
| 357 |
+
df["frame_index"] = np.arange(length, dtype=np.int64)
|
| 358 |
+
if "index" in df.columns:
|
| 359 |
+
df["index"] = np.arange(global_index, global_index + length, dtype=np.int64)
|
| 360 |
+
|
| 361 |
+
for column in TASK_COLUMNS:
|
| 362 |
+
if column in df.columns:
|
| 363 |
+
mapping = {}
|
| 364 |
+
for old_idx in sorted(pd.unique(df[column])):
|
| 365 |
+
old_idx_int = int(old_idx)
|
| 366 |
+
try:
|
| 367 |
+
mapping[old_idx_int] = task_id(src["tasks"][old_idx_int])
|
| 368 |
+
except KeyError as exc:
|
| 369 |
+
raise KeyError(f"{src['root']} has no task index {old_idx_int} for column {column}") from exc
|
| 370 |
+
df[column] = df[column].map(lambda v: mapping[int(v)]).astype("int64")
|
| 371 |
+
|
| 372 |
+
dst_parquet = out_data / f"episode_{new_ep:06d}.parquet"
|
| 373 |
+
df.to_parquet(dst_parquet, index=False)
|
| 374 |
+
parquet_paths.append(dst_parquet)
|
| 375 |
+
|
| 376 |
+
new_episode = {
|
| 377 |
+
"episode_index": new_ep,
|
| 378 |
+
"tasks": source_episode_meta.get("tasks", []),
|
| 379 |
+
"length": length,
|
| 380 |
+
}
|
| 381 |
+
new_episodes.append(new_episode)
|
| 382 |
+
if source_episode in src["episode_stats"]:
|
| 383 |
+
new_episode_stats.append(remap_episode_stats(src["episode_stats"][source_episode], new_ep, global_index, length))
|
| 384 |
+
|
| 385 |
+
extras_dir = args.output / "extras" / f"episode_{new_ep:06d}"
|
| 386 |
+
extras_dir.mkdir(parents=True, exist_ok=True)
|
| 387 |
+
if not args.skip_three_view:
|
| 388 |
+
create_three_view(
|
| 389 |
+
Path(view_outputs["robot0_agentview_left"]),
|
| 390 |
+
Path(view_outputs["robot0_agentview_right"]),
|
| 391 |
+
Path(view_outputs["robot0_eye_in_hand"]),
|
| 392 |
+
extras_dir / "three_view.mp4",
|
| 393 |
+
)
|
| 394 |
+
|
| 395 |
+
compact_row = {
|
| 396 |
+
k: v
|
| 397 |
+
for k, v in row.items()
|
| 398 |
+
if k
|
| 399 |
+
in {
|
| 400 |
+
"assignment_idx",
|
| 401 |
+
"sample_idx_for_target",
|
| 402 |
+
"target_mesh",
|
| 403 |
+
"target_category",
|
| 404 |
+
"target_model",
|
| 405 |
+
"case_dir",
|
| 406 |
+
"run_name",
|
| 407 |
+
"provenance",
|
| 408 |
+
"assignment_manifest",
|
| 409 |
+
}
|
| 410 |
+
}
|
| 411 |
+
dataset_manifest.append(
|
| 412 |
+
{
|
| 413 |
+
"episode_index": new_ep,
|
| 414 |
+
"source_task": source_task,
|
| 415 |
+
"source_episode_index": source_episode,
|
| 416 |
+
"source_length": length,
|
| 417 |
+
"global_index_start": global_index,
|
| 418 |
+
"global_index_end": global_index + length - 1,
|
| 419 |
+
"source": source,
|
| 420 |
+
"assignment": compact_row,
|
| 421 |
+
"vace_case_dir": str(case_dir),
|
| 422 |
+
"vace_view_frame_counts": view_frame_counts,
|
| 423 |
+
"videos": {
|
| 424 |
+
VIDEO_FEATURES[view]: view_outputs[view]
|
| 425 |
+
for view in VIEW_KEYS
|
| 426 |
+
},
|
| 427 |
+
"three_view": str(extras_dir / "three_view.mp4") if not args.skip_three_view else None,
|
| 428 |
+
}
|
| 429 |
+
)
|
| 430 |
+
|
| 431 |
+
source_counts[f"{source_task}/episode_{source_episode:06d}"] += 1
|
| 432 |
+
provenance_counts[str(row.get("provenance", "existing"))] += 1
|
| 433 |
+
global_index += length
|
| 434 |
+
|
| 435 |
+
write_jsonl(out_meta / "episodes.jsonl", new_episodes)
|
| 436 |
+
write_jsonl(out_meta / "tasks.jsonl", global_tasks)
|
| 437 |
+
if new_episode_stats:
|
| 438 |
+
write_jsonl(out_meta / "episodes_stats.jsonl", new_episode_stats)
|
| 439 |
+
|
| 440 |
+
# Copy the GR00T-compatible embodiment/modality contract, not the raw
|
| 441 |
+
# RoboCasa view naming contract.
|
| 442 |
+
for name in ("modality.json", "view_manifest.json"):
|
| 443 |
+
src = canonical_meta / name
|
| 444 |
+
if src.exists():
|
| 445 |
+
shutil.copy2(src, out_meta / name)
|
| 446 |
+
raw_embodiment = args.source_root / "PickPlaceCounterToCabinet" / "meta" / "embodiment.json"
|
| 447 |
+
if raw_embodiment.exists():
|
| 448 |
+
shutil.copy2(raw_embodiment, out_meta / "embodiment.json")
|
| 449 |
+
|
| 450 |
+
info = load_json(canonical_meta / "info.json")
|
| 451 |
+
chunks_size = int(info.get("chunks_size", 1000))
|
| 452 |
+
video_features = [k for k, v in info.get("features", {}).items() if v.get("dtype") == "video"]
|
| 453 |
+
info["total_episodes"] = len(new_episodes)
|
| 454 |
+
info["total_frames"] = global_index
|
| 455 |
+
info["total_videos"] = len(new_episodes) * len(video_features)
|
| 456 |
+
info["total_chunks"] = max(1, math.ceil(len(new_episodes) / chunks_size))
|
| 457 |
+
info["total_tasks"] = len(global_tasks)
|
| 458 |
+
info["splits"] = {"train": f"0:{len(new_episodes)}"}
|
| 459 |
+
info["created_from_vace_objectwise"] = {
|
| 460 |
+
"created_at": datetime.now(timezone.utc).isoformat(),
|
| 461 |
+
"vace_root": str(args.vace_root),
|
| 462 |
+
"manifests": [str(p) for p in manifests],
|
| 463 |
+
"source_root": str(args.source_root),
|
| 464 |
+
"canonical_meta_source": str(args.canonical_meta_source),
|
| 465 |
+
"include_random_fill": bool(args.include_random_fill),
|
| 466 |
+
"episode_count": len(new_episodes),
|
| 467 |
+
"provenance_counts": dict(sorted(provenance_counts.items())),
|
| 468 |
+
}
|
| 469 |
+
(out_meta / "info.json").write_text(json.dumps(info, indent=2) + "\n")
|
| 470 |
+
|
| 471 |
+
stats = calculate_stats(parquet_paths)
|
| 472 |
+
(out_meta / "stats.json").write_text(json.dumps(stats, indent=2) + "\n")
|
| 473 |
+
(out_meta / "relative_stats.json").write_text("{}\n")
|
| 474 |
+
(out_meta / "vace_objwise_dataset_manifest.json").write_text(
|
| 475 |
+
json.dumps(
|
| 476 |
+
{
|
| 477 |
+
"created_at": datetime.now(timezone.utc).isoformat(),
|
| 478 |
+
"output": str(args.output),
|
| 479 |
+
"episode_count": len(new_episodes),
|
| 480 |
+
"total_frames": global_index,
|
| 481 |
+
"include_random_fill": bool(args.include_random_fill),
|
| 482 |
+
"provenance_counts": dict(sorted(provenance_counts.items())),
|
| 483 |
+
"target_mesh_counts": dict(sorted(Counter(r["assignment"]["target_mesh"] for r in dataset_manifest).items())),
|
| 484 |
+
"source_episode_counts": dict(sorted(source_counts.items())),
|
| 485 |
+
"episodes": dataset_manifest,
|
| 486 |
+
},
|
| 487 |
+
indent=2,
|
| 488 |
+
)
|
| 489 |
+
+ "\n"
|
| 490 |
+
)
|
| 491 |
+
|
| 492 |
+
print(
|
| 493 |
+
json.dumps(
|
| 494 |
+
{
|
| 495 |
+
"output": str(args.output),
|
| 496 |
+
"episodes": len(new_episodes),
|
| 497 |
+
"frames": global_index,
|
| 498 |
+
"videos_per_view": len(new_episodes),
|
| 499 |
+
"three_view_videos": 0 if args.skip_three_view else len(new_episodes),
|
| 500 |
+
"provenance_counts": dict(sorted(provenance_counts.items())),
|
| 501 |
+
"total_tasks": len(global_tasks),
|
| 502 |
+
},
|
| 503 |
+
indent=2,
|
| 504 |
+
)
|
| 505 |
+
)
|
| 506 |
+
|
| 507 |
+
|
| 508 |
+
if __name__ == "__main__":
|
| 509 |
+
main()
|
dataset/merge_gr00t_view_datasets.py
ADDED
|
@@ -0,0 +1,339 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
"""Merge multiple GR00T/LeRobot view datasets into one physical dataset.
|
| 3 |
+
|
| 4 |
+
The merge preserves every source row value except dataset-local columns:
|
| 5 |
+
episode_index, frame_index, index, and task-id columns are remapped into the
|
| 6 |
+
merged dataset. Videos are symlinked from the source datasets.
|
| 7 |
+
"""
|
| 8 |
+
|
| 9 |
+
from __future__ import annotations
|
| 10 |
+
|
| 11 |
+
import argparse
|
| 12 |
+
import json
|
| 13 |
+
import math
|
| 14 |
+
import shutil
|
| 15 |
+
import statistics
|
| 16 |
+
import time
|
| 17 |
+
from collections import Counter, defaultdict
|
| 18 |
+
from datetime import datetime, timezone
|
| 19 |
+
from pathlib import Path
|
| 20 |
+
from typing import Any
|
| 21 |
+
|
| 22 |
+
import numpy as np
|
| 23 |
+
import pandas as pd
|
| 24 |
+
|
| 25 |
+
|
| 26 |
+
TASK_COLUMNS = (
|
| 27 |
+
"task_index",
|
| 28 |
+
"annotation.human.task_description",
|
| 29 |
+
"annotation.human.task_name",
|
| 30 |
+
)
|
| 31 |
+
|
| 32 |
+
|
| 33 |
+
def parse_args() -> argparse.Namespace:
|
| 34 |
+
parser = argparse.ArgumentParser(description=__doc__)
|
| 35 |
+
parser.add_argument("--source", type=Path, action="append", required=True)
|
| 36 |
+
parser.add_argument("--output", type=Path, required=True)
|
| 37 |
+
parser.add_argument("--copy-videos", action="store_true")
|
| 38 |
+
parser.add_argument("--force", action="store_true")
|
| 39 |
+
return parser.parse_args()
|
| 40 |
+
|
| 41 |
+
|
| 42 |
+
def load_json(path: Path) -> dict[str, Any]:
|
| 43 |
+
return json.loads(path.read_text())
|
| 44 |
+
|
| 45 |
+
|
| 46 |
+
def load_jsonl(path: Path) -> list[dict[str, Any]]:
|
| 47 |
+
with path.open() as f:
|
| 48 |
+
return [json.loads(line) for line in f if line.strip()]
|
| 49 |
+
|
| 50 |
+
|
| 51 |
+
def write_jsonl(path: Path, rows: list[dict[str, Any]]) -> None:
|
| 52 |
+
path.parent.mkdir(parents=True, exist_ok=True)
|
| 53 |
+
with path.open("w") as f:
|
| 54 |
+
for row in rows:
|
| 55 |
+
f.write(json.dumps(row, sort_keys=False) + "\n")
|
| 56 |
+
|
| 57 |
+
|
| 58 |
+
def copy_or_link(src: Path, dst: Path, copy: bool = False) -> None:
|
| 59 |
+
if not src.exists():
|
| 60 |
+
raise FileNotFoundError(src)
|
| 61 |
+
dst.parent.mkdir(parents=True, exist_ok=True)
|
| 62 |
+
if dst.exists() or dst.is_symlink():
|
| 63 |
+
if dst.is_dir() and not dst.is_symlink():
|
| 64 |
+
shutil.rmtree(dst)
|
| 65 |
+
else:
|
| 66 |
+
dst.unlink()
|
| 67 |
+
if copy:
|
| 68 |
+
shutil.copy2(src, dst)
|
| 69 |
+
else:
|
| 70 |
+
dst.symlink_to(src.resolve())
|
| 71 |
+
|
| 72 |
+
|
| 73 |
+
def episode_path(dataset_root: Path, episode_index: int, chunks_size: int) -> Path:
|
| 74 |
+
chunk = episode_index // chunks_size
|
| 75 |
+
return dataset_root / "data" / f"chunk-{chunk:03d}" / f"episode_{episode_index:06d}.parquet"
|
| 76 |
+
|
| 77 |
+
|
| 78 |
+
def video_features(info: dict[str, Any]) -> list[str]:
|
| 79 |
+
return [key for key, spec in info.get("features", {}).items() if spec.get("dtype") == "video"]
|
| 80 |
+
|
| 81 |
+
|
| 82 |
+
def source_video_path(dataset_root: Path, feature: str, episode_index: int, chunks_size: int) -> Path:
|
| 83 |
+
chunk = episode_index // chunks_size
|
| 84 |
+
return dataset_root / "videos" / f"chunk-{chunk:03d}" / feature / f"episode_{episode_index:06d}.mp4"
|
| 85 |
+
|
| 86 |
+
|
| 87 |
+
def remap_episode_stats(row: dict[str, Any], new_episode_index: int, global_start: int, length: int) -> dict[str, Any]:
|
| 88 |
+
row = json.loads(json.dumps(row))
|
| 89 |
+
row["episode_index"] = new_episode_index
|
| 90 |
+
stats = row.setdefault("stats", {})
|
| 91 |
+
stats["episode_index"] = {
|
| 92 |
+
"min": [new_episode_index],
|
| 93 |
+
"max": [new_episode_index],
|
| 94 |
+
"mean": [float(new_episode_index)],
|
| 95 |
+
"std": [0.0],
|
| 96 |
+
"count": [length],
|
| 97 |
+
}
|
| 98 |
+
stats["index"] = {
|
| 99 |
+
"min": [global_start],
|
| 100 |
+
"max": [global_start + length - 1],
|
| 101 |
+
"mean": [global_start + (length - 1) / 2.0],
|
| 102 |
+
"std": [statistics.pstdev(range(global_start, global_start + length))],
|
| 103 |
+
"count": [length],
|
| 104 |
+
}
|
| 105 |
+
return row
|
| 106 |
+
|
| 107 |
+
|
| 108 |
+
def column_values_to_matrix(values: pd.Series) -> np.ndarray:
|
| 109 |
+
return np.vstack([np.asarray(v, dtype=np.float64).reshape(1, -1) for v in values])
|
| 110 |
+
|
| 111 |
+
|
| 112 |
+
def calculate_stats(parquet_paths: list[Path]) -> dict[str, dict[str, list[float]]]:
|
| 113 |
+
accum: dict[str, list[np.ndarray]] = defaultdict(list)
|
| 114 |
+
for path in parquet_paths:
|
| 115 |
+
df = pd.read_parquet(path)
|
| 116 |
+
for column in df.columns:
|
| 117 |
+
accum[column].append(column_values_to_matrix(df[column]))
|
| 118 |
+
stats: dict[str, dict[str, list[float]]] = {}
|
| 119 |
+
for column, chunks in accum.items():
|
| 120 |
+
values = np.concatenate(chunks, axis=0)
|
| 121 |
+
stats[column] = {
|
| 122 |
+
"mean": np.mean(values, axis=0).tolist(),
|
| 123 |
+
"std": np.std(values, axis=0).tolist(),
|
| 124 |
+
"min": np.min(values, axis=0).tolist(),
|
| 125 |
+
"max": np.max(values, axis=0).tolist(),
|
| 126 |
+
"q01": np.quantile(values, 0.01, axis=0).tolist(),
|
| 127 |
+
"q99": np.quantile(values, 0.99, axis=0).tolist(),
|
| 128 |
+
}
|
| 129 |
+
return stats
|
| 130 |
+
|
| 131 |
+
|
| 132 |
+
def validate_contract(sources: list[Path]) -> tuple[dict[str, Any], list[str]]:
|
| 133 |
+
infos = [load_json(src / "meta" / "info.json") for src in sources]
|
| 134 |
+
base = infos[0]
|
| 135 |
+
base_features = base.get("features", {})
|
| 136 |
+
base_video_features = video_features(base)
|
| 137 |
+
for src, info in zip(sources[1:], infos[1:]):
|
| 138 |
+
if info.get("features", {}) != base_features:
|
| 139 |
+
raise RuntimeError(f"Feature contract differs for {src}")
|
| 140 |
+
if set(video_features(info)) != set(base_video_features):
|
| 141 |
+
raise RuntimeError(f"Video feature set differs for {src}")
|
| 142 |
+
return base, base_video_features
|
| 143 |
+
|
| 144 |
+
|
| 145 |
+
def main() -> None:
|
| 146 |
+
args = parse_args()
|
| 147 |
+
sources = [src.resolve() for src in args.source]
|
| 148 |
+
for src in sources:
|
| 149 |
+
if not (src / "meta" / "info.json").exists():
|
| 150 |
+
raise FileNotFoundError(src / "meta" / "info.json")
|
| 151 |
+
base_info, vid_features = validate_contract(sources)
|
| 152 |
+
|
| 153 |
+
if args.output.exists():
|
| 154 |
+
if not args.force:
|
| 155 |
+
raise FileExistsError(f"{args.output} exists; pass --force")
|
| 156 |
+
backup = args.output.with_name(f"{args.output.name}.bak_{time.strftime('%Y%m%d_%H%M%S')}")
|
| 157 |
+
args.output.rename(backup)
|
| 158 |
+
print(f"Moved existing output to {backup}")
|
| 159 |
+
|
| 160 |
+
out_meta = args.output / "meta"
|
| 161 |
+
out_data_root = args.output / "data"
|
| 162 |
+
out_video_root = args.output / "videos"
|
| 163 |
+
out_meta.mkdir(parents=True, exist_ok=True)
|
| 164 |
+
|
| 165 |
+
for name in ("modality.json", "view_manifest.json", "embodiment.json", "relative_stats.json"):
|
| 166 |
+
src_file = sources[0] / "meta" / name
|
| 167 |
+
if src_file.exists():
|
| 168 |
+
shutil.copy2(src_file, out_meta / name)
|
| 169 |
+
|
| 170 |
+
global_task_id: dict[str, int] = {}
|
| 171 |
+
global_tasks: list[dict[str, Any]] = []
|
| 172 |
+
|
| 173 |
+
def task_id(task_text: str) -> int:
|
| 174 |
+
if task_text not in global_task_id:
|
| 175 |
+
idx = len(global_task_id)
|
| 176 |
+
global_task_id[task_text] = idx
|
| 177 |
+
global_tasks.append({"task_index": idx, "task": task_text})
|
| 178 |
+
return global_task_id[task_text]
|
| 179 |
+
|
| 180 |
+
new_episodes: list[dict[str, Any]] = []
|
| 181 |
+
new_episode_stats: list[dict[str, Any]] = []
|
| 182 |
+
merge_manifest: list[dict[str, Any]] = []
|
| 183 |
+
parquet_paths: list[Path] = []
|
| 184 |
+
source_counts: Counter[str] = Counter()
|
| 185 |
+
source_frame_counts: Counter[str] = Counter()
|
| 186 |
+
global_index = 0
|
| 187 |
+
new_episode_index = 0
|
| 188 |
+
out_chunks_size = int(base_info.get("chunks_size", 1000))
|
| 189 |
+
|
| 190 |
+
for source_idx, src_root in enumerate(sources):
|
| 191 |
+
src_name = src_root.name
|
| 192 |
+
src_info = load_json(src_root / "meta" / "info.json")
|
| 193 |
+
src_chunks_size = int(src_info.get("chunks_size", 1000))
|
| 194 |
+
src_episodes = {int(r["episode_index"]): r for r in load_jsonl(src_root / "meta" / "episodes.jsonl")}
|
| 195 |
+
src_tasks = {int(r["task_index"]): r["task"] for r in load_jsonl(src_root / "meta" / "tasks.jsonl")}
|
| 196 |
+
src_episode_stats = (
|
| 197 |
+
{int(r["episode_index"]): r for r in load_jsonl(src_root / "meta" / "episodes_stats.jsonl")}
|
| 198 |
+
if (src_root / "meta" / "episodes_stats.jsonl").exists()
|
| 199 |
+
else {}
|
| 200 |
+
)
|
| 201 |
+
for src_episode_index in sorted(src_episodes):
|
| 202 |
+
src_episode_meta = src_episodes[src_episode_index]
|
| 203 |
+
src_parquet = episode_path(src_root, src_episode_index, src_chunks_size)
|
| 204 |
+
if not src_parquet.exists():
|
| 205 |
+
raise FileNotFoundError(src_parquet)
|
| 206 |
+
df = pd.read_parquet(src_parquet).copy()
|
| 207 |
+
length = len(df)
|
| 208 |
+
if int(src_episode_meta["length"]) != length:
|
| 209 |
+
raise RuntimeError(f"Length mismatch for {src_parquet}: parquet={length}, meta={src_episode_meta['length']}")
|
| 210 |
+
|
| 211 |
+
if "episode_index" in df.columns:
|
| 212 |
+
df["episode_index"] = np.full(length, new_episode_index, dtype=np.int64)
|
| 213 |
+
if "frame_index" in df.columns:
|
| 214 |
+
df["frame_index"] = np.arange(length, dtype=np.int64)
|
| 215 |
+
if "index" in df.columns:
|
| 216 |
+
df["index"] = np.arange(global_index, global_index + length, dtype=np.int64)
|
| 217 |
+
|
| 218 |
+
task_column_mappings: dict[str, dict[int, int]] = {}
|
| 219 |
+
for column in TASK_COLUMNS:
|
| 220 |
+
if column in df.columns:
|
| 221 |
+
mapping = {}
|
| 222 |
+
for old_idx in sorted(pd.unique(df[column])):
|
| 223 |
+
old_idx_int = int(old_idx)
|
| 224 |
+
mapping[old_idx_int] = task_id(src_tasks[old_idx_int])
|
| 225 |
+
df[column] = df[column].map(lambda v: mapping[int(v)]).astype("int64")
|
| 226 |
+
task_column_mappings[column] = mapping
|
| 227 |
+
|
| 228 |
+
out_parquet = episode_path(args.output, new_episode_index, out_chunks_size)
|
| 229 |
+
out_parquet.parent.mkdir(parents=True, exist_ok=True)
|
| 230 |
+
df.to_parquet(out_parquet, index=False)
|
| 231 |
+
parquet_paths.append(out_parquet)
|
| 232 |
+
|
| 233 |
+
video_links: dict[str, str] = {}
|
| 234 |
+
source_video_links: dict[str, str] = {}
|
| 235 |
+
for feature in vid_features:
|
| 236 |
+
src_video = source_video_path(src_root, feature, src_episode_index, src_chunks_size)
|
| 237 |
+
dst_video = source_video_path(args.output, feature, new_episode_index, out_chunks_size)
|
| 238 |
+
copy_or_link(src_video, dst_video, copy=args.copy_videos)
|
| 239 |
+
video_links[feature] = str(dst_video)
|
| 240 |
+
source_video_links[feature] = str(src_video)
|
| 241 |
+
|
| 242 |
+
extras_src = src_root / "extras" / f"episode_{src_episode_index:06d}"
|
| 243 |
+
extras_dst = args.output / "extras" / f"episode_{new_episode_index:06d}"
|
| 244 |
+
if extras_src.exists():
|
| 245 |
+
extras_dst.mkdir(parents=True, exist_ok=True)
|
| 246 |
+
for child in extras_src.iterdir():
|
| 247 |
+
copy_or_link(child, extras_dst / child.name, copy=args.copy_videos)
|
| 248 |
+
|
| 249 |
+
new_episodes.append(
|
| 250 |
+
{
|
| 251 |
+
"episode_index": new_episode_index,
|
| 252 |
+
"tasks": src_episode_meta.get("tasks", []),
|
| 253 |
+
"length": length,
|
| 254 |
+
}
|
| 255 |
+
)
|
| 256 |
+
if src_episode_index in src_episode_stats:
|
| 257 |
+
new_episode_stats.append(
|
| 258 |
+
remap_episode_stats(src_episode_stats[src_episode_index], new_episode_index, global_index, length)
|
| 259 |
+
)
|
| 260 |
+
|
| 261 |
+
merge_manifest.append(
|
| 262 |
+
{
|
| 263 |
+
"episode_index": new_episode_index,
|
| 264 |
+
"source_dataset_index": source_idx,
|
| 265 |
+
"source_dataset_name": src_name,
|
| 266 |
+
"source_dataset_root": str(src_root),
|
| 267 |
+
"source_episode_index": src_episode_index,
|
| 268 |
+
"source_length": length,
|
| 269 |
+
"source_parquet": str(src_parquet),
|
| 270 |
+
"global_index_start": global_index,
|
| 271 |
+
"global_index_end": global_index + length - 1,
|
| 272 |
+
"task_column_mappings": task_column_mappings,
|
| 273 |
+
"videos": video_links,
|
| 274 |
+
"source_videos": source_video_links,
|
| 275 |
+
}
|
| 276 |
+
)
|
| 277 |
+
source_counts[src_name] += 1
|
| 278 |
+
source_frame_counts[src_name] += length
|
| 279 |
+
global_index += length
|
| 280 |
+
new_episode_index += 1
|
| 281 |
+
|
| 282 |
+
write_jsonl(out_meta / "episodes.jsonl", new_episodes)
|
| 283 |
+
write_jsonl(out_meta / "tasks.jsonl", global_tasks)
|
| 284 |
+
if new_episode_stats:
|
| 285 |
+
write_jsonl(out_meta / "episodes_stats.jsonl", new_episode_stats)
|
| 286 |
+
|
| 287 |
+
info = json.loads(json.dumps(base_info))
|
| 288 |
+
info["total_episodes"] = len(new_episodes)
|
| 289 |
+
info["total_frames"] = global_index
|
| 290 |
+
info["total_videos"] = len(new_episodes) * len(vid_features)
|
| 291 |
+
info["total_chunks"] = max(1, math.ceil(len(new_episodes) / out_chunks_size))
|
| 292 |
+
info["total_tasks"] = len(global_tasks)
|
| 293 |
+
info["splits"] = {"train": f"0:{len(new_episodes)}"}
|
| 294 |
+
info["created_from_merged_gr00t_datasets"] = {
|
| 295 |
+
"created_at": datetime.now(timezone.utc).isoformat(),
|
| 296 |
+
"sources": [str(src) for src in sources],
|
| 297 |
+
"source_episode_counts": dict(source_counts),
|
| 298 |
+
"source_frame_counts": dict(source_frame_counts),
|
| 299 |
+
"video_policy": "symlink" if not args.copy_videos else "copy",
|
| 300 |
+
}
|
| 301 |
+
(out_meta / "info.json").write_text(json.dumps(info, indent=2) + "\n")
|
| 302 |
+
|
| 303 |
+
stats = calculate_stats(parquet_paths)
|
| 304 |
+
(out_meta / "stats.json").write_text(json.dumps(stats, indent=2) + "\n")
|
| 305 |
+
(out_meta / "merged_dataset_manifest.json").write_text(
|
| 306 |
+
json.dumps(
|
| 307 |
+
{
|
| 308 |
+
"created_at": datetime.now(timezone.utc).isoformat(),
|
| 309 |
+
"output": str(args.output),
|
| 310 |
+
"sources": [str(src) for src in sources],
|
| 311 |
+
"episode_count": len(new_episodes),
|
| 312 |
+
"total_frames": global_index,
|
| 313 |
+
"source_episode_counts": dict(source_counts),
|
| 314 |
+
"source_frame_counts": dict(source_frame_counts),
|
| 315 |
+
"episodes": merge_manifest,
|
| 316 |
+
},
|
| 317 |
+
indent=2,
|
| 318 |
+
)
|
| 319 |
+
+ "\n"
|
| 320 |
+
)
|
| 321 |
+
|
| 322 |
+
print(
|
| 323 |
+
json.dumps(
|
| 324 |
+
{
|
| 325 |
+
"output": str(args.output),
|
| 326 |
+
"episodes": len(new_episodes),
|
| 327 |
+
"frames": global_index,
|
| 328 |
+
"videos": len(new_episodes) * len(vid_features),
|
| 329 |
+
"tasks": len(global_tasks),
|
| 330 |
+
"source_episode_counts": dict(source_counts),
|
| 331 |
+
"source_frame_counts": dict(source_frame_counts),
|
| 332 |
+
},
|
| 333 |
+
indent=2,
|
| 334 |
+
)
|
| 335 |
+
)
|
| 336 |
+
|
| 337 |
+
|
| 338 |
+
if __name__ == "__main__":
|
| 339 |
+
main()
|
dataset/render_to_gr00t.py
ADDED
|
@@ -0,0 +1,353 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
"""Render mesh-swapped MimicGen demos into GR00T 1.5 LeRobot-style dataset.
|
| 3 |
+
|
| 4 |
+
Reads gen HDF5s from $WORKSPACE/gen_per_target_sam3d_50/<cat>_<model>/<cat>_<model>_gen/demo.hdf5,
|
| 5 |
+
writes parquet+meta+videos+extras to $DATASET_OUT.
|
| 6 |
+
|
| 7 |
+
This is a lightly-modified copy of /lp-dev/jonghoon/mimicgen_augment/render_per_target.py
|
| 8 |
+
that takes input/output paths from env vars (or defaults), so we can produce a fresh dataset
|
| 9 |
+
without clobbering the existing groot1.5_mimcgen_per_target.
|
| 10 |
+
"""
|
| 11 |
+
import os
|
| 12 |
+
os.environ["MUJOCO_GL"] = "egl"
|
| 13 |
+
|
| 14 |
+
import h5py, json, pathlib, glob, sys, gzip, re
|
| 15 |
+
import numpy as np
|
| 16 |
+
import pyarrow.parquet as pq
|
| 17 |
+
import pyarrow as pa
|
| 18 |
+
|
| 19 |
+
import robosuite, robocasa
|
| 20 |
+
from robomimic.envs.env_robosuite import EnvRobosuite
|
| 21 |
+
import robomimic.utils.obs_utils as ObsUtils
|
| 22 |
+
ObsUtils.initialize_obs_utils_with_obs_specs({'obs': {'low_dim': [], 'rgb': []}})
|
| 23 |
+
|
| 24 |
+
sys.path.insert(0, "/lp-dev/jonghoon/mimicgen_augment")
|
| 25 |
+
from swap_and_render import (
|
| 26 |
+
extract_obs_state, write_meta, CAMERAS, IMG_SIZE,
|
| 27 |
+
_bbox_diag, _bbox_diag_multi, _find_canonical_visual,
|
| 28 |
+
)
|
| 29 |
+
from build_sam3d_sources import find_sam3d_mesh as _find_sam3d_mesh_untextured
|
| 30 |
+
|
| 31 |
+
# The meshes build_sam3d_sources points at carry vertex colours but no UVs, so MuJoCo
|
| 32 |
+
# renders them flat grey. /data/minha/pose6daug/mesh_textured holds the same
|
| 33 |
+
# reconstructions re-baked with UVs + material_0.png, in the same <model>_to_* layout.
|
| 34 |
+
TEXTURED_PAIR_ROOT = pathlib.Path(os.environ.get(
|
| 35 |
+
"TEXTURED_PAIR_ROOT", "/data/minha/pose6daug/mesh_textured/pairs"))
|
| 36 |
+
|
| 37 |
+
|
| 38 |
+
def find_sam3d_mesh(target_cat, target_model):
|
| 39 |
+
for root in sorted(TEXTURED_PAIR_ROOT.glob("*")):
|
| 40 |
+
for pair_dir in sorted(root.glob(f"{target_model}_to_*")):
|
| 41 |
+
obj = pair_dir / "target_rotated" / "original.obj"
|
| 42 |
+
if obj.exists():
|
| 43 |
+
return obj
|
| 44 |
+
return _find_sam3d_mesh_untextured(target_cat, target_model)
|
| 45 |
+
import mujoco, imageio
|
| 46 |
+
|
| 47 |
+
GEN_DIR = pathlib.Path(os.environ.get(
|
| 48 |
+
"GEN_DIR", "/lp-dev/jonghoon/mimicgen_aug_groot15/gen_per_target_sam3d_50"))
|
| 49 |
+
DATASET_ROOT = pathlib.Path(os.environ.get(
|
| 50 |
+
"DATASET_OUT", "/lp-dev/jonghoon/mimicgen_aug_groot15/dataset/groot1.5_mimcgen_per_target_50"))
|
| 51 |
+
|
| 52 |
+
TARGETS = [
|
| 53 |
+
("donut", "donut_5"),
|
| 54 |
+
("jar", "Jar023"),
|
| 55 |
+
("measuring_cup", "MeasuringCup009"),
|
| 56 |
+
("soap_dispenser", "SoapDispenser010"),
|
| 57 |
+
("steak", "steak_8"),
|
| 58 |
+
("syrup_bottle", "SyrupBottle005"),
|
| 59 |
+
("teapot", "teapot_6"),
|
| 60 |
+
("teapot", "teapot_7"),
|
| 61 |
+
]
|
| 62 |
+
|
| 63 |
+
|
| 64 |
+
def swap_visual_to_sam3d(model_xml, sam3d_obj_path):
|
| 65 |
+
tex = pathlib.Path(sam3d_obj_path).parent / "material_0.png"
|
| 66 |
+
swap_texture = tex if tex.exists() else None
|
| 67 |
+
canonical_file, canonical_scale_xyz = _find_canonical_visual(model_xml)
|
| 68 |
+
if canonical_file is None:
|
| 69 |
+
return None
|
| 70 |
+
sam3d_diag = _bbox_diag(str(sam3d_obj_path))
|
| 71 |
+
# _find_canonical_visual returns a LIST of visual submeshes (a robocasa object can be
|
| 72 |
+
# split into several, e.g. teapot = lid + base) and its docstring says to take the union
|
| 73 |
+
# bbox with _bbox_diag_multi. Upstream passed that list to _bbox_diag, which wants one
|
| 74 |
+
# path, so the diagonal came back None and swap_visual_to_sam3d silently returned None --
|
| 75 |
+
# the swap never happened, on any target.
|
| 76 |
+
canonical_diag = (_bbox_diag_multi(canonical_file)
|
| 77 |
+
if isinstance(canonical_file, (list, tuple))
|
| 78 |
+
else _bbox_diag(canonical_file))
|
| 79 |
+
if sam3d_diag is None or canonical_diag is None:
|
| 80 |
+
return None
|
| 81 |
+
mean_canonical = float(np.mean(canonical_scale_xyz))
|
| 82 |
+
target_scale = (canonical_diag * mean_canonical) / sam3d_diag
|
| 83 |
+
scale_str = f'scale="{target_scale:.6f} {target_scale:.6f} {target_scale:.6f}"'
|
| 84 |
+
|
| 85 |
+
out = []
|
| 86 |
+
for line in model_xml.split("\n"):
|
| 87 |
+
s = line.strip()
|
| 88 |
+
if s.startswith("<mesh") and 'name="obj_' in s and "collision" not in s.lower():
|
| 89 |
+
line = re.sub(r'file="[^"]+"', f'file="{sam3d_obj_path}"', line)
|
| 90 |
+
line = re.sub(r' scale="[^"]+"', f' {scale_str}', line)
|
| 91 |
+
if "scale=" not in line:
|
| 92 |
+
line = line.rstrip("/>").rstrip() + f" {scale_str}/>"
|
| 93 |
+
elif s.startswith("<texture") and 'name="obj_' in s:
|
| 94 |
+
if swap_texture is not None:
|
| 95 |
+
# the replacement mesh is UV-mapped and ships its own baked texture
|
| 96 |
+
line = re.sub(r'file="[^"]+"', f'file="{swap_texture}"', line)
|
| 97 |
+
line = re.sub(r' builtin="[^"]+"', "", line)
|
| 98 |
+
else:
|
| 99 |
+
line = re.sub(r'file="[^"]+"', 'builtin="flat"', line)
|
| 100 |
+
if "width=" not in line:
|
| 101 |
+
line = line.rstrip("/>").rstrip() + ' width="2" height="2"/>'
|
| 102 |
+
elif s.startswith("<material") and 'name="obj_' in s:
|
| 103 |
+
if swap_texture is None:
|
| 104 |
+
# no texture to apply -- flat grey beats whatever the stale binding gives
|
| 105 |
+
forced_rgba = "0.7 0.7 0.7 1.0"
|
| 106 |
+
if re.search(r'rgba="[^"]+"', line):
|
| 107 |
+
line = re.sub(r'rgba="[^"]+"', f'rgba="{forced_rgba}"', line)
|
| 108 |
+
else:
|
| 109 |
+
line = line.rstrip("/>").rstrip() + f' rgba="{forced_rgba}"/>'
|
| 110 |
+
line = re.sub(r' texture="[^"]+"', "", line)
|
| 111 |
+
elif re.search(r'rgba="[^"]+"', line):
|
| 112 |
+
line = re.sub(r'rgba="[^"]+"', 'rgba="1 1 1 1"', line)
|
| 113 |
+
out.append(line)
|
| 114 |
+
return "\n".join(out)
|
| 115 |
+
|
| 116 |
+
|
| 117 |
+
def render_demo(env, demo_grp, out_root, episode_index, row_info, global_frame_offset, sam3d_obj_path=None):
|
| 118 |
+
out_root = pathlib.Path(out_root)
|
| 119 |
+
model_xml = demo_grp.attrs["model_file"]
|
| 120 |
+
if sam3d_obj_path is not None:
|
| 121 |
+
swapped = swap_visual_to_sam3d(model_xml, str(sam3d_obj_path))
|
| 122 |
+
if swapped is not None:
|
| 123 |
+
model_xml = swapped
|
| 124 |
+
|
| 125 |
+
states = demo_grp["states"][0]
|
| 126 |
+
state = dict(model=model_xml, states=states)
|
| 127 |
+
if "ep_meta" in demo_grp.attrs:
|
| 128 |
+
state["ep_meta"] = demo_grp.attrs["ep_meta"]
|
| 129 |
+
|
| 130 |
+
try:
|
| 131 |
+
env.reset_to(state)
|
| 132 |
+
except Exception as e:
|
| 133 |
+
print(f" Reset failed: {e}")
|
| 134 |
+
return False, 0
|
| 135 |
+
|
| 136 |
+
all_states = demo_grp["states"][:]
|
| 137 |
+
all_actions = demo_grp["actions"][:]
|
| 138 |
+
sim = env.base_env.sim
|
| 139 |
+
mj_model = sim.model._model
|
| 140 |
+
mj_data = sim.data._data
|
| 141 |
+
n_frames = len(all_states)
|
| 142 |
+
|
| 143 |
+
cam_ids = {cam: mujoco.mj_name2id(mj_model, mujoco.mjtObj.mjOBJ_CAMERA, cam) for cam in CAMERAS}
|
| 144 |
+
scene_option = mujoco.MjvOption()
|
| 145 |
+
scene_option.geomgroup[:] = 0
|
| 146 |
+
scene_option.geomgroup[1] = 1
|
| 147 |
+
scene_option.sitegroup[:] = 0
|
| 148 |
+
|
| 149 |
+
renderer = mujoco.Renderer(mj_model, height=IMG_SIZE, width=IMG_SIZE)
|
| 150 |
+
ep_tag = f"episode_{episode_index:06d}"
|
| 151 |
+
vid_dir = out_root / "videos" / "chunk-000"
|
| 152 |
+
writers = {}
|
| 153 |
+
for cam in CAMERAS:
|
| 154 |
+
cam_dir = vid_dir / f"observation.images.{cam}"
|
| 155 |
+
cam_dir.mkdir(parents=True, exist_ok=True)
|
| 156 |
+
writers[cam] = imageio.get_writer(
|
| 157 |
+
str(cam_dir / f"{ep_tag}.mp4"), fps=20, codec="libx264",
|
| 158 |
+
macro_block_size=1, ffmpeg_params=["-pix_fmt", "yuv420p"],
|
| 159 |
+
)
|
| 160 |
+
|
| 161 |
+
obs_states = []
|
| 162 |
+
for t in range(n_frames):
|
| 163 |
+
sim.set_state_from_flattened(all_states[t])
|
| 164 |
+
sim.forward()
|
| 165 |
+
obs_states.append(extract_obs_state(env))
|
| 166 |
+
for cam in CAMERAS:
|
| 167 |
+
renderer.update_scene(mj_data, camera=cam_ids[cam], scene_option=scene_option)
|
| 168 |
+
writers[cam].append_data(renderer.render())
|
| 169 |
+
|
| 170 |
+
renderer.close()
|
| 171 |
+
for w in writers.values():
|
| 172 |
+
w.close()
|
| 173 |
+
|
| 174 |
+
obs_states = np.array(obs_states)
|
| 175 |
+
try:
|
| 176 |
+
ep_meta_obj = json.loads(demo_grp.attrs.get("ep_meta", "{}"))
|
| 177 |
+
except Exception:
|
| 178 |
+
ep_meta_obj = {}
|
| 179 |
+
|
| 180 |
+
# The HDF5 holds actions in the simulator's own order,
|
| 181 |
+
# [eef_pos(3), eef_rot(3), gripper, base(3), torso, base_mode]
|
| 182 |
+
# while meta/modality.json declares the LeRobot order,
|
| 183 |
+
# [base_motion(4), control_mode, eef_pos(3), eef_rot(3), gripper_close].
|
| 184 |
+
# Upstream wrote them through unchanged, so every slice landed on the wrong dimensions:
|
| 185 |
+
# arm motion was read as base motion and base_mode as the gripper, which trains a policy
|
| 186 |
+
# that drives the base away and never closes the gripper (measured 0/160 at eval).
|
| 187 |
+
actions = all_actions[:, [7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6]]
|
| 188 |
+
if len(actions) < n_frames:
|
| 189 |
+
pad = np.zeros((n_frames - len(actions), actions.shape[1]))
|
| 190 |
+
actions = np.vstack([actions, pad])
|
| 191 |
+
|
| 192 |
+
records = []
|
| 193 |
+
for t in range(n_frames):
|
| 194 |
+
is_last = (t == n_frames - 1)
|
| 195 |
+
records.append({
|
| 196 |
+
"observation.state": obs_states[t].tolist(),
|
| 197 |
+
"action": actions[t].tolist(),
|
| 198 |
+
"next.reward": 1.0 if is_last else 0.0,
|
| 199 |
+
"next.done": is_last,
|
| 200 |
+
"timestamp": t / 20.0,
|
| 201 |
+
"frame_index": t,
|
| 202 |
+
"episode_index": episode_index,
|
| 203 |
+
"index": global_frame_offset + t,
|
| 204 |
+
"task_index": 0,
|
| 205 |
+
"annotation.human.task_description": 0,
|
| 206 |
+
"annotation.human.task_name": 0,
|
| 207 |
+
})
|
| 208 |
+
|
| 209 |
+
schema = pa.schema([
|
| 210 |
+
("observation.state", pa.list_(pa.float64())),
|
| 211 |
+
("action", pa.list_(pa.float64())),
|
| 212 |
+
("next.reward", pa.float32()),
|
| 213 |
+
("next.done", pa.bool_()),
|
| 214 |
+
("timestamp", pa.float32()),
|
| 215 |
+
("frame_index", pa.int64()),
|
| 216 |
+
("episode_index", pa.int64()),
|
| 217 |
+
("index", pa.int64()),
|
| 218 |
+
("task_index", pa.int64()),
|
| 219 |
+
("annotation.human.task_description", pa.int64()),
|
| 220 |
+
("annotation.human.task_name", pa.int64()),
|
| 221 |
+
])
|
| 222 |
+
table = pa.Table.from_pylist(records, schema=schema)
|
| 223 |
+
data_dir = out_root / "data" / "chunk-000"
|
| 224 |
+
data_dir.mkdir(parents=True, exist_ok=True)
|
| 225 |
+
pq.write_table(table, str(data_dir / f"{ep_tag}.parquet"))
|
| 226 |
+
|
| 227 |
+
extras_dir = out_root / "extras" / ep_tag
|
| 228 |
+
extras_dir.mkdir(parents=True, exist_ok=True)
|
| 229 |
+
aug_meta = dict(ep_meta_obj)
|
| 230 |
+
aug_meta["augmentation"] = {
|
| 231 |
+
"type": "mimicgen_per_target_sam3d_50",
|
| 232 |
+
"target_mesh": row_info["target_mesh"],
|
| 233 |
+
"source_demo": row_info["source_demo"],
|
| 234 |
+
}
|
| 235 |
+
(extras_dir / "ep_meta.json").write_text(json.dumps(aug_meta, indent=2))
|
| 236 |
+
with gzip.open(str(extras_dir / "model.xml.gz"), "wt") as f:
|
| 237 |
+
f.write(model_xml)
|
| 238 |
+
np.savez_compressed(str(extras_dir / "states.npz"), states=all_states)
|
| 239 |
+
|
| 240 |
+
return True, n_frames
|
| 241 |
+
|
| 242 |
+
|
| 243 |
+
def main():
|
| 244 |
+
DATASET_ROOT.mkdir(parents=True, exist_ok=True)
|
| 245 |
+
print(f"GEN_DIR={GEN_DIR}")
|
| 246 |
+
print(f"DATASET_ROOT={DATASET_ROOT}")
|
| 247 |
+
|
| 248 |
+
target_demos = {}
|
| 249 |
+
for cat, model in TARGETS:
|
| 250 |
+
name = f"{cat}_{model}"
|
| 251 |
+
gen_paths = sorted(glob.glob(f"{GEN_DIR}/{name}*/{name}_gen/demo.hdf5"))
|
| 252 |
+
if not gen_paths:
|
| 253 |
+
print(f" WARN: no demos for {cat}/{model} in {GEN_DIR}/{name}")
|
| 254 |
+
target_demos[(cat, model)] = []
|
| 255 |
+
continue
|
| 256 |
+
target_demos[(cat, model)] = gen_paths
|
| 257 |
+
|
| 258 |
+
first_hdf5 = None
|
| 259 |
+
for paths in target_demos.values():
|
| 260 |
+
if paths:
|
| 261 |
+
first_hdf5 = paths[0]
|
| 262 |
+
break
|
| 263 |
+
if first_hdf5 is None:
|
| 264 |
+
print("No demos found at all!")
|
| 265 |
+
return
|
| 266 |
+
|
| 267 |
+
fh0 = h5py.File(first_hdf5, "r")
|
| 268 |
+
env_meta = json.loads(fh0["data"].attrs["env_args"])
|
| 269 |
+
fh0.close()
|
| 270 |
+
env_kwargs = dict(env_meta["env_kwargs"])
|
| 271 |
+
env_kwargs["has_offscreen_renderer"] = True
|
| 272 |
+
env_kwargs["use_camera_obs"] = False
|
| 273 |
+
env = EnvRobosuite(env_name=env_meta["env_name"], **env_kwargs)
|
| 274 |
+
|
| 275 |
+
episodes_info = []
|
| 276 |
+
task_descriptions = set()
|
| 277 |
+
global_frame_offset = 0
|
| 278 |
+
success_count = 0
|
| 279 |
+
ep_idx = 0
|
| 280 |
+
|
| 281 |
+
only = os.environ.get("TARGET_FILTER") # "<cat>/<model>" -> shard this run to one target
|
| 282 |
+
for cat, model in TARGETS:
|
| 283 |
+
if only and f"{cat}/{model}" != only:
|
| 284 |
+
continue
|
| 285 |
+
gen_paths = target_demos[(cat, model)]
|
| 286 |
+
target_label = f"{cat}/{model}"
|
| 287 |
+
# The visual-mesh swap is OFF here. Upstream replaces the object's visual mesh with
|
| 288 |
+
# a reconstruction from an eval rollout and forces it grey; neither the Ours nor the
|
| 289 |
+
# VACE dataset builder does that, and our episodes were generated (and already
|
| 290 |
+
# inspected in output_aug_256) with the plain RoboCasa asset. USE_MESH_SWAP=1 restores
|
| 291 |
+
# the upstream behaviour.
|
| 292 |
+
sam3d_mesh = find_sam3d_mesh(cat, model) if os.environ.get("USE_MESH_SWAP") == "1" else None
|
| 293 |
+
print(f" -> {target_label}: mesh swap = {sam3d_mesh}")
|
| 294 |
+
n_for_target = 0
|
| 295 |
+
for hp in gen_paths:
|
| 296 |
+
fh = h5py.File(hp, "r")
|
| 297 |
+
for dk in sorted(fh["data"].keys(), key=lambda x: int(x.split("_")[1])):
|
| 298 |
+
row_info = {
|
| 299 |
+
"target_mesh": target_label,
|
| 300 |
+
"source_demo": f"{hp}::{dk}",
|
| 301 |
+
}
|
| 302 |
+
|
| 303 |
+
ep_tag = f"episode_{ep_idx:06d}"
|
| 304 |
+
parquet_path = DATASET_ROOT / "data" / "chunk-000" / f"{ep_tag}.parquet"
|
| 305 |
+
if parquet_path.exists():
|
| 306 |
+
tbl = pq.read_table(str(parquet_path))
|
| 307 |
+
n = len(tbl)
|
| 308 |
+
em_path = DATASET_ROOT / "extras" / ep_tag / "ep_meta.json"
|
| 309 |
+
td = "Pick and place"
|
| 310 |
+
if em_path.exists():
|
| 311 |
+
em = json.loads(em_path.read_text())
|
| 312 |
+
td = em.get("lang", td)
|
| 313 |
+
episodes_info.append({"episode_index": ep_idx, "tasks": [td], "length": n})
|
| 314 |
+
task_descriptions.add(td)
|
| 315 |
+
global_frame_offset += n
|
| 316 |
+
success_count += 1
|
| 317 |
+
n_for_target += 1
|
| 318 |
+
ep_idx += 1
|
| 319 |
+
print(f" ep{ep_idx-1:03d} [{target_label}] {dk} — already done")
|
| 320 |
+
continue
|
| 321 |
+
|
| 322 |
+
print(f" ep{ep_idx:03d} [{target_label}] {dk}", flush=True)
|
| 323 |
+
ok, n_frames = render_demo(
|
| 324 |
+
env, fh[f"data/{dk}"],
|
| 325 |
+
DATASET_ROOT, ep_idx, row_info, global_frame_offset,
|
| 326 |
+
sam3d_obj_path=sam3d_mesh,
|
| 327 |
+
)
|
| 328 |
+
if ok:
|
| 329 |
+
success_count += 1
|
| 330 |
+
n_for_target += 1
|
| 331 |
+
em_path = DATASET_ROOT / "extras" / ep_tag / "ep_meta.json"
|
| 332 |
+
em = json.loads(em_path.read_text()) if em_path.exists() else {}
|
| 333 |
+
td = em.get("lang", f"Pick and place {cat}")
|
| 334 |
+
episodes_info.append({"episode_index": ep_idx, "tasks": [td], "length": n_frames})
|
| 335 |
+
task_descriptions.add(td)
|
| 336 |
+
global_frame_offset += n_frames
|
| 337 |
+
print(f" OK ({n_frames} frames)")
|
| 338 |
+
else:
|
| 339 |
+
print(f" FAILED")
|
| 340 |
+
ep_idx += 1
|
| 341 |
+
fh.close()
|
| 342 |
+
print(f" -> {target_label}: {n_for_target} episodes")
|
| 343 |
+
|
| 344 |
+
env.close()
|
| 345 |
+
|
| 346 |
+
if episodes_info:
|
| 347 |
+
write_meta(DATASET_ROOT, episodes_info, sorted(task_descriptions))
|
| 348 |
+
print(f"Wrote meta/ with {len(episodes_info)} episodes")
|
| 349 |
+
print(f"Done: {success_count} episodes total")
|
| 350 |
+
|
| 351 |
+
|
| 352 |
+
if __name__ == "__main__":
|
| 353 |
+
main()
|
dataset/repair_action_order.py
ADDED
|
@@ -0,0 +1,71 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
"""Reorder the action column of an already-built gr00t_views dataset, in place.
|
| 3 |
+
|
| 4 |
+
Cheaper than rebuilding: the videos and observation.state are correct, only `action` was
|
| 5 |
+
written in simulator order. Rewrites every episode parquet and recomputes meta/stats.json for
|
| 6 |
+
the action key (mean/std/min/max/q01/q99), leaving every other stats entry untouched.
|
| 7 |
+
|
| 8 |
+
Refuses to run unless the dataset currently looks like simulator order, so it cannot be
|
| 9 |
+
applied twice.
|
| 10 |
+
"""
|
| 11 |
+
import argparse, glob, json, os, shutil, sys
|
| 12 |
+
import numpy as np
|
| 13 |
+
import pandas as pd
|
| 14 |
+
|
| 15 |
+
FIX = [7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6]
|
| 16 |
+
|
| 17 |
+
|
| 18 |
+
def looks_robosuite(a):
|
| 19 |
+
zero = [i for i in range(12) if np.abs(a[:, i]).max() < 1e-6]
|
| 20 |
+
const = [i for i in range(12) if len(np.unique(np.round(a[:, i], 4))) == 1]
|
| 21 |
+
two = [i for i in range(12) if len(np.unique(np.round(a[:, i], 4))) == 2]
|
| 22 |
+
return set([7, 8, 9, 10]).issubset(zero) and 11 in const and 6 in two
|
| 23 |
+
|
| 24 |
+
|
| 25 |
+
def main():
|
| 26 |
+
ap = argparse.ArgumentParser()
|
| 27 |
+
ap.add_argument("--dataset", required=True)
|
| 28 |
+
ap.add_argument("--dry-run", action="store_true")
|
| 29 |
+
a = ap.parse_args()
|
| 30 |
+
|
| 31 |
+
paths = sorted(glob.glob(os.path.join(a.dataset, "data", "chunk-*", "*.parquet")))
|
| 32 |
+
if not paths:
|
| 33 |
+
sys.exit(f"no parquet under {a.dataset}")
|
| 34 |
+
|
| 35 |
+
probe = np.vstack([np.stack(pd.read_parquet(p, columns=["action"])["action"].tolist())
|
| 36 |
+
for p in paths[:10]])
|
| 37 |
+
if not looks_robosuite(probe):
|
| 38 |
+
sys.exit("dataset does not look like simulator order -- refusing "
|
| 39 |
+
"(already fixed, or a layout this tool does not know)")
|
| 40 |
+
|
| 41 |
+
print(f"{len(paths)} episodes to rewrite")
|
| 42 |
+
if a.dry_run:
|
| 43 |
+
return
|
| 44 |
+
|
| 45 |
+
all_actions = []
|
| 46 |
+
for i, p in enumerate(paths):
|
| 47 |
+
df = pd.read_parquet(p)
|
| 48 |
+
arr = np.stack(df["action"].tolist())[:, FIX]
|
| 49 |
+
df["action"] = list(arr)
|
| 50 |
+
df.to_parquet(p, index=False)
|
| 51 |
+
all_actions.append(arr)
|
| 52 |
+
if (i + 1) % 64 == 0:
|
| 53 |
+
print(f" {i+1}/{len(paths)}", flush=True)
|
| 54 |
+
A = np.vstack(all_actions)
|
| 55 |
+
|
| 56 |
+
sp = os.path.join(a.dataset, "meta", "stats.json")
|
| 57 |
+
shutil.copy(sp, sp + ".bak_action_order")
|
| 58 |
+
stats = json.load(open(sp))
|
| 59 |
+
stats["action"] = {
|
| 60 |
+
"mean": A.mean(0).tolist(), "std": A.std(0).tolist(),
|
| 61 |
+
"min": A.min(0).tolist(), "max": A.max(0).tolist(),
|
| 62 |
+
"q01": np.percentile(A, 1, axis=0).tolist(),
|
| 63 |
+
"q99": np.percentile(A, 99, axis=0).tolist(),
|
| 64 |
+
}
|
| 65 |
+
json.dump(stats, open(sp, "w"), indent=2)
|
| 66 |
+
print(f"rewrote {len(paths)} parquet, recomputed action stats over {A.shape[0]} frames")
|
| 67 |
+
print(f" stats backup: {sp}.bak_action_order")
|
| 68 |
+
|
| 69 |
+
|
| 70 |
+
if __name__ == "__main__":
|
| 71 |
+
raise SystemExit(main())
|
dataset/repair_mimicgen_task_ids.py
ADDED
|
@@ -0,0 +1,208 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
"""Repair per-episode task-id columns and modality aliases in MimicGen GR00T datasets.
|
| 3 |
+
|
| 4 |
+
The MimicGen swap/render writer stores all parquet task columns as 0 while
|
| 5 |
+
episodes.jsonl stores the intended language per episode. GR00T training reads
|
| 6 |
+
the parquet task ids through tasks.jsonl, so every episode becomes task 0 unless
|
| 7 |
+
we remap these columns.
|
| 8 |
+
|
| 9 |
+
It also writes raw RoboCasa camera names as modality keys. The
|
| 10 |
+
single_panda_gripper config expects the canonical GR00T view names
|
| 11 |
+
left_view/right_view/wrist_view, each pointing to the raw observation key.
|
| 12 |
+
"""
|
| 13 |
+
|
| 14 |
+
from __future__ import annotations
|
| 15 |
+
|
| 16 |
+
import argparse
|
| 17 |
+
import json
|
| 18 |
+
from collections import Counter
|
| 19 |
+
from pathlib import Path
|
| 20 |
+
|
| 21 |
+
import numpy as np
|
| 22 |
+
import pandas as pd
|
| 23 |
+
|
| 24 |
+
|
| 25 |
+
TASK_COLUMNS = (
|
| 26 |
+
"task_index",
|
| 27 |
+
"annotation.human.task_description",
|
| 28 |
+
"annotation.human.task_name",
|
| 29 |
+
)
|
| 30 |
+
|
| 31 |
+
CANONICAL_VIDEO_MODALITY = {
|
| 32 |
+
"left_view": {"original_key": "observation.images.robot0_agentview_left"},
|
| 33 |
+
"right_view": {"original_key": "observation.images.robot0_agentview_right"},
|
| 34 |
+
"wrist_view": {"original_key": "observation.images.robot0_eye_in_hand"},
|
| 35 |
+
}
|
| 36 |
+
|
| 37 |
+
CANONICAL_ANNOTATION_MODALITY = {
|
| 38 |
+
"human.action.task_description": {"original_key": "annotation.human.task_description"},
|
| 39 |
+
}
|
| 40 |
+
|
| 41 |
+
CANONICAL_STATE_MODALITY = {
|
| 42 |
+
"base_position": {"original_key": "observation.state", "start": 0, "end": 3},
|
| 43 |
+
"base_rotation": {
|
| 44 |
+
"original_key": "observation.state",
|
| 45 |
+
"start": 3,
|
| 46 |
+
"end": 7,
|
| 47 |
+
"rotation_type": "quaternion",
|
| 48 |
+
},
|
| 49 |
+
"end_effector_position_relative": {"original_key": "observation.state", "start": 7, "end": 10},
|
| 50 |
+
"end_effector_rotation_relative": {
|
| 51 |
+
"original_key": "observation.state",
|
| 52 |
+
"start": 10,
|
| 53 |
+
"end": 14,
|
| 54 |
+
"rotation_type": "quaternion",
|
| 55 |
+
},
|
| 56 |
+
"gripper_qpos": {"original_key": "observation.state", "start": 14, "end": 16},
|
| 57 |
+
}
|
| 58 |
+
|
| 59 |
+
CANONICAL_ACTION_MODALITY = {
|
| 60 |
+
"base_motion": {"original_key": "action", "start": 0, "end": 4},
|
| 61 |
+
"control_mode": {"original_key": "action", "start": 4, "end": 5},
|
| 62 |
+
"end_effector_position": {"original_key": "action", "start": 5, "end": 8},
|
| 63 |
+
"end_effector_rotation": {"original_key": "action", "start": 8, "end": 11},
|
| 64 |
+
"gripper_close": {"original_key": "action", "start": 11, "end": 12},
|
| 65 |
+
}
|
| 66 |
+
|
| 67 |
+
|
| 68 |
+
def parse_args() -> argparse.Namespace:
|
| 69 |
+
parser = argparse.ArgumentParser(description=__doc__)
|
| 70 |
+
parser.add_argument("--dataset", type=Path, required=True)
|
| 71 |
+
parser.add_argument("--dry-run", action="store_true")
|
| 72 |
+
return parser.parse_args()
|
| 73 |
+
|
| 74 |
+
|
| 75 |
+
def load_jsonl(path: Path) -> list[dict]:
|
| 76 |
+
with path.open() as f:
|
| 77 |
+
return [json.loads(line) for line in f if line.strip()]
|
| 78 |
+
|
| 79 |
+
|
| 80 |
+
def episode_path(dataset_root: Path, episode_index: int, chunks_size: int) -> Path:
|
| 81 |
+
chunk = episode_index // chunks_size
|
| 82 |
+
return dataset_root / "data" / f"chunk-{chunk:03d}" / f"episode_{episode_index:06d}.parquet"
|
| 83 |
+
|
| 84 |
+
|
| 85 |
+
def main() -> None:
|
| 86 |
+
args = parse_args()
|
| 87 |
+
dataset = args.dataset.resolve()
|
| 88 |
+
meta = dataset / "meta"
|
| 89 |
+
info_path = meta / "info.json"
|
| 90 |
+
if not info_path.exists():
|
| 91 |
+
raise FileNotFoundError(f"missing {info_path}")
|
| 92 |
+
|
| 93 |
+
info = json.loads(info_path.read_text())
|
| 94 |
+
chunks_size = int(info.get("chunks_size", 1000))
|
| 95 |
+
episodes = load_jsonl(meta / "episodes.jsonl")
|
| 96 |
+
tasks_path = meta / "tasks.jsonl"
|
| 97 |
+
tasks = load_jsonl(tasks_path)
|
| 98 |
+
|
| 99 |
+
episode_task_texts: list[str] = []
|
| 100 |
+
for episode in episodes:
|
| 101 |
+
for task_text in episode.get("tasks") or []:
|
| 102 |
+
if task_text not in episode_task_texts:
|
| 103 |
+
episode_task_texts.append(str(task_text))
|
| 104 |
+
|
| 105 |
+
existing_task_to_id = {row["task"]: int(row["task_index"]) for row in tasks}
|
| 106 |
+
missing_tasks = [task for task in episode_task_texts if task not in existing_task_to_id]
|
| 107 |
+
tasks_rebuilt = False
|
| 108 |
+
if missing_tasks:
|
| 109 |
+
# Some subset datasets were written with tasks.jsonl containing only task 0
|
| 110 |
+
# even though episodes.jsonl had the full per-episode language. Rebuild the
|
| 111 |
+
# task table from episodes.jsonl so parquet ids can be repaired coherently.
|
| 112 |
+
tasks = [{"task_index": idx, "task": task} for idx, task in enumerate(episode_task_texts)]
|
| 113 |
+
tasks_rebuilt = True
|
| 114 |
+
if not args.dry_run:
|
| 115 |
+
tasks_path.write_text("".join(json.dumps(row, sort_keys=False) + "\n" for row in tasks))
|
| 116 |
+
info["total_tasks"] = len(tasks)
|
| 117 |
+
info_path.write_text(json.dumps(info, indent=2, sort_keys=False) + "\n")
|
| 118 |
+
|
| 119 |
+
task_to_id = {row["task"]: int(row["task_index"]) for row in tasks}
|
| 120 |
+
|
| 121 |
+
modality_path = meta / "modality.json"
|
| 122 |
+
modality_changed = False
|
| 123 |
+
if modality_path.exists():
|
| 124 |
+
modality = json.loads(modality_path.read_text())
|
| 125 |
+
if modality.get("state") != CANONICAL_STATE_MODALITY:
|
| 126 |
+
modality["state"] = CANONICAL_STATE_MODALITY
|
| 127 |
+
modality_changed = True
|
| 128 |
+
if modality.get("action") != CANONICAL_ACTION_MODALITY:
|
| 129 |
+
modality["action"] = CANONICAL_ACTION_MODALITY
|
| 130 |
+
modality_changed = True
|
| 131 |
+
if modality.get("video") != CANONICAL_VIDEO_MODALITY:
|
| 132 |
+
modality["video"] = CANONICAL_VIDEO_MODALITY
|
| 133 |
+
modality_changed = True
|
| 134 |
+
if modality.get("annotation") != CANONICAL_ANNOTATION_MODALITY:
|
| 135 |
+
modality["annotation"] = CANONICAL_ANNOTATION_MODALITY
|
| 136 |
+
modality_changed = True
|
| 137 |
+
if modality_changed and not args.dry_run:
|
| 138 |
+
modality_path.write_text(json.dumps(modality, indent=2, sort_keys=False) + "\n")
|
| 139 |
+
|
| 140 |
+
rewrites: list[dict] = []
|
| 141 |
+
failures: list[str] = []
|
| 142 |
+
text_counts: Counter[str] = Counter()
|
| 143 |
+
|
| 144 |
+
for episode in episodes:
|
| 145 |
+
episode_index = int(episode["episode_index"])
|
| 146 |
+
task_texts = episode.get("tasks") or []
|
| 147 |
+
if len(task_texts) != 1:
|
| 148 |
+
failures.append(f"episode {episode_index}: expected exactly one task, got {task_texts!r}")
|
| 149 |
+
continue
|
| 150 |
+
task_text = str(task_texts[0])
|
| 151 |
+
if task_text not in task_to_id:
|
| 152 |
+
failures.append(f"episode {episode_index}: task text missing from tasks.jsonl: {task_text!r}")
|
| 153 |
+
continue
|
| 154 |
+
task_id = int(task_to_id[task_text])
|
| 155 |
+
path = episode_path(dataset, episode_index, chunks_size)
|
| 156 |
+
if not path.exists():
|
| 157 |
+
failures.append(f"episode {episode_index}: missing parquet {path}")
|
| 158 |
+
continue
|
| 159 |
+
|
| 160 |
+
df = pd.read_parquet(path)
|
| 161 |
+
before = {}
|
| 162 |
+
changed = False
|
| 163 |
+
for column in TASK_COLUMNS:
|
| 164 |
+
if column not in df.columns:
|
| 165 |
+
failures.append(f"episode {episode_index}: missing task column {column}")
|
| 166 |
+
continue
|
| 167 |
+
unique_before = sorted({int(v) for v in df[column].unique()})
|
| 168 |
+
before[column] = unique_before
|
| 169 |
+
desired = np.full(len(df), task_id, dtype=np.int64)
|
| 170 |
+
if unique_before != [task_id]:
|
| 171 |
+
df[column] = desired
|
| 172 |
+
changed = True
|
| 173 |
+
if changed and not args.dry_run:
|
| 174 |
+
df.to_parquet(path, index=False)
|
| 175 |
+
rewrites.append(
|
| 176 |
+
{
|
| 177 |
+
"episode_index": episode_index,
|
| 178 |
+
"task": task_text,
|
| 179 |
+
"task_id": task_id,
|
| 180 |
+
"before": before,
|
| 181 |
+
"changed": changed,
|
| 182 |
+
"length": len(df),
|
| 183 |
+
}
|
| 184 |
+
)
|
| 185 |
+
text_counts[task_text] += 1
|
| 186 |
+
|
| 187 |
+
report = {
|
| 188 |
+
"dataset": str(dataset),
|
| 189 |
+
"dry_run": bool(args.dry_run),
|
| 190 |
+
"episodes_checked": len(episodes),
|
| 191 |
+
"tasks_rebuilt": tasks_rebuilt,
|
| 192 |
+
"missing_tasks_before_rebuild": missing_tasks,
|
| 193 |
+
"modality_changed": modality_changed,
|
| 194 |
+
"rewritten_episodes": sum(1 for row in rewrites if row["changed"]),
|
| 195 |
+
"task_counts": dict(text_counts),
|
| 196 |
+
"failures": failures,
|
| 197 |
+
"passed": not failures,
|
| 198 |
+
}
|
| 199 |
+
out = meta / "mimicgen_task_id_repair.json"
|
| 200 |
+
if not args.dry_run:
|
| 201 |
+
out.write_text(json.dumps({**report, "episodes": rewrites}, indent=2, sort_keys=True) + "\n")
|
| 202 |
+
print(json.dumps(report, indent=2, sort_keys=True))
|
| 203 |
+
if failures:
|
| 204 |
+
raise SystemExit(1)
|
| 205 |
+
|
| 206 |
+
|
| 207 |
+
if __name__ == "__main__":
|
| 208 |
+
main()
|
dataset/run_convert.sh
ADDED
|
@@ -0,0 +1,22 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env bash
|
| 2 |
+
# gr00t_views conversion, SAM3D textured mesh (USE_MESH_SWAP=1), one output per shard;
|
| 3 |
+
# merged afterwards. Each shard needs its own GEN_DIR or the glob would pick up the others.
|
| 4 |
+
set -u
|
| 5 |
+
N=/tmp/claude-1001/-home-nvidia/68c27ffb-706b-4cb3-8bf2-4618a200d378/scratchpad/natural256
|
| 6 |
+
B=/home/nvidia/minha/pose6daug/baseline/mimicgen
|
| 7 |
+
PY=/lp-dev/jonghoon/mimicgen_augment/envs/mimicgen/bin/python
|
| 8 |
+
GPUS=(0 2 3)
|
| 9 |
+
mkdir -p "$N/logs" "$B/gr00t_views/natural_shards"
|
| 10 |
+
i=0
|
| 11 |
+
for d in "$N"/gen/shard_*; do
|
| 12 |
+
k=$(basename "$d")
|
| 13 |
+
tgt=$(basename "$(ls -d "$d"/*/ | head -1)")
|
| 14 |
+
g=${GPUS[$((i % ${#GPUS[@]}))]}
|
| 15 |
+
MUJOCO_GL=egl PYOPENGL_PLATFORM=egl MUJOCO_EGL_DEVICE_ID=$g \
|
| 16 |
+
GEN_DIR="$d" DATASET_OUT="$B/gr00t_views/natural_shards/$k" USE_MESH_SWAP=1 \
|
| 17 |
+
taskset -c 0-254 nohup "$PY" -u "$B/gr00t_build/render_to_gr00t.py" \
|
| 18 |
+
> "$N/logs/conv_$k.log" 2>&1 &
|
| 19 |
+
i=$((i+1))
|
| 20 |
+
done
|
| 21 |
+
wait
|
| 22 |
+
echo "[nat] convert done $(date -u +%FT%TZ)"
|
dataset/verify_geom_objwise_gr00t_dataset.py
ADDED
|
@@ -0,0 +1,233 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
"""Verify a geometric object-wise GR00T dataset preserves source episode data.
|
| 3 |
+
|
| 4 |
+
Expected differences from the source parquets are limited to:
|
| 5 |
+
- episode_index remapped to the new dataset episode id
|
| 6 |
+
- frame_index reset to 0..N-1
|
| 7 |
+
- index remapped to the new global frame range
|
| 8 |
+
- task id columns remapped to this combined dataset's tasks.jsonl ids
|
| 9 |
+
|
| 10 |
+
All state/action/reward/done/timestamp values must match the source episode.
|
| 11 |
+
Videos must resolve to the augmented render output recorded by the builder.
|
| 12 |
+
"""
|
| 13 |
+
|
| 14 |
+
from __future__ import annotations
|
| 15 |
+
|
| 16 |
+
import argparse
|
| 17 |
+
import json
|
| 18 |
+
import subprocess
|
| 19 |
+
from pathlib import Path
|
| 20 |
+
from typing import Any
|
| 21 |
+
|
| 22 |
+
import numpy as np
|
| 23 |
+
import pandas as pd
|
| 24 |
+
|
| 25 |
+
|
| 26 |
+
TASK_COLUMNS = (
|
| 27 |
+
"task_index",
|
| 28 |
+
"annotation.human.task_description",
|
| 29 |
+
"annotation.human.task_name",
|
| 30 |
+
)
|
| 31 |
+
|
| 32 |
+
INDEX_COLUMNS = ("episode_index", "frame_index", "index")
|
| 33 |
+
|
| 34 |
+
|
| 35 |
+
def parse_args() -> argparse.Namespace:
|
| 36 |
+
parser = argparse.ArgumentParser(description=__doc__)
|
| 37 |
+
parser.add_argument("--dataset", type=Path, required=True)
|
| 38 |
+
parser.add_argument("--source-root", type=Path, default=Path("/home/nvidia/jonghoon/robocasa_full/pickplace_target_human"))
|
| 39 |
+
parser.add_argument("--render-root", type=Path, default=Path("/home/nvidia/jonghoon/robocasa_full/augmented_video/groot1.5_objwise_ours"))
|
| 40 |
+
parser.add_argument("--check-video-frames", action="store_true")
|
| 41 |
+
parser.add_argument("--allow-copied-videos", action="store_true")
|
| 42 |
+
return parser.parse_args()
|
| 43 |
+
|
| 44 |
+
|
| 45 |
+
def load_json(path: Path) -> dict[str, Any]:
|
| 46 |
+
return json.loads(path.read_text())
|
| 47 |
+
|
| 48 |
+
|
| 49 |
+
def load_jsonl(path: Path) -> list[dict[str, Any]]:
|
| 50 |
+
with path.open() as f:
|
| 51 |
+
return [json.loads(line) for line in f if line.strip()]
|
| 52 |
+
|
| 53 |
+
|
| 54 |
+
def source_tasks(source_root: Path) -> dict[str, dict[int, str]]:
|
| 55 |
+
out: dict[str, dict[int, str]] = {}
|
| 56 |
+
for meta in sorted(source_root.glob("*/meta/tasks.jsonl")):
|
| 57 |
+
task_name = meta.parent.parent.name
|
| 58 |
+
out[task_name] = {int(r["task_index"]): r["task"] for r in load_jsonl(meta)}
|
| 59 |
+
return out
|
| 60 |
+
|
| 61 |
+
|
| 62 |
+
def output_tasks(dataset: Path) -> dict[int, str]:
|
| 63 |
+
return {int(r["task_index"]): r["task"] for r in load_jsonl(dataset / "meta" / "tasks.jsonl")}
|
| 64 |
+
|
| 65 |
+
|
| 66 |
+
def dataset_episode_path(dataset: Path, episode_index: int, chunks_size: int) -> Path:
|
| 67 |
+
chunk = episode_index // chunks_size
|
| 68 |
+
return dataset / "data" / f"chunk-{chunk:03d}" / f"episode_{episode_index:06d}.parquet"
|
| 69 |
+
|
| 70 |
+
|
| 71 |
+
def arrays_equal(left: Any, right: Any) -> bool:
|
| 72 |
+
left_arr = np.asarray(left)
|
| 73 |
+
right_arr = np.asarray(right)
|
| 74 |
+
if left_arr.shape != right_arr.shape:
|
| 75 |
+
return False
|
| 76 |
+
return bool(np.array_equal(left_arr, right_arr))
|
| 77 |
+
|
| 78 |
+
|
| 79 |
+
def series_equal(left: pd.Series, right: pd.Series) -> tuple[bool, str | None]:
|
| 80 |
+
if len(left) != len(right):
|
| 81 |
+
return False, f"length mismatch {len(left)} != {len(right)}"
|
| 82 |
+
for idx, (lv, rv) in enumerate(zip(left.tolist(), right.tolist())):
|
| 83 |
+
if not arrays_equal(lv, rv):
|
| 84 |
+
return False, f"row {idx}: {lv!r} != {rv!r}"
|
| 85 |
+
return True, None
|
| 86 |
+
|
| 87 |
+
|
| 88 |
+
def frame_count(path: Path) -> int | None:
|
| 89 |
+
commands = [
|
| 90 |
+
[
|
| 91 |
+
"ffprobe",
|
| 92 |
+
"-v",
|
| 93 |
+
"error",
|
| 94 |
+
"-count_frames",
|
| 95 |
+
"-select_streams",
|
| 96 |
+
"v:0",
|
| 97 |
+
"-show_entries",
|
| 98 |
+
"stream=nb_read_frames",
|
| 99 |
+
"-of",
|
| 100 |
+
"csv=p=0",
|
| 101 |
+
str(path),
|
| 102 |
+
],
|
| 103 |
+
[
|
| 104 |
+
"ffprobe",
|
| 105 |
+
"-v",
|
| 106 |
+
"error",
|
| 107 |
+
"-select_streams",
|
| 108 |
+
"v:0",
|
| 109 |
+
"-show_entries",
|
| 110 |
+
"stream=nb_frames",
|
| 111 |
+
"-of",
|
| 112 |
+
"csv=p=0",
|
| 113 |
+
str(path),
|
| 114 |
+
],
|
| 115 |
+
]
|
| 116 |
+
for cmd in commands:
|
| 117 |
+
proc = subprocess.run(cmd, check=False, text=True, capture_output=True)
|
| 118 |
+
value = proc.stdout.strip().splitlines()[0] if proc.stdout.strip() else ""
|
| 119 |
+
if value and value != "N/A":
|
| 120 |
+
try:
|
| 121 |
+
return int(value)
|
| 122 |
+
except ValueError:
|
| 123 |
+
continue
|
| 124 |
+
return None
|
| 125 |
+
|
| 126 |
+
|
| 127 |
+
def main() -> None:
|
| 128 |
+
args = parse_args()
|
| 129 |
+
info = load_json(args.dataset / "meta" / "info.json")
|
| 130 |
+
chunks_size = int(info.get("chunks_size", 1000))
|
| 131 |
+
manifest_path = args.dataset / "meta" / "geom_objwise_dataset_manifest.json"
|
| 132 |
+
manifest = load_json(manifest_path)
|
| 133 |
+
src_tasks = source_tasks(args.source_root)
|
| 134 |
+
dst_tasks = output_tasks(args.dataset)
|
| 135 |
+
|
| 136 |
+
failures: list[str] = []
|
| 137 |
+
checked_rows = 0
|
| 138 |
+
changed_column_counts = {name: 0 for name in INDEX_COLUMNS + TASK_COLUMNS}
|
| 139 |
+
video_checks = 0
|
| 140 |
+
frame_checks = 0
|
| 141 |
+
|
| 142 |
+
for episode in manifest["episodes"]:
|
| 143 |
+
new_ep = int(episode["episode_index"])
|
| 144 |
+
source_task = str(episode["source_task"])
|
| 145 |
+
source_episode = int(episode["source_episode_index"])
|
| 146 |
+
length = int(episode["source_length"])
|
| 147 |
+
global_start = int(episode["global_index_start"])
|
| 148 |
+
src_parquet = Path(episode["source_parquet"])
|
| 149 |
+
dst_parquet = dataset_episode_path(args.dataset, new_ep, chunks_size)
|
| 150 |
+
src_df = pd.read_parquet(src_parquet)
|
| 151 |
+
dst_df = pd.read_parquet(dst_parquet)
|
| 152 |
+
if len(src_df) != length or len(dst_df) != length:
|
| 153 |
+
failures.append(f"episode {new_ep}: length mismatch source={len(src_df)} dst={len(dst_df)} manifest={length}")
|
| 154 |
+
continue
|
| 155 |
+
if list(src_df.columns) != list(dst_df.columns):
|
| 156 |
+
failures.append(f"episode {new_ep}: column mismatch")
|
| 157 |
+
continue
|
| 158 |
+
|
| 159 |
+
for column in src_df.columns:
|
| 160 |
+
if column == "episode_index":
|
| 161 |
+
expected = np.full(length, new_ep, dtype=np.int64)
|
| 162 |
+
if not np.array_equal(dst_df[column].to_numpy(), expected):
|
| 163 |
+
failures.append(f"episode {new_ep}: episode_index remap invalid")
|
| 164 |
+
changed_column_counts[column] += 1
|
| 165 |
+
elif column == "frame_index":
|
| 166 |
+
expected = np.arange(length, dtype=np.int64)
|
| 167 |
+
if not np.array_equal(dst_df[column].to_numpy(), expected):
|
| 168 |
+
failures.append(f"episode {new_ep}: frame_index remap invalid")
|
| 169 |
+
changed_column_counts[column] += 1
|
| 170 |
+
elif column == "index":
|
| 171 |
+
expected = np.arange(global_start, global_start + length, dtype=np.int64)
|
| 172 |
+
if not np.array_equal(dst_df[column].to_numpy(), expected):
|
| 173 |
+
failures.append(f"episode {new_ep}: index remap invalid")
|
| 174 |
+
changed_column_counts[column] += 1
|
| 175 |
+
elif column in TASK_COLUMNS:
|
| 176 |
+
if column in src_df.columns:
|
| 177 |
+
for row_idx, (src_id, dst_id) in enumerate(zip(src_df[column].tolist(), dst_df[column].tolist())):
|
| 178 |
+
src_text = src_tasks[source_task][int(src_id)]
|
| 179 |
+
dst_text = dst_tasks[int(dst_id)]
|
| 180 |
+
if src_text != dst_text:
|
| 181 |
+
failures.append(
|
| 182 |
+
f"episode {new_ep}: {column} row {row_idx} task text mismatch {src_text!r} != {dst_text!r}"
|
| 183 |
+
)
|
| 184 |
+
break
|
| 185 |
+
changed_column_counts[column] += 1
|
| 186 |
+
else:
|
| 187 |
+
ok, reason = series_equal(src_df[column], dst_df[column])
|
| 188 |
+
if not ok:
|
| 189 |
+
failures.append(f"episode {new_ep}: data column {column} changed: {reason}")
|
| 190 |
+
break
|
| 191 |
+
|
| 192 |
+
for feature, dst_video in episode["videos"].items():
|
| 193 |
+
dst_path = Path(dst_video)
|
| 194 |
+
expected_source = Path(episode["resolved_augmented_video_sources"][feature]).resolve()
|
| 195 |
+
if not dst_path.exists():
|
| 196 |
+
failures.append(f"episode {new_ep}: missing output video {dst_path}")
|
| 197 |
+
continue
|
| 198 |
+
if not args.allow_copied_videos and not dst_path.is_symlink():
|
| 199 |
+
failures.append(f"episode {new_ep}: output video is not symlink: {dst_path}")
|
| 200 |
+
if dst_path.resolve() != expected_source:
|
| 201 |
+
failures.append(f"episode {new_ep}: video link mismatch {dst_path.resolve()} != {expected_source}")
|
| 202 |
+
if args.source_root.resolve() in dst_path.resolve().parents:
|
| 203 |
+
failures.append(f"episode {new_ep}: video resolves under source root: {dst_path.resolve()}")
|
| 204 |
+
if args.render_root.resolve() not in expected_source.parents:
|
| 205 |
+
failures.append(f"episode {new_ep}: augmented video source is outside render root: {expected_source}")
|
| 206 |
+
if args.check_video_frames:
|
| 207 |
+
frames = frame_count(dst_path.resolve())
|
| 208 |
+
if frames != length:
|
| 209 |
+
failures.append(f"episode {new_ep}: video frame count {frames} != parquet length {length} for {feature}")
|
| 210 |
+
frame_checks += 1
|
| 211 |
+
video_checks += 1
|
| 212 |
+
checked_rows += length
|
| 213 |
+
|
| 214 |
+
summary = {
|
| 215 |
+
"dataset": str(args.dataset),
|
| 216 |
+
"episodes_checked": len(manifest["episodes"]),
|
| 217 |
+
"rows_checked": checked_rows,
|
| 218 |
+
"video_links_checked": video_checks,
|
| 219 |
+
"video_frame_counts_checked": frame_checks,
|
| 220 |
+
"expected_changed_columns_checked": changed_column_counts,
|
| 221 |
+
"failures": failures[:50],
|
| 222 |
+
"failure_count": len(failures),
|
| 223 |
+
"passed": not failures,
|
| 224 |
+
}
|
| 225 |
+
out = args.dataset / "meta" / "geom_objwise_verification.json"
|
| 226 |
+
out.write_text(json.dumps(summary, indent=2) + "\n")
|
| 227 |
+
print(json.dumps(summary, indent=2))
|
| 228 |
+
if failures:
|
| 229 |
+
raise SystemExit(1)
|
| 230 |
+
|
| 231 |
+
|
| 232 |
+
if __name__ == "__main__":
|
| 233 |
+
main()
|
eval/eval_groot15_exact_replay.sh
ADDED
|
@@ -0,0 +1,389 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env bash
|
| 2 |
+
set -uEo pipefail
|
| 3 |
+
|
| 4 |
+
# Usually this is the training dataset name used to produce the checkpoint.
|
| 5 |
+
# It is used for default labels only; MODEL_PATH controls the checkpoint.
|
| 6 |
+
DATASETNAME="${DATASETNAME:-PickPlaceCounterToCabinet_150eps_augmented_groot15_v3_100_clean}"
|
| 7 |
+
|
| 8 |
+
MYGROOT_ROOT="${MYGROOT_ROOT:-/home/nvidia/jonghoon/myGR00T}"
|
| 9 |
+
MYGROOT_PY="${MYGROOT_PY:-/data/nvidia/gripper_augmentator/conda-envs/mygr00t/bin/python}"
|
| 10 |
+
ROBOCASA_ROOT="${ROBOCASA_ROOT:-/lp-dev/jonghoon/robocasa_calib/repos/robocasa}"
|
| 11 |
+
ROBOSUITE_ROOT="${ROBOSUITE_ROOT:-/lp-dev/jonghoon/robocasa_calib/repos/robosuite_robocasa_dev}"
|
| 12 |
+
ROBOCASA_PY="${ROBOCASA_PY:-/lp-dev/jonghoon/robocasa_calib/envs/robocasa/bin/python}"
|
| 13 |
+
EVAL_CLIENT="${EVAL_CLIENT:-$MYGROOT_ROOT/scripts/eval_robocasa_replay_state.py}"
|
| 14 |
+
|
| 15 |
+
OUTPUT_BASE="${OUTPUT_BASE:-/lp-dev/jonghoon/myGR00T_outputs}"
|
| 16 |
+
EVAL_OUTPUT_BASE="${EVAL_OUTPUT_BASE:-/lp-dev/jonghoon/isaac-gr00t/eval_results}"
|
| 17 |
+
CHECKPOINT_STEP="${CHECKPOINT_STEP:-30000}"
|
| 18 |
+
MODEL_RUN_NAME="${MODEL_RUN_NAME:-}"
|
| 19 |
+
if [[ -z "${MODEL_PATH:-}" ]]; then
|
| 20 |
+
if [[ -n "$MODEL_RUN_NAME" ]]; then
|
| 21 |
+
MODEL_PATH="${OUTPUT_BASE}/${MODEL_RUN_NAME}/checkpoint-${CHECKPOINT_STEP}"
|
| 22 |
+
else
|
| 23 |
+
MODEL_PATH="${OUTPUT_BASE}/pnpcountertocab_all502_gbs64_wandb_60k_save5k_20260427_193901/checkpoint-60000"
|
| 24 |
+
fi
|
| 25 |
+
fi
|
| 26 |
+
|
| 27 |
+
REPLAY_STATE_ROOT="${REPLAY_STATE_ROOT:-/lp-dev/jonghoon/isaac-gr00t/eval_results/pnpcountertocab_nojar_pseudo100_exact_replay_episode_set_20260430}"
|
| 28 |
+
EVAL_TASK="${EVAL_TASK:-PnPCounterToCab}"
|
| 29 |
+
ENV_NAME="${ENV_NAME:-PickPlaceCounterToCabinet}"
|
| 30 |
+
FINETUNED_TASK="${FINETUNED_TASK:-ft-${ENV_NAME}_${DATASETNAME}_ckpt${CHECKPOINT_STEP}}"
|
| 31 |
+
N_EPISODES="${N_EPISODES:-100}"
|
| 32 |
+
EXPECTED_REPLAY_STATES="${EXPECTED_REPLAY_STATES:-$N_EPISODES}"
|
| 33 |
+
SEED_BASE="${SEED_BASE:-42}"
|
| 34 |
+
MAX_EPISODE_STEPS="${MAX_EPISODE_STEPS:-750}"
|
| 35 |
+
ACTION_HORIZON="${ACTION_HORIZON:-16}"
|
| 36 |
+
GPUS="${GPUS:-0,1,2,3}"
|
| 37 |
+
PORT_BASE="${PORT_BASE:-8777}"
|
| 38 |
+
SAVE_VIDEOS="${SAVE_VIDEOS:-1}"
|
| 39 |
+
POLICY_SEED="${POLICY_SEED-42}"
|
| 40 |
+
RUN_ID="${RUN_ID:-$(date -u +%Y%m%d_%H%M%S)}"
|
| 41 |
+
EVAL_RUN_NAME="${EVAL_RUN_NAME:-mygr00t_${DATASETNAME}_ckpt${CHECKPOINT_STEP}_eval-${EVAL_TASK}_exact${N_EPISODES}_replay_${RUN_ID}}"
|
| 42 |
+
ROOT="${ROOT:-${EVAL_OUTPUT_BASE}/${EVAL_RUN_NAME}}"
|
| 43 |
+
MAIN_LOG="$ROOT/launcher.log"
|
| 44 |
+
|
| 45 |
+
SERVER_PIDS=()
|
| 46 |
+
CLIENT_PIDS=()
|
| 47 |
+
CLEANUP_RAN=0
|
| 48 |
+
|
| 49 |
+
ts() { date -u +"%Y-%m-%dT%H:%M:%SZ"; }
|
| 50 |
+
log() { echo "[$(ts)] $*" | tee -a "$MAIN_LOG"; }
|
| 51 |
+
|
| 52 |
+
cleanup() {
|
| 53 |
+
if [[ "$CLEANUP_RAN" -eq 1 ]]; then
|
| 54 |
+
return
|
| 55 |
+
fi
|
| 56 |
+
CLEANUP_RAN=1
|
| 57 |
+
local pid
|
| 58 |
+
for pid in "${CLIENT_PIDS[@]:-}"; do kill "$pid" 2>/dev/null || true; done
|
| 59 |
+
for pid in "${SERVER_PIDS[@]:-}"; do kill "$pid" 2>/dev/null || true; done
|
| 60 |
+
sleep 2
|
| 61 |
+
for pid in "${CLIENT_PIDS[@]:-}" "${SERVER_PIDS[@]:-}"; do kill -9 "$pid" 2>/dev/null || true; done
|
| 62 |
+
for pid in "${CLIENT_PIDS[@]:-}" "${SERVER_PIDS[@]:-}"; do wait "$pid" 2>/dev/null || true; done
|
| 63 |
+
}
|
| 64 |
+
trap cleanup EXIT
|
| 65 |
+
trap 'cleanup; exit 130' INT
|
| 66 |
+
trap 'cleanup; exit 143' TERM
|
| 67 |
+
|
| 68 |
+
pick_port() {
|
| 69 |
+
local port="$1"
|
| 70 |
+
while ss -ltn "sport = :$port" | grep -q LISTEN; do
|
| 71 |
+
port=$((port + 1))
|
| 72 |
+
done
|
| 73 |
+
echo "$port"
|
| 74 |
+
}
|
| 75 |
+
|
| 76 |
+
wait_for_port() {
|
| 77 |
+
local port="$1"
|
| 78 |
+
for _ in $(seq 1 240); do
|
| 79 |
+
if nc -z 127.0.0.1 "$port" 2>/dev/null; then
|
| 80 |
+
return 0
|
| 81 |
+
fi
|
| 82 |
+
sleep 5
|
| 83 |
+
done
|
| 84 |
+
return 1
|
| 85 |
+
}
|
| 86 |
+
|
| 87 |
+
mkdir -p "$ROOT/workers"
|
| 88 |
+
: > "$MAIN_LOG"
|
| 89 |
+
|
| 90 |
+
if [[ ! -f "$MODEL_PATH/model.safetensors.index.json" ]]; then
|
| 91 |
+
log "FATAL: checkpoint is missing or incomplete: $MODEL_PATH"
|
| 92 |
+
exit 1
|
| 93 |
+
fi
|
| 94 |
+
|
| 95 |
+
if ! python3 - "$REPLAY_STATE_ROOT" "$EXPECTED_REPLAY_STATES" <<'PY'
|
| 96 |
+
import json
|
| 97 |
+
import sys
|
| 98 |
+
from pathlib import Path
|
| 99 |
+
|
| 100 |
+
root = Path(sys.argv[1])
|
| 101 |
+
expected = int(sys.argv[2])
|
| 102 |
+
paths = sorted(root.glob("workers/worker_*/replay_states/episode_*_replay.json"))
|
| 103 |
+
paths.extend(sorted(root.glob("replay_states/episode_*_replay.json")))
|
| 104 |
+
rows = [json.loads(path.read_text()) for path in paths]
|
| 105 |
+
missing = [
|
| 106 |
+
r.get("episode_index")
|
| 107 |
+
for r in rows
|
| 108 |
+
if not Path(str(r.get("model_xml_gz", ""))).is_file()
|
| 109 |
+
or not Path(str(r.get("state_npz", ""))).is_file()
|
| 110 |
+
]
|
| 111 |
+
raise SystemExit(0 if len(rows) == expected and not missing else 1)
|
| 112 |
+
PY
|
| 113 |
+
then
|
| 114 |
+
log "FATAL: replay source must have $EXPECTED_REPLAY_STATES replay jsons with model XML and state NPZ artifacts: $REPLAY_STATE_ROOT"
|
| 115 |
+
exit 1
|
| 116 |
+
fi
|
| 117 |
+
|
| 118 |
+
IFS=',' read -r -a GPU_LIST <<< "$GPUS"
|
| 119 |
+
N_WORKERS="${#GPU_LIST[@]}"
|
| 120 |
+
if [[ "$N_WORKERS" -lt 1 ]]; then
|
| 121 |
+
log "FATAL: no GPUs specified"
|
| 122 |
+
exit 1
|
| 123 |
+
fi
|
| 124 |
+
|
| 125 |
+
BASE_COUNT=$((N_EPISODES / N_WORKERS))
|
| 126 |
+
REMAINDER=$((N_EPISODES % N_WORKERS))
|
| 127 |
+
|
| 128 |
+
export PYTHONPATH="$MYGROOT_ROOT:${PYTHONPATH:-}"
|
| 129 |
+
CLIENT_PYTHONPATH="$ROBOSUITE_ROOT:$ROBOCASA_ROOT:$MYGROOT_ROOT:${PYTHONPATH:-}"
|
| 130 |
+
export MUJOCO_GL=egl
|
| 131 |
+
export PYOPENGL_PLATFORM=egl
|
| 132 |
+
|
| 133 |
+
python3 - "$ROOT/run_info.json" <<PY
|
| 134 |
+
import json, sys, time
|
| 135 |
+
info = {
|
| 136 |
+
"dataset_name": "$DATASETNAME",
|
| 137 |
+
"model_path": "$MODEL_PATH",
|
| 138 |
+
"finetuned_task": "$FINETUNED_TASK",
|
| 139 |
+
"replay_state_root": "$REPLAY_STATE_ROOT",
|
| 140 |
+
"eval_task": "$EVAL_TASK",
|
| 141 |
+
"env_name": "$ENV_NAME",
|
| 142 |
+
"n_episodes_requested": int("$N_EPISODES"),
|
| 143 |
+
"expected_replay_states": int("$EXPECTED_REPLAY_STATES"),
|
| 144 |
+
"seed_base": int("$SEED_BASE"),
|
| 145 |
+
"gpus": "$GPUS".split(","),
|
| 146 |
+
"n_workers": int("$N_WORKERS"),
|
| 147 |
+
"max_episode_steps": int("$MAX_EPISODE_STEPS"),
|
| 148 |
+
"action_horizon": int("$ACTION_HORIZON"),
|
| 149 |
+
"obj_instance_split": "target",
|
| 150 |
+
"legacy_object_split_label": "B",
|
| 151 |
+
"layout_and_style_ids": [[i, i] for i in range(1, 11)],
|
| 152 |
+
"save_videos": bool(int("$SAVE_VIDEOS")),
|
| 153 |
+
"policy_seed": None if "$POLICY_SEED" == "" else int("$POLICY_SEED"),
|
| 154 |
+
"canonical_episode_definition": "episode_index restores model XML and flattened MuJoCo state from replay_state_root",
|
| 155 |
+
"replay_mode": "exact_model_xml_plus_mujoco_state",
|
| 156 |
+
"exact_replay_artifacts": "source workers/worker_*/replay_states/episode_*.xml.gz plus *_state.npz",
|
| 157 |
+
"codebase": "myGR00T",
|
| 158 |
+
"data_config": "single_panda_gripper",
|
| 159 |
+
"embodiment_tag": "new_embodiment",
|
| 160 |
+
"started_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
|
| 161 |
+
}
|
| 162 |
+
open(sys.argv[1], "w").write(json.dumps(info, indent=2, sort_keys=True) + "\\n")
|
| 163 |
+
PY
|
| 164 |
+
|
| 165 |
+
MANIFEST="$ROOT/seed_shard_manifest.jsonl"
|
| 166 |
+
: > "$MANIFEST"
|
| 167 |
+
|
| 168 |
+
log "launching exact-state replay eval"
|
| 169 |
+
log "root=$ROOT"
|
| 170 |
+
log "dataset_name=$DATASETNAME"
|
| 171 |
+
log "model=$MODEL_PATH"
|
| 172 |
+
log "replay_state_root=$REPLAY_STATE_ROOT"
|
| 173 |
+
log "episodes=$N_EPISODES workers=$N_WORKERS gpus=$GPUS seed_base=$SEED_BASE"
|
| 174 |
+
|
| 175 |
+
POLICY_SERVER_SEED_ARGS=()
|
| 176 |
+
POLICY_CLIENT_SEED_ARGS=()
|
| 177 |
+
if [[ -n "$POLICY_SEED" ]]; then
|
| 178 |
+
POLICY_SERVER_SEED_ARGS+=(--seed "$POLICY_SEED")
|
| 179 |
+
POLICY_CLIENT_SEED_ARGS+=(--policy_seed "$POLICY_SEED")
|
| 180 |
+
fi
|
| 181 |
+
|
| 182 |
+
offset=0
|
| 183 |
+
port="$PORT_BASE"
|
| 184 |
+
WORKER_DIRS=()
|
| 185 |
+
WORKER_COUNTS=()
|
| 186 |
+
WORKER_OFFSETS=()
|
| 187 |
+
WORKER_SEEDS=()
|
| 188 |
+
WORKER_PORTS=()
|
| 189 |
+
|
| 190 |
+
for worker_id in "${!GPU_LIST[@]}"; do
|
| 191 |
+
count="$BASE_COUNT"
|
| 192 |
+
if [[ "$worker_id" -lt "$REMAINDER" ]]; then
|
| 193 |
+
count=$((count + 1))
|
| 194 |
+
fi
|
| 195 |
+
seed=$((SEED_BASE + worker_id))
|
| 196 |
+
gpu="${GPU_LIST[$worker_id]}"
|
| 197 |
+
worker_dir="$ROOT/workers/worker_$(printf '%02d' "$worker_id")_seed_${seed}"
|
| 198 |
+
mkdir -p "$worker_dir"
|
| 199 |
+
for local_idx in $(seq 0 $((count - 1))); do
|
| 200 |
+
printf '{"episode_index":%d,"worker_id":"%02d","episode_seed":%d,"local_episode_index":%d}\n' \
|
| 201 |
+
"$((offset + local_idx))" "$worker_id" "$seed" "$local_idx" >> "$MANIFEST"
|
| 202 |
+
done
|
| 203 |
+
|
| 204 |
+
port="$(pick_port "$port")"
|
| 205 |
+
WORKER_DIRS+=("$worker_dir")
|
| 206 |
+
WORKER_COUNTS+=("$count")
|
| 207 |
+
WORKER_OFFSETS+=("$offset")
|
| 208 |
+
WORKER_SEEDS+=("$seed")
|
| 209 |
+
WORKER_PORTS+=("$port")
|
| 210 |
+
|
| 211 |
+
log "starting server worker=$worker_id gpu=$gpu seed=$seed episodes=$count offset=$offset port=$port"
|
| 212 |
+
(
|
| 213 |
+
cd "$MYGROOT_ROOT"
|
| 214 |
+
OMP_NUM_THREADS=4 MKL_NUM_THREADS=4 OPENBLAS_NUM_THREADS=4 \
|
| 215 |
+
CUDA_VISIBLE_DEVICES="$gpu" "$MYGROOT_PY" scripts/inference_service.py \
|
| 216 |
+
--server \
|
| 217 |
+
--port "$port" \
|
| 218 |
+
--model_path "$MODEL_PATH" \
|
| 219 |
+
--data_config single_panda_gripper \
|
| 220 |
+
--embodiment_tag new_embodiment \
|
| 221 |
+
"${POLICY_SERVER_SEED_ARGS[@]}"
|
| 222 |
+
) > "$worker_dir/server.log" 2>&1 &
|
| 223 |
+
SERVER_PIDS+=("$!")
|
| 224 |
+
echo "${SERVER_PIDS[-1]}" > "$worker_dir/server.pid"
|
| 225 |
+
|
| 226 |
+
offset=$((offset + count))
|
| 227 |
+
port=$((port + 1))
|
| 228 |
+
done
|
| 229 |
+
|
| 230 |
+
for worker_id in "${!WORKER_PORTS[@]}"; do
|
| 231 |
+
port="${WORKER_PORTS[$worker_id]}"
|
| 232 |
+
worker_dir="${WORKER_DIRS[$worker_id]}"
|
| 233 |
+
if ! wait_for_port "$port"; then
|
| 234 |
+
log "FATAL: policy server for worker=$worker_id did not open port=$port"
|
| 235 |
+
tail -n 80 "$worker_dir/server.log" | tee -a "$MAIN_LOG" || true
|
| 236 |
+
cleanup
|
| 237 |
+
exit 1
|
| 238 |
+
fi
|
| 239 |
+
log "server ready worker=$worker_id port=$port"
|
| 240 |
+
done
|
| 241 |
+
|
| 242 |
+
for worker_id in "${!GPU_LIST[@]}"; do
|
| 243 |
+
gpu="${GPU_LIST[$worker_id]}"
|
| 244 |
+
port="${WORKER_PORTS[$worker_id]}"
|
| 245 |
+
worker_dir="${WORKER_DIRS[$worker_id]}"
|
| 246 |
+
count="${WORKER_COUNTS[$worker_id]}"
|
| 247 |
+
offset="${WORKER_OFFSETS[$worker_id]}"
|
| 248 |
+
seed="${WORKER_SEEDS[$worker_id]}"
|
| 249 |
+
SAVE_VIDEO_ARGS=()
|
| 250 |
+
if [[ "$SAVE_VIDEOS" -eq 1 ]]; then
|
| 251 |
+
SAVE_VIDEO_ARGS+=(--save_videos)
|
| 252 |
+
fi
|
| 253 |
+
log "starting client worker=$worker_id gpu=$gpu seed=$seed episodes=$count offset=$offset port=$port"
|
| 254 |
+
PYTHONPATH="$CLIENT_PYTHONPATH" CUDA_VISIBLE_DEVICES="$gpu" "$ROBOCASA_PY" "$EVAL_CLIENT" \
|
| 255 |
+
--host 127.0.0.1 \
|
| 256 |
+
--port "$port" \
|
| 257 |
+
--model_path "$MODEL_PATH" \
|
| 258 |
+
--output_dir "$worker_dir" \
|
| 259 |
+
--env_name "$ENV_NAME" \
|
| 260 |
+
--eval_task "$EVAL_TASK" \
|
| 261 |
+
--finetuned_task "$FINETUNED_TASK" \
|
| 262 |
+
--replay_state_root "$REPLAY_STATE_ROOT" \
|
| 263 |
+
--n_episodes "$count" \
|
| 264 |
+
--episode_offset "$offset" \
|
| 265 |
+
--worker_id "$(printf '%02d' "$worker_id")" \
|
| 266 |
+
--seed "$seed" \
|
| 267 |
+
--max_episode_steps "$MAX_EPISODE_STEPS" \
|
| 268 |
+
--action_horizon "$ACTION_HORIZON" \
|
| 269 |
+
--robots PandaOmron \
|
| 270 |
+
--generative_textures \
|
| 271 |
+
"${POLICY_CLIENT_SEED_ARGS[@]}" \
|
| 272 |
+
"${SAVE_VIDEO_ARGS[@]}" \
|
| 273 |
+
> "$worker_dir/client.log" 2>&1 &
|
| 274 |
+
CLIENT_PIDS+=("$!")
|
| 275 |
+
echo "${CLIENT_PIDS[-1]}" > "$worker_dir/client.pid"
|
| 276 |
+
done
|
| 277 |
+
|
| 278 |
+
client_rc=0
|
| 279 |
+
for worker_id in "${!CLIENT_PIDS[@]}"; do
|
| 280 |
+
pid="${CLIENT_PIDS[$worker_id]}"
|
| 281 |
+
if wait "$pid"; then
|
| 282 |
+
log "client finished worker=$worker_id rc=0"
|
| 283 |
+
else
|
| 284 |
+
rc="$?"
|
| 285 |
+
log "client failed worker=$worker_id rc=$rc"
|
| 286 |
+
client_rc="$rc"
|
| 287 |
+
fi
|
| 288 |
+
done
|
| 289 |
+
|
| 290 |
+
cleanup
|
| 291 |
+
|
| 292 |
+
if [[ "$client_rc" -ne 0 ]]; then
|
| 293 |
+
log "FATAL: at least one worker failed"
|
| 294 |
+
exit "$client_rc"
|
| 295 |
+
fi
|
| 296 |
+
|
| 297 |
+
python3 - "$ROOT" <<'PY'
|
| 298 |
+
import csv
|
| 299 |
+
import json
|
| 300 |
+
import sys
|
| 301 |
+
import time
|
| 302 |
+
from pathlib import Path
|
| 303 |
+
|
| 304 |
+
root = Path(sys.argv[1])
|
| 305 |
+
workers = sorted((root / "workers").glob("worker_*"))
|
| 306 |
+
|
| 307 |
+
def load_jsonl(path):
|
| 308 |
+
rows = []
|
| 309 |
+
if path.exists():
|
| 310 |
+
with path.open() as f:
|
| 311 |
+
for line in f:
|
| 312 |
+
line = line.strip()
|
| 313 |
+
if line:
|
| 314 |
+
rows.append(json.loads(line))
|
| 315 |
+
return rows
|
| 316 |
+
|
| 317 |
+
episodes = []
|
| 318 |
+
full_meta = []
|
| 319 |
+
for worker in workers:
|
| 320 |
+
episodes.extend(load_jsonl(worker / "episodes.jsonl"))
|
| 321 |
+
full_meta.extend(load_jsonl(worker / "episodes_full_meta.jsonl"))
|
| 322 |
+
|
| 323 |
+
episodes.sort(key=lambda r: int(r["episode_index"]))
|
| 324 |
+
full_meta.sort(key=lambda r: int(r["episode_index"]))
|
| 325 |
+
|
| 326 |
+
with (root / "episodes.jsonl").open("w") as f:
|
| 327 |
+
for row in episodes:
|
| 328 |
+
f.write(json.dumps(row, sort_keys=True) + "\n")
|
| 329 |
+
|
| 330 |
+
with (root / "episodes_full_meta.jsonl").open("w") as f:
|
| 331 |
+
for row in full_meta:
|
| 332 |
+
f.write(json.dumps(row, sort_keys=True) + "\n")
|
| 333 |
+
|
| 334 |
+
fieldnames = [
|
| 335 |
+
"episode_index",
|
| 336 |
+
"local_episode_index",
|
| 337 |
+
"worker_id",
|
| 338 |
+
"episode_seed",
|
| 339 |
+
"success",
|
| 340 |
+
"reward",
|
| 341 |
+
"policy_steps",
|
| 342 |
+
"sim_step_budget",
|
| 343 |
+
"video_path",
|
| 344 |
+
"layout_id",
|
| 345 |
+
"style_id",
|
| 346 |
+
"language",
|
| 347 |
+
"target_mesh_keys",
|
| 348 |
+
]
|
| 349 |
+
with (root / "simulation_results.csv").open("w", newline="") as f:
|
| 350 |
+
writer = csv.DictWriter(f, fieldnames=fieldnames)
|
| 351 |
+
writer.writeheader()
|
| 352 |
+
for row in episodes:
|
| 353 |
+
writer.writerow(
|
| 354 |
+
{
|
| 355 |
+
"episode_index": row.get("episode_index"),
|
| 356 |
+
"local_episode_index": row.get("local_episode_index"),
|
| 357 |
+
"worker_id": row.get("worker_id"),
|
| 358 |
+
"episode_seed": row.get("episode_seed"),
|
| 359 |
+
"success": int(bool(row.get("success"))),
|
| 360 |
+
"reward": row.get("reward"),
|
| 361 |
+
"policy_steps": row.get("policy_steps"),
|
| 362 |
+
"sim_step_budget": row.get("sim_step_budget"),
|
| 363 |
+
"video_path": row.get("video_path", ""),
|
| 364 |
+
"layout_id": row.get("layout_id"),
|
| 365 |
+
"style_id": row.get("style_id"),
|
| 366 |
+
"language": row.get("language"),
|
| 367 |
+
"target_mesh_keys": ";".join(
|
| 368 |
+
str(obj.get("mesh_key"))
|
| 369 |
+
for obj in row.get("objects", [])
|
| 370 |
+
if obj.get("mesh_key")
|
| 371 |
+
),
|
| 372 |
+
}
|
| 373 |
+
)
|
| 374 |
+
|
| 375 |
+
run_info = json.loads((root / "run_info.json").read_text())
|
| 376 |
+
n_success = sum(1 for row in episodes if row.get("success"))
|
| 377 |
+
summary = {
|
| 378 |
+
**run_info,
|
| 379 |
+
"finished_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
|
| 380 |
+
"n_episodes": len(episodes),
|
| 381 |
+
"n_success": int(n_success),
|
| 382 |
+
"success_rate": float(n_success / max(len(episodes), 1)),
|
| 383 |
+
}
|
| 384 |
+
(root / "summary.json").write_text(json.dumps(summary, indent=2, sort_keys=True) + "\n")
|
| 385 |
+
print(json.dumps(summary, indent=2, sort_keys=True))
|
| 386 |
+
PY
|
| 387 |
+
|
| 388 |
+
log "merged worker outputs"
|
| 389 |
+
log "done root=$ROOT"
|
eval/eval_robocasa_replay_state_3view.py
ADDED
|
@@ -0,0 +1,590 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
"""Run RoboCasa eval from saved model XML + MuJoCo state artifacts.
|
| 3 |
+
|
| 4 |
+
This is the strict replay path for apples-to-apples evaluation after a
|
| 5 |
+
reconstructed episode bank has been created by `eval_robocasa_reconstruct_xyz.py`.
|
| 6 |
+
Each episode is restored from:
|
| 7 |
+
|
| 8 |
+
- `replay_states/episode_XXXXXX.xml.gz`
|
| 9 |
+
- `replay_states/episode_XXXXXX_state.npz`
|
| 10 |
+
- `replay_states/episode_XXXXXX_replay.json`
|
| 11 |
+
"""
|
| 12 |
+
|
| 13 |
+
from __future__ import annotations
|
| 14 |
+
|
| 15 |
+
import argparse
|
| 16 |
+
import csv
|
| 17 |
+
import gzip
|
| 18 |
+
import json
|
| 19 |
+
import pickle
|
| 20 |
+
import time
|
| 21 |
+
from io import BytesIO
|
| 22 |
+
from pathlib import Path
|
| 23 |
+
from typing import Any
|
| 24 |
+
|
| 25 |
+
import gymnasium as gym
|
| 26 |
+
import numpy as np
|
| 27 |
+
import robocasa # noqa: F401 - registers RoboCasa environments with robosuite
|
| 28 |
+
import torch
|
| 29 |
+
import zmq
|
| 30 |
+
from tqdm import trange
|
| 31 |
+
|
| 32 |
+
from gr00t.eval.wrappers.multistep_wrapper import MultiStepWrapper
|
| 33 |
+
from gr00t.eval.wrappers.record_video import RecordVideo
|
| 34 |
+
from gr00t.eval.wrappers.robocasa_wrapper import RoboCasaWrapper, load_robocasa_gym_env
|
| 35 |
+
|
| 36 |
+
TARGET_LAYOUT_AND_STYLE_IDS = [[i, i] for i in range(1, 11)]
|
| 37 |
+
TARGET_OBJ_INSTANCE_SPLIT = "target"
|
| 38 |
+
ACTION_SEED_KEY = "_policy_action_seed"
|
| 39 |
+
ACTION_SEED_EPISODE_STRIDE = 1_000_003
|
| 40 |
+
TORCH_SEED_MODULUS = 2**63 - 1
|
| 41 |
+
|
| 42 |
+
|
| 43 |
+
class PolicyClient:
|
| 44 |
+
"""Minimal client for myGR00T's RobotInferenceServer."""
|
| 45 |
+
|
| 46 |
+
def __init__(self, host: str, port: int):
|
| 47 |
+
self.context = zmq.Context()
|
| 48 |
+
self.socket = self.context.socket(zmq.REQ)
|
| 49 |
+
self.socket.connect(f"tcp://{host}:{port}")
|
| 50 |
+
|
| 51 |
+
@staticmethod
|
| 52 |
+
def to_bytes(data: dict[str, Any]) -> bytes:
|
| 53 |
+
buffer = BytesIO()
|
| 54 |
+
torch.save(data, buffer)
|
| 55 |
+
return buffer.getvalue()
|
| 56 |
+
|
| 57 |
+
@staticmethod
|
| 58 |
+
def from_bytes(data: bytes) -> dict[str, Any]:
|
| 59 |
+
buffer = BytesIO(data)
|
| 60 |
+
return torch.load(buffer, weights_only=False)
|
| 61 |
+
|
| 62 |
+
def call(self, endpoint: str, data: dict[str, Any] | None = None) -> dict[str, Any]:
|
| 63 |
+
request: dict[str, Any] = {"endpoint": endpoint}
|
| 64 |
+
if data is not None:
|
| 65 |
+
request["data"] = data
|
| 66 |
+
self.socket.send(self.to_bytes(request))
|
| 67 |
+
message = self.socket.recv()
|
| 68 |
+
if message == b"ERROR":
|
| 69 |
+
raise RuntimeError("policy server returned ERROR")
|
| 70 |
+
return self.from_bytes(message)
|
| 71 |
+
|
| 72 |
+
def get_action(self, observations: dict[str, Any]) -> dict[str, Any]:
|
| 73 |
+
return self.call("get_action", observations)
|
| 74 |
+
|
| 75 |
+
def close(self) -> None:
|
| 76 |
+
self.socket.close()
|
| 77 |
+
self.context.term()
|
| 78 |
+
|
| 79 |
+
|
| 80 |
+
def as_bool(value: Any) -> bool:
|
| 81 |
+
if isinstance(value, np.ndarray):
|
| 82 |
+
return bool(np.any(value))
|
| 83 |
+
if isinstance(value, (list, tuple)):
|
| 84 |
+
return any(as_bool(v) for v in value)
|
| 85 |
+
if isinstance(value, dict):
|
| 86 |
+
return any(as_bool(v) for v in value.values())
|
| 87 |
+
return bool(value)
|
| 88 |
+
|
| 89 |
+
|
| 90 |
+
def json_default(value: Any) -> Any:
|
| 91 |
+
if isinstance(value, np.ndarray):
|
| 92 |
+
return value.tolist()
|
| 93 |
+
if isinstance(value, np.generic):
|
| 94 |
+
return value.item()
|
| 95 |
+
if isinstance(value, Path):
|
| 96 |
+
return str(value)
|
| 97 |
+
return str(value)
|
| 98 |
+
|
| 99 |
+
|
| 100 |
+
def find_robocasa_inner(env: Any) -> Any | None:
|
| 101 |
+
cur = env
|
| 102 |
+
for _ in range(80):
|
| 103 |
+
if cur is None:
|
| 104 |
+
return None
|
| 105 |
+
nxt = getattr(cur, "env", None)
|
| 106 |
+
if nxt is None or nxt is cur:
|
| 107 |
+
break
|
| 108 |
+
cur = nxt
|
| 109 |
+
if hasattr(cur, "sim") and hasattr(cur, "get_ep_meta"):
|
| 110 |
+
return cur
|
| 111 |
+
return None
|
| 112 |
+
|
| 113 |
+
|
| 114 |
+
def parse_mjcf_path(mjcf_path: str | None) -> dict[str, str | None]:
|
| 115 |
+
if not mjcf_path:
|
| 116 |
+
return {"registry": None, "instance_id": None, "mesh_key": None}
|
| 117 |
+
path = Path(str(mjcf_path))
|
| 118 |
+
parts = path.parts
|
| 119 |
+
registry = None
|
| 120 |
+
for candidate in ("objaverse", "aigen_objs", "lightwheel"):
|
| 121 |
+
if candidate in parts:
|
| 122 |
+
registry = candidate
|
| 123 |
+
break
|
| 124 |
+
instance_id = path.parent.name if path.name.endswith(".xml") else None
|
| 125 |
+
mesh_key = None
|
| 126 |
+
if registry and registry in parts and instance_id:
|
| 127 |
+
idx = parts.index(registry)
|
| 128 |
+
if registry == "lightwheel" and idx + 1 < len(parts):
|
| 129 |
+
mesh_key = "/".join((parts[idx + 1], instance_id))
|
| 130 |
+
elif idx + 1 < len(parts):
|
| 131 |
+
mesh_key = "/".join(parts[idx + 1 : -1])
|
| 132 |
+
return {"registry": registry, "instance_id": instance_id, "mesh_key": mesh_key}
|
| 133 |
+
|
| 134 |
+
|
| 135 |
+
def summarize_ep_meta(ep_meta: dict[str, Any]) -> dict[str, Any]:
|
| 136 |
+
summary: dict[str, Any] = {
|
| 137 |
+
"layout_id": ep_meta.get("layout_id"),
|
| 138 |
+
"style_id": ep_meta.get("style_id"),
|
| 139 |
+
"language": ep_meta.get("lang", ""),
|
| 140 |
+
"objects": [],
|
| 141 |
+
}
|
| 142 |
+
for cfg in ep_meta.get("object_cfgs", []) or []:
|
| 143 |
+
if not isinstance(cfg, dict):
|
| 144 |
+
continue
|
| 145 |
+
info = cfg.get("info", {}) if isinstance(cfg.get("info", {}), dict) else {}
|
| 146 |
+
mjcf_path = info.get("mjcf_path")
|
| 147 |
+
parsed = parse_mjcf_path(mjcf_path)
|
| 148 |
+
summary["objects"].append(
|
| 149 |
+
{
|
| 150 |
+
"name": cfg.get("name"),
|
| 151 |
+
"category": info.get("cat"),
|
| 152 |
+
"mjcf_path": mjcf_path,
|
| 153 |
+
"registry": parsed["registry"],
|
| 154 |
+
"instance_id": parsed["instance_id"],
|
| 155 |
+
"mesh_key": parsed["mesh_key"],
|
| 156 |
+
}
|
| 157 |
+
)
|
| 158 |
+
fixtures = ep_meta.get("fixtures")
|
| 159 |
+
if isinstance(fixtures, dict):
|
| 160 |
+
summary["fixture_names"] = list(fixtures.keys())
|
| 161 |
+
return summary
|
| 162 |
+
|
| 163 |
+
|
| 164 |
+
def capture_ep_meta(env: Any, pickle_path: Path) -> tuple[dict[str, Any], dict[str, Any]]:
|
| 165 |
+
inner = find_robocasa_inner(env)
|
| 166 |
+
if inner is None:
|
| 167 |
+
return {}, {"error": "could_not_find_robocasa_inner_env"}
|
| 168 |
+
try:
|
| 169 |
+
ep_meta = inner.get_ep_meta() or {}
|
| 170 |
+
except Exception as exc:
|
| 171 |
+
return {}, {"error": f"get_ep_meta_failed: {type(exc).__name__}: {exc}"}
|
| 172 |
+
with pickle_path.open("wb") as f:
|
| 173 |
+
pickle.dump(ep_meta, f)
|
| 174 |
+
return ep_meta, summarize_ep_meta(ep_meta)
|
| 175 |
+
|
| 176 |
+
|
| 177 |
+
def write_jsonl(path: Path, row: dict[str, Any]) -> None:
|
| 178 |
+
with path.open("a") as f:
|
| 179 |
+
f.write(json.dumps(row, default=json_default, sort_keys=True) + "\n")
|
| 180 |
+
f.flush()
|
| 181 |
+
|
| 182 |
+
|
| 183 |
+
def make_policy_action_seed(policy_seed: int, episode_index: int, policy_step: int) -> int:
|
| 184 |
+
seed = int(policy_seed) + int(episode_index) * ACTION_SEED_EPISODE_STRIDE + int(policy_step)
|
| 185 |
+
return seed % TORCH_SEED_MODULUS
|
| 186 |
+
|
| 187 |
+
|
| 188 |
+
def resolve_artifact_path(value: str, state_json_path: Path) -> Path:
|
| 189 |
+
path = Path(value)
|
| 190 |
+
if path.is_file():
|
| 191 |
+
return path
|
| 192 |
+
candidate = state_json_path.parent / path.name
|
| 193 |
+
if candidate.is_file():
|
| 194 |
+
return candidate
|
| 195 |
+
raise FileNotFoundError(f"missing replay artifact: {value}")
|
| 196 |
+
|
| 197 |
+
|
| 198 |
+
def load_replay_state_index(root: Path) -> dict[int, dict[str, Any]]:
|
| 199 |
+
if not root.is_dir():
|
| 200 |
+
raise FileNotFoundError(f"missing replay_state_root: {root}")
|
| 201 |
+
paths = sorted(root.glob("workers/worker_*/replay_states/episode_*_replay.json"))
|
| 202 |
+
paths.extend(sorted(root.glob("replay_states/episode_*_replay.json")))
|
| 203 |
+
index: dict[int, dict[str, Any]] = {}
|
| 204 |
+
for path in paths:
|
| 205 |
+
row = json.loads(path.read_text())
|
| 206 |
+
episode_index = int(row["episode_index"])
|
| 207 |
+
row["replay_json"] = str(path)
|
| 208 |
+
row["model_xml_gz"] = str(resolve_artifact_path(str(row["model_xml_gz"]), path))
|
| 209 |
+
row["state_npz"] = str(resolve_artifact_path(str(row["state_npz"]), path))
|
| 210 |
+
index[episode_index] = row
|
| 211 |
+
if not index:
|
| 212 |
+
raise FileNotFoundError(f"no replay state jsons found under {root}")
|
| 213 |
+
return index
|
| 214 |
+
|
| 215 |
+
|
| 216 |
+
def load_ep_meta_from_state_info(state_info: dict[str, Any]) -> dict[str, Any] | None:
|
| 217 |
+
for key in ("regenerated_ep_meta_pickle", "source_ep_meta_pickle", "ep_meta_pickle"):
|
| 218 |
+
path_value = state_info.get(key)
|
| 219 |
+
if not path_value:
|
| 220 |
+
continue
|
| 221 |
+
path = Path(str(path_value))
|
| 222 |
+
if not path.is_file():
|
| 223 |
+
continue
|
| 224 |
+
with path.open("rb") as f:
|
| 225 |
+
return pickle.load(f)
|
| 226 |
+
return None
|
| 227 |
+
|
| 228 |
+
|
| 229 |
+
def build_fresh_groot_obs(robo_wrapper: RoboCasaWrapper, inner: Any) -> dict[str, Any]:
|
| 230 |
+
gym_env = robo_wrapper.env
|
| 231 |
+
if getattr(inner, "viewer_get_obs", False) and getattr(inner, "viewer", None) is not None:
|
| 232 |
+
raw_obs = inner.viewer._get_observations(force_update=True)
|
| 233 |
+
else:
|
| 234 |
+
raw_obs = inner._get_observations(force_update=True)
|
| 235 |
+
if getattr(gym_env, "flatten_obs", False):
|
| 236 |
+
obs = gym_env._flatten_obs(raw_obs)
|
| 237 |
+
else:
|
| 238 |
+
obs = gym_env._filter_obs(raw_obs)
|
| 239 |
+
new_obs: dict[str, Any] = {}
|
| 240 |
+
for key, value in obs.items():
|
| 241 |
+
if key in robo_wrapper._robocasa_keys_to_gr00t_keys:
|
| 242 |
+
new_obs[robo_wrapper._robocasa_keys_to_gr00t_keys[key]] = value
|
| 243 |
+
new_obs["annotation.human.action.task_description"] = [robo_wrapper.language_instruction]
|
| 244 |
+
return new_obs
|
| 245 |
+
|
| 246 |
+
|
| 247 |
+
class ExactStateReplayWrapper(gym.Wrapper):
|
| 248 |
+
def __init__(self, env: RoboCasaWrapper, replay_state_index: dict[int, dict[str, Any]]):
|
| 249 |
+
super().__init__(env)
|
| 250 |
+
self.replay_state_index = replay_state_index
|
| 251 |
+
self.current_episode_index: int | None = None
|
| 252 |
+
self.last_state_info: dict[str, Any] = {}
|
| 253 |
+
|
| 254 |
+
def set_episode_index(self, episode_index: int) -> None:
|
| 255 |
+
self.current_episode_index = int(episode_index)
|
| 256 |
+
|
| 257 |
+
def reset(self, seed=None, options=None):
|
| 258 |
+
if self.current_episode_index is None:
|
| 259 |
+
raise RuntimeError("set_episode_index must be called before reset")
|
| 260 |
+
if self.current_episode_index not in self.replay_state_index:
|
| 261 |
+
raise KeyError(f"missing replay state for episode_index={self.current_episode_index}")
|
| 262 |
+
state_info = self.replay_state_index[self.current_episode_index]
|
| 263 |
+
inner = find_robocasa_inner(self.env)
|
| 264 |
+
if inner is None or not hasattr(inner, "reset_from_xml_string"):
|
| 265 |
+
raise RuntimeError("could not find RoboCasa inner env with reset_from_xml_string")
|
| 266 |
+
|
| 267 |
+
ep_meta = load_ep_meta_from_state_info(state_info)
|
| 268 |
+
if ep_meta is not None and hasattr(inner, "set_ep_meta"):
|
| 269 |
+
inner.set_ep_meta(ep_meta)
|
| 270 |
+
|
| 271 |
+
with gzip.open(Path(state_info["model_xml_gz"]), "rt") as f:
|
| 272 |
+
model_xml = f.read()
|
| 273 |
+
saved = np.load(Path(state_info["state_npz"]))
|
| 274 |
+
saved_state = saved["states"]
|
| 275 |
+
saved_ctrl = saved["ctrl"] if "ctrl" in saved.files else None
|
| 276 |
+
|
| 277 |
+
inner.reset()
|
| 278 |
+
if ep_meta is not None and hasattr(inner, "set_ep_meta"):
|
| 279 |
+
inner.set_ep_meta(ep_meta)
|
| 280 |
+
inner.reset_from_xml_string(model_xml)
|
| 281 |
+
inner.sim.reset()
|
| 282 |
+
inner.sim.set_state_from_flattened(saved_state)
|
| 283 |
+
if saved_ctrl is not None and saved_ctrl.shape == inner.sim.data.ctrl.shape:
|
| 284 |
+
inner.sim.data.ctrl[:] = saved_ctrl
|
| 285 |
+
inner.sim.forward()
|
| 286 |
+
|
| 287 |
+
# --- FIX B: re-sync each part controller to the RESTORED state -------------
|
| 288 |
+
# reset_from_xml_string() above triggers a throwaway env reset, during which
|
| 289 |
+
# every controller latches initial_joint (used for the OSC nullspace bias) and
|
| 290 |
+
# goal_pos from the robot's freshly *randomised* init pose (robot.py: init_qpos
|
| 291 |
+
# += gaussian noise, magnitude 0.02). set_state_from_flattened() then teleports
|
| 292 |
+
# the robot to the episode's saved pose, but nothing re-syncs the controllers,
|
| 293 |
+
# so they keep biasing toward a pose this episode never had -- and that stale
|
| 294 |
+
# pose depends on the client --seed, which leaks worker sharding into results.
|
| 295 |
+
# Re-seed each controller from the restored qpos instead.
|
| 296 |
+
for _robot in getattr(inner, "robots", []) or []:
|
| 297 |
+
_cc = getattr(_robot, "composite_controller", None)
|
| 298 |
+
if _cc is None:
|
| 299 |
+
continue
|
| 300 |
+
for _ctrl in getattr(_cc, "part_controllers", {}).values():
|
| 301 |
+
_qidx = getattr(_ctrl, "qpos_index", None)
|
| 302 |
+
if _qidx is None or getattr(_ctrl, "ref_name", None) is None:
|
| 303 |
+
continue # e.g. MobileBaseJointVelocityController has no ref_pos
|
| 304 |
+
try:
|
| 305 |
+
_ctrl.update_initial_joints(inner.sim.data.qpos[_qidx])
|
| 306 |
+
except Exception as _exc:
|
| 307 |
+
print(f"[fixB] update_initial_joints skipped: {_exc}", flush=True)
|
| 308 |
+
try:
|
| 309 |
+
_cc.reset() # reset_goal() on every part controller
|
| 310 |
+
except Exception as _exc:
|
| 311 |
+
print(f"[fixB] composite reset skipped: {_exc}", flush=True)
|
| 312 |
+
inner.sim.forward()
|
| 313 |
+
# --- end FIX B -------------------------------------------------------------
|
| 314 |
+
|
| 315 |
+
obs = build_fresh_groot_obs(self.env, inner)
|
| 316 |
+
info = {
|
| 317 |
+
"is_success": self.env.is_success()["task"],
|
| 318 |
+
"replay_state_npz": state_info["state_npz"],
|
| 319 |
+
"replay_model_xml_gz": state_info["model_xml_gz"],
|
| 320 |
+
"replay_state_json": state_info["replay_json"],
|
| 321 |
+
"replay_state_flat_sha256": state_info.get("state_flat_sha256"),
|
| 322 |
+
"source_episode_index": state_info.get("source_episode_index"),
|
| 323 |
+
"source_episode_seed": state_info.get("source_episode_seed"),
|
| 324 |
+
"source_worker_id": state_info.get("source_worker_id"),
|
| 325 |
+
"source_local_episode_index": state_info.get("source_local_episode_index"),
|
| 326 |
+
"source_ep_meta_pickle": state_info.get("source_ep_meta_pickle"),
|
| 327 |
+
"regenerated_ep_meta_pickle": state_info.get("regenerated_ep_meta_pickle"),
|
| 328 |
+
}
|
| 329 |
+
self.last_state_info = state_info
|
| 330 |
+
return obs, info
|
| 331 |
+
|
| 332 |
+
|
| 333 |
+
def main() -> None:
|
| 334 |
+
parser = argparse.ArgumentParser()
|
| 335 |
+
parser.add_argument("--host", default="127.0.0.1")
|
| 336 |
+
parser.add_argument("--port", type=int, required=True)
|
| 337 |
+
parser.add_argument("--model_path", required=True)
|
| 338 |
+
parser.add_argument("--output_dir", required=True)
|
| 339 |
+
parser.add_argument("--env_name", default="PickPlaceCounterToCabinet")
|
| 340 |
+
parser.add_argument("--eval_task", default="PnPCounterToCab")
|
| 341 |
+
parser.add_argument("--finetuned_task", required=True)
|
| 342 |
+
parser.add_argument("--n_episodes", type=int, default=200)
|
| 343 |
+
parser.add_argument("--episode_offset", type=int, default=0)
|
| 344 |
+
parser.add_argument("--worker_id", default="0")
|
| 345 |
+
parser.add_argument("--seed", type=int, default=42)
|
| 346 |
+
parser.add_argument("--max_episode_steps", type=int, default=750)
|
| 347 |
+
parser.add_argument("--action_horizon", type=int, default=16)
|
| 348 |
+
parser.add_argument("--robots", default="PandaOmron")
|
| 349 |
+
parser.add_argument("--generative_textures", action="store_true")
|
| 350 |
+
parser.add_argument("--save_videos", action="store_true")
|
| 351 |
+
parser.add_argument("--replay_state_root", required=True)
|
| 352 |
+
parser.add_argument("--policy_seed", type=int, default=None)
|
| 353 |
+
args = parser.parse_args()
|
| 354 |
+
|
| 355 |
+
out_dir = Path(args.output_dir)
|
| 356 |
+
video_dir = out_dir / "videos"
|
| 357 |
+
meta_pickle_dir = out_dir / "ep_meta_pickles"
|
| 358 |
+
out_dir.mkdir(parents=True, exist_ok=True)
|
| 359 |
+
if args.save_videos:
|
| 360 |
+
video_dir.mkdir(parents=True, exist_ok=True)
|
| 361 |
+
meta_pickle_dir.mkdir(parents=True, exist_ok=True)
|
| 362 |
+
|
| 363 |
+
replay_state_root = Path(args.replay_state_root)
|
| 364 |
+
replay_state_index = load_replay_state_index(replay_state_root)
|
| 365 |
+
|
| 366 |
+
run_info = {
|
| 367 |
+
"model_path": args.model_path,
|
| 368 |
+
"finetuned_task": args.finetuned_task,
|
| 369 |
+
"eval_task": args.eval_task,
|
| 370 |
+
"env_name": args.env_name,
|
| 371 |
+
"n_episodes_requested": args.n_episodes,
|
| 372 |
+
"episode_offset": args.episode_offset,
|
| 373 |
+
"worker_id": args.worker_id,
|
| 374 |
+
"seed": args.seed,
|
| 375 |
+
"max_episode_steps": args.max_episode_steps,
|
| 376 |
+
"action_horizon": args.action_horizon,
|
| 377 |
+
"robots": args.robots,
|
| 378 |
+
"layout": None,
|
| 379 |
+
"style": None,
|
| 380 |
+
"layout_and_style_ids": TARGET_LAYOUT_AND_STYLE_IDS,
|
| 381 |
+
"obj_instance_split": TARGET_OBJ_INSTANCE_SPLIT,
|
| 382 |
+
"generative_textures": bool(args.generative_textures),
|
| 383 |
+
"save_videos": bool(args.save_videos),
|
| 384 |
+
"policy_seed": args.policy_seed,
|
| 385 |
+
"policy_action_seed_key": ACTION_SEED_KEY if args.policy_seed is not None else None,
|
| 386 |
+
"policy_action_seed_formula": (
|
| 387 |
+
f"(policy_seed + episode_index * {ACTION_SEED_EPISODE_STRIDE} + policy_step)"
|
| 388 |
+
f" % {TORCH_SEED_MODULUS}"
|
| 389 |
+
if args.policy_seed is not None
|
| 390 |
+
else None
|
| 391 |
+
),
|
| 392 |
+
"codebase": "myGR00T",
|
| 393 |
+
"data_config": "single_panda_gripper",
|
| 394 |
+
"embodiment_tag": "new_embodiment",
|
| 395 |
+
"replay_state_root": str(replay_state_root),
|
| 396 |
+
"replay_state_count": len(replay_state_index),
|
| 397 |
+
"replay_mode": "exact_model_xml_plus_mujoco_state",
|
| 398 |
+
"canonical_episode_definition": "episode_index restores model XML and flattened MuJoCo state from replay_state_root",
|
| 399 |
+
"started_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
|
| 400 |
+
}
|
| 401 |
+
(out_dir / "run_info.json").write_text(json.dumps(run_info, indent=2, sort_keys=True) + "\n")
|
| 402 |
+
|
| 403 |
+
client = PolicyClient(host=args.host, port=args.port)
|
| 404 |
+
env = load_robocasa_gym_env(
|
| 405 |
+
args.env_name,
|
| 406 |
+
seed=args.seed,
|
| 407 |
+
robots=args.robots,
|
| 408 |
+
camera_widths=256,
|
| 409 |
+
camera_heights=256,
|
| 410 |
+
render_onscreen=False,
|
| 411 |
+
obj_instance_split=TARGET_OBJ_INSTANCE_SPLIT,
|
| 412 |
+
generative_textures="100p" if args.generative_textures else None,
|
| 413 |
+
randomize_cameras=False,
|
| 414 |
+
layout_and_style_ids=TARGET_LAYOUT_AND_STYLE_IDS,
|
| 415 |
+
layout_ids=None,
|
| 416 |
+
style_ids=None,
|
| 417 |
+
)
|
| 418 |
+
env = ExactStateReplayWrapper(RoboCasaWrapper(env), replay_state_index)
|
| 419 |
+
replay_wrapper = env
|
| 420 |
+
if args.save_videos:
|
| 421 |
+
env = RecordVideo(
|
| 422 |
+
env,
|
| 423 |
+
str(video_dir),
|
| 424 |
+
disable_logger=True,
|
| 425 |
+
episode_trigger=lambda episode_id: True,
|
| 426 |
+
fps=20,
|
| 427 |
+
name_prefix=args.eval_task,
|
| 428 |
+
)
|
| 429 |
+
env = MultiStepWrapper(
|
| 430 |
+
env,
|
| 431 |
+
video_delta_indices=np.arange(1),
|
| 432 |
+
state_delta_indices=np.arange(1),
|
| 433 |
+
n_action_steps=args.action_horizon,
|
| 434 |
+
max_episode_steps=args.max_episode_steps,
|
| 435 |
+
)
|
| 436 |
+
|
| 437 |
+
episodes_jsonl = out_dir / "episodes.jsonl"
|
| 438 |
+
full_meta_jsonl = out_dir / "episodes_full_meta.jsonl"
|
| 439 |
+
csv_path = out_dir / "simulation_results.csv"
|
| 440 |
+
fieldnames = [
|
| 441 |
+
"episode_index",
|
| 442 |
+
"local_episode_index",
|
| 443 |
+
"worker_id",
|
| 444 |
+
"episode_seed",
|
| 445 |
+
"success",
|
| 446 |
+
"reward",
|
| 447 |
+
"policy_steps",
|
| 448 |
+
"sim_step_budget",
|
| 449 |
+
"video_path",
|
| 450 |
+
"layout_id",
|
| 451 |
+
"style_id",
|
| 452 |
+
"language",
|
| 453 |
+
"target_mesh_keys",
|
| 454 |
+
]
|
| 455 |
+
with csv_path.open("w", newline="") as f:
|
| 456 |
+
csv.DictWriter(f, fieldnames=fieldnames).writeheader()
|
| 457 |
+
|
| 458 |
+
successes: list[bool] = []
|
| 459 |
+
try:
|
| 460 |
+
for local_episode_index in trange(args.n_episodes, desc=f"{args.eval_task}/w{args.worker_id}"):
|
| 461 |
+
episode_index = args.episode_offset + local_episode_index
|
| 462 |
+
replay_wrapper.set_episode_index(episode_index)
|
| 463 |
+
obs, info = env.reset()
|
| 464 |
+
reset_info = dict(info)
|
| 465 |
+
ep_meta_pickle = meta_pickle_dir / f"episode_{episode_index:06d}.pkl"
|
| 466 |
+
ep_meta, ep_summary = capture_ep_meta(env, ep_meta_pickle)
|
| 467 |
+
done = False
|
| 468 |
+
policy_steps = 0
|
| 469 |
+
reward_total = 0.0
|
| 470 |
+
last_info: dict[str, Any] = info
|
| 471 |
+
|
| 472 |
+
view_frames = {"wrist_view": []} # center is already covered by RecordVideo
|
| 473 |
+
|
| 474 |
+
while not done:
|
| 475 |
+
for _vk in view_frames:
|
| 476 |
+
_f = obs.get(f"video.{_vk}")
|
| 477 |
+
if _f is not None:
|
| 478 |
+
_f = np.asarray(_f)
|
| 479 |
+
view_frames[_vk].append(_f[0] if _f.ndim == 4 else _f)
|
| 480 |
+
policy_obs = dict(obs)
|
| 481 |
+
policy_obs["video.left_view"] = np.flip(obs["video.left_view"], axis=1)
|
| 482 |
+
policy_obs["video.right_view"] = np.flip(obs["video.right_view"], axis=1)
|
| 483 |
+
policy_obs["video.wrist_view"] = np.flip(obs["video.wrist_view"], axis=1)
|
| 484 |
+
if args.policy_seed is not None:
|
| 485 |
+
policy_obs[ACTION_SEED_KEY] = make_policy_action_seed(
|
| 486 |
+
args.policy_seed,
|
| 487 |
+
episode_index,
|
| 488 |
+
policy_steps,
|
| 489 |
+
)
|
| 490 |
+
action = client.get_action(policy_obs)
|
| 491 |
+
post_action = {}
|
| 492 |
+
for key, value in action.items():
|
| 493 |
+
post_action[key] = value[..., None] if getattr(value, "ndim", 0) == 1 else value
|
| 494 |
+
obs, reward, terminated, truncated, last_info = env.step(post_action)
|
| 495 |
+
reward_total += float(np.max(reward) if isinstance(reward, np.ndarray) else reward)
|
| 496 |
+
policy_steps += 1
|
| 497 |
+
done = bool(terminated or truncated)
|
| 498 |
+
|
| 499 |
+
if args.save_videos:
|
| 500 |
+
import imageio
|
| 501 |
+
ep_dir = video_dir / f"episode_{episode_index:06d}"
|
| 502 |
+
for _vk, _frames in view_frames.items():
|
| 503 |
+
if not _frames:
|
| 504 |
+
continue
|
| 505 |
+
ep_dir.mkdir(parents=True, exist_ok=True)
|
| 506 |
+
imageio.mimwrite(str(ep_dir / "wrist.mp4"), _frames, fps=20,
|
| 507 |
+
codec="libx264", macro_block_size=1,
|
| 508 |
+
ffmpeg_params=["-pix_fmt", "yuv420p"])
|
| 509 |
+
|
| 510 |
+
success = as_bool(last_info.get("is_success", False))
|
| 511 |
+
successes.append(success)
|
| 512 |
+
video_path = (
|
| 513 |
+
str(video_dir / f"{args.eval_task}-episode_{local_episode_index}.mp4")
|
| 514 |
+
if args.save_videos
|
| 515 |
+
else ""
|
| 516 |
+
)
|
| 517 |
+
row = {
|
| 518 |
+
"episode_index": episode_index,
|
| 519 |
+
"local_episode_index": local_episode_index,
|
| 520 |
+
"worker_id": args.worker_id,
|
| 521 |
+
"episode_seed": args.seed,
|
| 522 |
+
"canonical_episode_id": f"state_replay_episode_{episode_index:06d}",
|
| 523 |
+
"success": success,
|
| 524 |
+
"reward": reward_total,
|
| 525 |
+
"policy_steps": policy_steps,
|
| 526 |
+
"sim_step_budget": policy_steps * args.action_horizon,
|
| 527 |
+
"video_path": video_path,
|
| 528 |
+
"ep_meta_pickle": str(ep_meta_pickle),
|
| 529 |
+
"replay_state_npz": reset_info.get("replay_state_npz"),
|
| 530 |
+
"replay_model_xml_gz": reset_info.get("replay_model_xml_gz"),
|
| 531 |
+
"replay_state_json": reset_info.get("replay_state_json"),
|
| 532 |
+
"replay_state_flat_sha256": reset_info.get("replay_state_flat_sha256"),
|
| 533 |
+
"source_ep_meta_pickle": reset_info.get("source_ep_meta_pickle"),
|
| 534 |
+
"regenerated_ep_meta_pickle": reset_info.get("regenerated_ep_meta_pickle"),
|
| 535 |
+
"model_path": args.model_path,
|
| 536 |
+
"finetuned_task": args.finetuned_task,
|
| 537 |
+
"eval_task": args.eval_task,
|
| 538 |
+
"env_name": args.env_name,
|
| 539 |
+
"seed": args.seed,
|
| 540 |
+
**ep_summary,
|
| 541 |
+
}
|
| 542 |
+
write_jsonl(episodes_jsonl, row)
|
| 543 |
+
write_jsonl(
|
| 544 |
+
full_meta_jsonl,
|
| 545 |
+
{
|
| 546 |
+
"episode_index": episode_index,
|
| 547 |
+
"ep_meta_pickle": str(ep_meta_pickle),
|
| 548 |
+
"ep_meta": ep_meta,
|
| 549 |
+
},
|
| 550 |
+
)
|
| 551 |
+
with csv_path.open("a", newline="") as f:
|
| 552 |
+
writer = csv.DictWriter(f, fieldnames=fieldnames)
|
| 553 |
+
writer.writerow(
|
| 554 |
+
{
|
| 555 |
+
"episode_index": row["episode_index"],
|
| 556 |
+
"local_episode_index": row["local_episode_index"],
|
| 557 |
+
"worker_id": row["worker_id"],
|
| 558 |
+
"episode_seed": row["episode_seed"],
|
| 559 |
+
"success": int(success),
|
| 560 |
+
"reward": reward_total,
|
| 561 |
+
"policy_steps": policy_steps,
|
| 562 |
+
"sim_step_budget": policy_steps * args.action_horizon,
|
| 563 |
+
"video_path": video_path,
|
| 564 |
+
"layout_id": ep_summary.get("layout_id"),
|
| 565 |
+
"style_id": ep_summary.get("style_id"),
|
| 566 |
+
"language": ep_summary.get("language"),
|
| 567 |
+
"target_mesh_keys": ";".join(
|
| 568 |
+
str(obj.get("mesh_key"))
|
| 569 |
+
for obj in ep_summary.get("objects", [])
|
| 570 |
+
if obj.get("mesh_key")
|
| 571 |
+
),
|
| 572 |
+
}
|
| 573 |
+
)
|
| 574 |
+
finally:
|
| 575 |
+
env.close()
|
| 576 |
+
client.close()
|
| 577 |
+
|
| 578 |
+
summary = {
|
| 579 |
+
**run_info,
|
| 580 |
+
"finished_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
|
| 581 |
+
"n_episodes": len(successes),
|
| 582 |
+
"n_success": int(sum(successes)),
|
| 583 |
+
"success_rate": float(sum(successes) / max(len(successes), 1)),
|
| 584 |
+
}
|
| 585 |
+
(out_dir / "summary.json").write_text(json.dumps(summary, indent=2, sort_keys=True) + "\n")
|
| 586 |
+
print(json.dumps(summary, indent=2, sort_keys=True))
|
| 587 |
+
|
| 588 |
+
|
| 589 |
+
if __name__ == "__main__":
|
| 590 |
+
main()
|
eval/eval_robocasa_replay_state_grasp.py
ADDED
|
@@ -0,0 +1,633 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
"""Run RoboCasa eval from saved model XML + MuJoCo state artifacts.
|
| 3 |
+
|
| 4 |
+
This is the strict replay path for apples-to-apples evaluation after a
|
| 5 |
+
reconstructed episode bank has been created by `eval_robocasa_reconstruct_xyz.py`.
|
| 6 |
+
Each episode is restored from:
|
| 7 |
+
|
| 8 |
+
- `replay_states/episode_XXXXXX.xml.gz`
|
| 9 |
+
- `replay_states/episode_XXXXXX_state.npz`
|
| 10 |
+
- `replay_states/episode_XXXXXX_replay.json`
|
| 11 |
+
"""
|
| 12 |
+
|
| 13 |
+
from __future__ import annotations
|
| 14 |
+
|
| 15 |
+
import argparse
|
| 16 |
+
import csv
|
| 17 |
+
import gzip
|
| 18 |
+
import json
|
| 19 |
+
import pickle
|
| 20 |
+
import time
|
| 21 |
+
from io import BytesIO
|
| 22 |
+
from pathlib import Path
|
| 23 |
+
from typing import Any
|
| 24 |
+
|
| 25 |
+
import gymnasium as gym
|
| 26 |
+
import numpy as np
|
| 27 |
+
import robocasa # noqa: F401 - registers RoboCasa environments with robosuite
|
| 28 |
+
import torch
|
| 29 |
+
import zmq
|
| 30 |
+
from tqdm import trange
|
| 31 |
+
|
| 32 |
+
from gr00t.eval.wrappers.multistep_wrapper import MultiStepWrapper
|
| 33 |
+
from gr00t.eval.wrappers.record_video import RecordVideo
|
| 34 |
+
from gr00t.eval.wrappers.robocasa_wrapper import RoboCasaWrapper, load_robocasa_gym_env
|
| 35 |
+
|
| 36 |
+
TARGET_LAYOUT_AND_STYLE_IDS = [[i, i] for i in range(1, 11)]
|
| 37 |
+
TARGET_OBJ_INSTANCE_SPLIT = "target"
|
| 38 |
+
ACTION_SEED_KEY = "_policy_action_seed"
|
| 39 |
+
ACTION_SEED_EPISODE_STRIDE = 1_000_003
|
| 40 |
+
TORCH_SEED_MODULUS = 2**63 - 1
|
| 41 |
+
|
| 42 |
+
|
| 43 |
+
class PolicyClient:
|
| 44 |
+
"""Minimal client for myGR00T's RobotInferenceServer."""
|
| 45 |
+
|
| 46 |
+
def __init__(self, host: str, port: int):
|
| 47 |
+
self.context = zmq.Context()
|
| 48 |
+
self.socket = self.context.socket(zmq.REQ)
|
| 49 |
+
self.socket.connect(f"tcp://{host}:{port}")
|
| 50 |
+
|
| 51 |
+
@staticmethod
|
| 52 |
+
def to_bytes(data: dict[str, Any]) -> bytes:
|
| 53 |
+
buffer = BytesIO()
|
| 54 |
+
torch.save(data, buffer)
|
| 55 |
+
return buffer.getvalue()
|
| 56 |
+
|
| 57 |
+
@staticmethod
|
| 58 |
+
def from_bytes(data: bytes) -> dict[str, Any]:
|
| 59 |
+
buffer = BytesIO(data)
|
| 60 |
+
return torch.load(buffer, weights_only=False)
|
| 61 |
+
|
| 62 |
+
def call(self, endpoint: str, data: dict[str, Any] | None = None) -> dict[str, Any]:
|
| 63 |
+
request: dict[str, Any] = {"endpoint": endpoint}
|
| 64 |
+
if data is not None:
|
| 65 |
+
request["data"] = data
|
| 66 |
+
self.socket.send(self.to_bytes(request))
|
| 67 |
+
message = self.socket.recv()
|
| 68 |
+
if message == b"ERROR":
|
| 69 |
+
raise RuntimeError("policy server returned ERROR")
|
| 70 |
+
return self.from_bytes(message)
|
| 71 |
+
|
| 72 |
+
def get_action(self, observations: dict[str, Any]) -> dict[str, Any]:
|
| 73 |
+
return self.call("get_action", observations)
|
| 74 |
+
|
| 75 |
+
def close(self) -> None:
|
| 76 |
+
self.socket.close()
|
| 77 |
+
self.context.term()
|
| 78 |
+
|
| 79 |
+
|
| 80 |
+
def as_bool(value: Any) -> bool:
|
| 81 |
+
if isinstance(value, np.ndarray):
|
| 82 |
+
return bool(np.any(value))
|
| 83 |
+
if isinstance(value, (list, tuple)):
|
| 84 |
+
return any(as_bool(v) for v in value)
|
| 85 |
+
if isinstance(value, dict):
|
| 86 |
+
return any(as_bool(v) for v in value.values())
|
| 87 |
+
return bool(value)
|
| 88 |
+
|
| 89 |
+
|
| 90 |
+
def json_default(value: Any) -> Any:
|
| 91 |
+
if isinstance(value, np.ndarray):
|
| 92 |
+
return value.tolist()
|
| 93 |
+
if isinstance(value, np.generic):
|
| 94 |
+
return value.item()
|
| 95 |
+
if isinstance(value, Path):
|
| 96 |
+
return str(value)
|
| 97 |
+
return str(value)
|
| 98 |
+
|
| 99 |
+
|
| 100 |
+
def find_robocasa_inner(env: Any) -> Any | None:
|
| 101 |
+
cur = env
|
| 102 |
+
for _ in range(80):
|
| 103 |
+
if cur is None:
|
| 104 |
+
return None
|
| 105 |
+
nxt = getattr(cur, "env", None)
|
| 106 |
+
if nxt is None or nxt is cur:
|
| 107 |
+
break
|
| 108 |
+
cur = nxt
|
| 109 |
+
if hasattr(cur, "sim") and hasattr(cur, "get_ep_meta"):
|
| 110 |
+
return cur
|
| 111 |
+
return None
|
| 112 |
+
|
| 113 |
+
|
| 114 |
+
def parse_mjcf_path(mjcf_path: str | None) -> dict[str, str | None]:
|
| 115 |
+
if not mjcf_path:
|
| 116 |
+
return {"registry": None, "instance_id": None, "mesh_key": None}
|
| 117 |
+
path = Path(str(mjcf_path))
|
| 118 |
+
parts = path.parts
|
| 119 |
+
registry = None
|
| 120 |
+
for candidate in ("objaverse", "aigen_objs", "lightwheel"):
|
| 121 |
+
if candidate in parts:
|
| 122 |
+
registry = candidate
|
| 123 |
+
break
|
| 124 |
+
instance_id = path.parent.name if path.name.endswith(".xml") else None
|
| 125 |
+
mesh_key = None
|
| 126 |
+
if registry and registry in parts and instance_id:
|
| 127 |
+
idx = parts.index(registry)
|
| 128 |
+
if registry == "lightwheel" and idx + 1 < len(parts):
|
| 129 |
+
mesh_key = "/".join((parts[idx + 1], instance_id))
|
| 130 |
+
elif idx + 1 < len(parts):
|
| 131 |
+
mesh_key = "/".join(parts[idx + 1 : -1])
|
| 132 |
+
return {"registry": registry, "instance_id": instance_id, "mesh_key": mesh_key}
|
| 133 |
+
|
| 134 |
+
|
| 135 |
+
def summarize_ep_meta(ep_meta: dict[str, Any]) -> dict[str, Any]:
|
| 136 |
+
summary: dict[str, Any] = {
|
| 137 |
+
"layout_id": ep_meta.get("layout_id"),
|
| 138 |
+
"style_id": ep_meta.get("style_id"),
|
| 139 |
+
"language": ep_meta.get("lang", ""),
|
| 140 |
+
"objects": [],
|
| 141 |
+
}
|
| 142 |
+
for cfg in ep_meta.get("object_cfgs", []) or []:
|
| 143 |
+
if not isinstance(cfg, dict):
|
| 144 |
+
continue
|
| 145 |
+
info = cfg.get("info", {}) if isinstance(cfg.get("info", {}), dict) else {}
|
| 146 |
+
mjcf_path = info.get("mjcf_path")
|
| 147 |
+
parsed = parse_mjcf_path(mjcf_path)
|
| 148 |
+
summary["objects"].append(
|
| 149 |
+
{
|
| 150 |
+
"name": cfg.get("name"),
|
| 151 |
+
"category": info.get("cat"),
|
| 152 |
+
"mjcf_path": mjcf_path,
|
| 153 |
+
"registry": parsed["registry"],
|
| 154 |
+
"instance_id": parsed["instance_id"],
|
| 155 |
+
"mesh_key": parsed["mesh_key"],
|
| 156 |
+
}
|
| 157 |
+
)
|
| 158 |
+
fixtures = ep_meta.get("fixtures")
|
| 159 |
+
if isinstance(fixtures, dict):
|
| 160 |
+
summary["fixture_names"] = list(fixtures.keys())
|
| 161 |
+
return summary
|
| 162 |
+
|
| 163 |
+
|
| 164 |
+
def capture_ep_meta(env: Any, pickle_path: Path) -> tuple[dict[str, Any], dict[str, Any]]:
|
| 165 |
+
inner = find_robocasa_inner(env)
|
| 166 |
+
if inner is None:
|
| 167 |
+
return {}, {"error": "could_not_find_robocasa_inner_env"}
|
| 168 |
+
try:
|
| 169 |
+
ep_meta = inner.get_ep_meta() or {}
|
| 170 |
+
except Exception as exc:
|
| 171 |
+
return {}, {"error": f"get_ep_meta_failed: {type(exc).__name__}: {exc}"}
|
| 172 |
+
with pickle_path.open("wb") as f:
|
| 173 |
+
pickle.dump(ep_meta, f)
|
| 174 |
+
return ep_meta, summarize_ep_meta(ep_meta)
|
| 175 |
+
|
| 176 |
+
|
| 177 |
+
def write_jsonl(path: Path, row: dict[str, Any]) -> None:
|
| 178 |
+
with path.open("a") as f:
|
| 179 |
+
f.write(json.dumps(row, default=json_default, sort_keys=True) + "\n")
|
| 180 |
+
f.flush()
|
| 181 |
+
|
| 182 |
+
|
| 183 |
+
def make_policy_action_seed(policy_seed: int, episode_index: int, policy_step: int) -> int:
|
| 184 |
+
seed = int(policy_seed) + int(episode_index) * ACTION_SEED_EPISODE_STRIDE + int(policy_step)
|
| 185 |
+
return seed % TORCH_SEED_MODULUS
|
| 186 |
+
|
| 187 |
+
|
| 188 |
+
def resolve_artifact_path(value: str, state_json_path: Path) -> Path:
|
| 189 |
+
path = Path(value)
|
| 190 |
+
if path.is_file():
|
| 191 |
+
return path
|
| 192 |
+
candidate = state_json_path.parent / path.name
|
| 193 |
+
if candidate.is_file():
|
| 194 |
+
return candidate
|
| 195 |
+
raise FileNotFoundError(f"missing replay artifact: {value}")
|
| 196 |
+
|
| 197 |
+
|
| 198 |
+
def load_replay_state_index(root: Path) -> dict[int, dict[str, Any]]:
|
| 199 |
+
if not root.is_dir():
|
| 200 |
+
raise FileNotFoundError(f"missing replay_state_root: {root}")
|
| 201 |
+
paths = sorted(root.glob("workers/worker_*/replay_states/episode_*_replay.json"))
|
| 202 |
+
paths.extend(sorted(root.glob("replay_states/episode_*_replay.json")))
|
| 203 |
+
index: dict[int, dict[str, Any]] = {}
|
| 204 |
+
for path in paths:
|
| 205 |
+
row = json.loads(path.read_text())
|
| 206 |
+
episode_index = int(row["episode_index"])
|
| 207 |
+
row["replay_json"] = str(path)
|
| 208 |
+
row["model_xml_gz"] = str(resolve_artifact_path(str(row["model_xml_gz"]), path))
|
| 209 |
+
row["state_npz"] = str(resolve_artifact_path(str(row["state_npz"]), path))
|
| 210 |
+
index[episode_index] = row
|
| 211 |
+
if not index:
|
| 212 |
+
raise FileNotFoundError(f"no replay state jsons found under {root}")
|
| 213 |
+
return index
|
| 214 |
+
|
| 215 |
+
|
| 216 |
+
def load_ep_meta_from_state_info(state_info: dict[str, Any]) -> dict[str, Any] | None:
|
| 217 |
+
for key in ("regenerated_ep_meta_pickle", "source_ep_meta_pickle", "ep_meta_pickle"):
|
| 218 |
+
path_value = state_info.get(key)
|
| 219 |
+
if not path_value:
|
| 220 |
+
continue
|
| 221 |
+
path = Path(str(path_value))
|
| 222 |
+
if not path.is_file():
|
| 223 |
+
continue
|
| 224 |
+
with path.open("rb") as f:
|
| 225 |
+
return pickle.load(f)
|
| 226 |
+
return None
|
| 227 |
+
|
| 228 |
+
|
| 229 |
+
def build_fresh_groot_obs(robo_wrapper: RoboCasaWrapper, inner: Any) -> dict[str, Any]:
|
| 230 |
+
gym_env = robo_wrapper.env
|
| 231 |
+
if getattr(inner, "viewer_get_obs", False) and getattr(inner, "viewer", None) is not None:
|
| 232 |
+
raw_obs = inner.viewer._get_observations(force_update=True)
|
| 233 |
+
else:
|
| 234 |
+
raw_obs = inner._get_observations(force_update=True)
|
| 235 |
+
if getattr(gym_env, "flatten_obs", False):
|
| 236 |
+
obs = gym_env._flatten_obs(raw_obs)
|
| 237 |
+
else:
|
| 238 |
+
obs = gym_env._filter_obs(raw_obs)
|
| 239 |
+
new_obs: dict[str, Any] = {}
|
| 240 |
+
for key, value in obs.items():
|
| 241 |
+
if key in robo_wrapper._robocasa_keys_to_gr00t_keys:
|
| 242 |
+
new_obs[robo_wrapper._robocasa_keys_to_gr00t_keys[key]] = value
|
| 243 |
+
new_obs["annotation.human.action.task_description"] = [robo_wrapper.language_instruction]
|
| 244 |
+
return new_obs
|
| 245 |
+
|
| 246 |
+
|
| 247 |
+
class ExactStateReplayWrapper(gym.Wrapper):
|
| 248 |
+
def __init__(self, env: RoboCasaWrapper, replay_state_index: dict[int, dict[str, Any]]):
|
| 249 |
+
super().__init__(env)
|
| 250 |
+
self.replay_state_index = replay_state_index
|
| 251 |
+
self.current_episode_index: int | None = None
|
| 252 |
+
self.last_state_info: dict[str, Any] = {}
|
| 253 |
+
|
| 254 |
+
def set_episode_index(self, episode_index: int) -> None:
|
| 255 |
+
self.current_episode_index = int(episode_index)
|
| 256 |
+
|
| 257 |
+
def reset(self, seed=None, options=None):
|
| 258 |
+
if self.current_episode_index is None:
|
| 259 |
+
raise RuntimeError("set_episode_index must be called before reset")
|
| 260 |
+
if self.current_episode_index not in self.replay_state_index:
|
| 261 |
+
raise KeyError(f"missing replay state for episode_index={self.current_episode_index}")
|
| 262 |
+
state_info = self.replay_state_index[self.current_episode_index]
|
| 263 |
+
inner = find_robocasa_inner(self.env)
|
| 264 |
+
if inner is None or not hasattr(inner, "reset_from_xml_string"):
|
| 265 |
+
raise RuntimeError("could not find RoboCasa inner env with reset_from_xml_string")
|
| 266 |
+
|
| 267 |
+
ep_meta = load_ep_meta_from_state_info(state_info)
|
| 268 |
+
if ep_meta is not None and hasattr(inner, "set_ep_meta"):
|
| 269 |
+
inner.set_ep_meta(ep_meta)
|
| 270 |
+
|
| 271 |
+
with gzip.open(Path(state_info["model_xml_gz"]), "rt") as f:
|
| 272 |
+
model_xml = f.read()
|
| 273 |
+
saved = np.load(Path(state_info["state_npz"]))
|
| 274 |
+
saved_state = saved["states"]
|
| 275 |
+
saved_ctrl = saved["ctrl"] if "ctrl" in saved.files else None
|
| 276 |
+
|
| 277 |
+
inner.reset()
|
| 278 |
+
if ep_meta is not None and hasattr(inner, "set_ep_meta"):
|
| 279 |
+
inner.set_ep_meta(ep_meta)
|
| 280 |
+
inner.reset_from_xml_string(model_xml)
|
| 281 |
+
inner.sim.reset()
|
| 282 |
+
inner.sim.set_state_from_flattened(saved_state)
|
| 283 |
+
if saved_ctrl is not None and saved_ctrl.shape == inner.sim.data.ctrl.shape:
|
| 284 |
+
inner.sim.data.ctrl[:] = saved_ctrl
|
| 285 |
+
inner.sim.forward()
|
| 286 |
+
|
| 287 |
+
# --- FIX B: re-sync each part controller to the RESTORED state -------------
|
| 288 |
+
# reset_from_xml_string() above triggers a throwaway env reset, during which
|
| 289 |
+
# every controller latches initial_joint (used for the OSC nullspace bias) and
|
| 290 |
+
# goal_pos from the robot's freshly *randomised* init pose (robot.py: init_qpos
|
| 291 |
+
# += gaussian noise, magnitude 0.02). set_state_from_flattened() then teleports
|
| 292 |
+
# the robot to the episode's saved pose, but nothing re-syncs the controllers,
|
| 293 |
+
# so they keep biasing toward a pose this episode never had -- and that stale
|
| 294 |
+
# pose depends on the client --seed, which leaks worker sharding into results.
|
| 295 |
+
# Re-seed each controller from the restored qpos instead.
|
| 296 |
+
for _robot in getattr(inner, "robots", []) or []:
|
| 297 |
+
_cc = getattr(_robot, "composite_controller", None)
|
| 298 |
+
if _cc is None:
|
| 299 |
+
continue
|
| 300 |
+
for _ctrl in getattr(_cc, "part_controllers", {}).values():
|
| 301 |
+
_qidx = getattr(_ctrl, "qpos_index", None)
|
| 302 |
+
if _qidx is None or getattr(_ctrl, "ref_name", None) is None:
|
| 303 |
+
continue # e.g. MobileBaseJointVelocityController has no ref_pos
|
| 304 |
+
try:
|
| 305 |
+
_ctrl.update_initial_joints(inner.sim.data.qpos[_qidx])
|
| 306 |
+
except Exception as _exc:
|
| 307 |
+
print(f"[fixB] update_initial_joints skipped: {_exc}", flush=True)
|
| 308 |
+
try:
|
| 309 |
+
_cc.reset() # reset_goal() on every part controller
|
| 310 |
+
except Exception as _exc:
|
| 311 |
+
print(f"[fixB] composite reset skipped: {_exc}", flush=True)
|
| 312 |
+
inner.sim.forward()
|
| 313 |
+
# --- end FIX B -------------------------------------------------------------
|
| 314 |
+
|
| 315 |
+
obs = build_fresh_groot_obs(self.env, inner)
|
| 316 |
+
info = {
|
| 317 |
+
"is_success": self.env.is_success()["task"],
|
| 318 |
+
"replay_state_npz": state_info["state_npz"],
|
| 319 |
+
"replay_model_xml_gz": state_info["model_xml_gz"],
|
| 320 |
+
"replay_state_json": state_info["replay_json"],
|
| 321 |
+
"replay_state_flat_sha256": state_info.get("state_flat_sha256"),
|
| 322 |
+
"source_episode_index": state_info.get("source_episode_index"),
|
| 323 |
+
"source_episode_seed": state_info.get("source_episode_seed"),
|
| 324 |
+
"source_worker_id": state_info.get("source_worker_id"),
|
| 325 |
+
"source_local_episode_index": state_info.get("source_local_episode_index"),
|
| 326 |
+
"source_ep_meta_pickle": state_info.get("source_ep_meta_pickle"),
|
| 327 |
+
"regenerated_ep_meta_pickle": state_info.get("regenerated_ep_meta_pickle"),
|
| 328 |
+
}
|
| 329 |
+
self.last_state_info = state_info
|
| 330 |
+
return obs, info
|
| 331 |
+
|
| 332 |
+
|
| 333 |
+
def main() -> None:
|
| 334 |
+
parser = argparse.ArgumentParser()
|
| 335 |
+
parser.add_argument("--host", default="127.0.0.1")
|
| 336 |
+
parser.add_argument("--port", type=int, required=True)
|
| 337 |
+
parser.add_argument("--model_path", required=True)
|
| 338 |
+
parser.add_argument("--output_dir", required=True)
|
| 339 |
+
parser.add_argument("--env_name", default="PickPlaceCounterToCabinet")
|
| 340 |
+
parser.add_argument("--eval_task", default="PnPCounterToCab")
|
| 341 |
+
parser.add_argument("--finetuned_task", required=True)
|
| 342 |
+
parser.add_argument("--n_episodes", type=int, default=200)
|
| 343 |
+
parser.add_argument("--episode_offset", type=int, default=0)
|
| 344 |
+
parser.add_argument("--episode_indices", default=None,
|
| 345 |
+
help="comma-separated global episode indices; overrides offset/n_episodes")
|
| 346 |
+
parser.add_argument("--worker_id", default="0")
|
| 347 |
+
parser.add_argument("--seed", type=int, default=42)
|
| 348 |
+
parser.add_argument("--max_episode_steps", type=int, default=750)
|
| 349 |
+
parser.add_argument("--action_horizon", type=int, default=16)
|
| 350 |
+
parser.add_argument("--robots", default="PandaOmron")
|
| 351 |
+
parser.add_argument("--generative_textures", action="store_true")
|
| 352 |
+
parser.add_argument("--save_videos", action="store_true")
|
| 353 |
+
parser.add_argument("--replay_state_root", required=True)
|
| 354 |
+
parser.add_argument("--policy_seed", type=int, default=None)
|
| 355 |
+
args = parser.parse_args()
|
| 356 |
+
|
| 357 |
+
out_dir = Path(args.output_dir)
|
| 358 |
+
video_dir = out_dir / "videos"
|
| 359 |
+
meta_pickle_dir = out_dir / "ep_meta_pickles"
|
| 360 |
+
out_dir.mkdir(parents=True, exist_ok=True)
|
| 361 |
+
if args.save_videos:
|
| 362 |
+
video_dir.mkdir(parents=True, exist_ok=True)
|
| 363 |
+
meta_pickle_dir.mkdir(parents=True, exist_ok=True)
|
| 364 |
+
|
| 365 |
+
replay_state_root = Path(args.replay_state_root)
|
| 366 |
+
replay_state_index = load_replay_state_index(replay_state_root)
|
| 367 |
+
|
| 368 |
+
run_info = {
|
| 369 |
+
"model_path": args.model_path,
|
| 370 |
+
"finetuned_task": args.finetuned_task,
|
| 371 |
+
"eval_task": args.eval_task,
|
| 372 |
+
"env_name": args.env_name,
|
| 373 |
+
"n_episodes_requested": args.n_episodes,
|
| 374 |
+
"episode_offset": args.episode_offset,
|
| 375 |
+
"worker_id": args.worker_id,
|
| 376 |
+
"seed": args.seed,
|
| 377 |
+
"max_episode_steps": args.max_episode_steps,
|
| 378 |
+
"action_horizon": args.action_horizon,
|
| 379 |
+
"robots": args.robots,
|
| 380 |
+
"layout": None,
|
| 381 |
+
"style": None,
|
| 382 |
+
"layout_and_style_ids": TARGET_LAYOUT_AND_STYLE_IDS,
|
| 383 |
+
"obj_instance_split": TARGET_OBJ_INSTANCE_SPLIT,
|
| 384 |
+
"generative_textures": bool(args.generative_textures),
|
| 385 |
+
"save_videos": bool(args.save_videos),
|
| 386 |
+
"policy_seed": args.policy_seed,
|
| 387 |
+
"policy_action_seed_key": ACTION_SEED_KEY if args.policy_seed is not None else None,
|
| 388 |
+
"policy_action_seed_formula": (
|
| 389 |
+
f"(policy_seed + episode_index * {ACTION_SEED_EPISODE_STRIDE} + policy_step)"
|
| 390 |
+
f" % {TORCH_SEED_MODULUS}"
|
| 391 |
+
if args.policy_seed is not None
|
| 392 |
+
else None
|
| 393 |
+
),
|
| 394 |
+
"codebase": "myGR00T",
|
| 395 |
+
"data_config": "single_panda_gripper",
|
| 396 |
+
"embodiment_tag": "new_embodiment",
|
| 397 |
+
"replay_state_root": str(replay_state_root),
|
| 398 |
+
"replay_state_count": len(replay_state_index),
|
| 399 |
+
"replay_mode": "exact_model_xml_plus_mujoco_state",
|
| 400 |
+
"canonical_episode_definition": "episode_index restores model XML and flattened MuJoCo state from replay_state_root",
|
| 401 |
+
"started_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
|
| 402 |
+
}
|
| 403 |
+
(out_dir / "run_info.json").write_text(json.dumps(run_info, indent=2, sort_keys=True) + "\n")
|
| 404 |
+
|
| 405 |
+
client = PolicyClient(host=args.host, port=args.port)
|
| 406 |
+
env = load_robocasa_gym_env(
|
| 407 |
+
args.env_name,
|
| 408 |
+
seed=args.seed,
|
| 409 |
+
robots=args.robots,
|
| 410 |
+
camera_widths=256,
|
| 411 |
+
camera_heights=256,
|
| 412 |
+
render_onscreen=False,
|
| 413 |
+
obj_instance_split=TARGET_OBJ_INSTANCE_SPLIT,
|
| 414 |
+
generative_textures="100p" if args.generative_textures else None,
|
| 415 |
+
randomize_cameras=False,
|
| 416 |
+
layout_and_style_ids=TARGET_LAYOUT_AND_STYLE_IDS,
|
| 417 |
+
layout_ids=None,
|
| 418 |
+
style_ids=None,
|
| 419 |
+
)
|
| 420 |
+
env = ExactStateReplayWrapper(RoboCasaWrapper(env), replay_state_index)
|
| 421 |
+
replay_wrapper = env
|
| 422 |
+
if args.save_videos:
|
| 423 |
+
env = RecordVideo(
|
| 424 |
+
env,
|
| 425 |
+
str(video_dir),
|
| 426 |
+
disable_logger=True,
|
| 427 |
+
episode_trigger=lambda episode_id: True,
|
| 428 |
+
fps=20,
|
| 429 |
+
name_prefix=args.eval_task,
|
| 430 |
+
)
|
| 431 |
+
env = MultiStepWrapper(
|
| 432 |
+
env,
|
| 433 |
+
video_delta_indices=np.arange(1),
|
| 434 |
+
state_delta_indices=np.arange(1),
|
| 435 |
+
n_action_steps=args.action_horizon,
|
| 436 |
+
max_episode_steps=args.max_episode_steps,
|
| 437 |
+
)
|
| 438 |
+
|
| 439 |
+
episodes_jsonl = out_dir / "episodes.jsonl"
|
| 440 |
+
full_meta_jsonl = out_dir / "episodes_full_meta.jsonl"
|
| 441 |
+
csv_path = out_dir / "simulation_results.csv"
|
| 442 |
+
fieldnames = [
|
| 443 |
+
"episode_index",
|
| 444 |
+
"local_episode_index",
|
| 445 |
+
"worker_id",
|
| 446 |
+
"episode_seed",
|
| 447 |
+
"success",
|
| 448 |
+
"reward",
|
| 449 |
+
"policy_steps",
|
| 450 |
+
"sim_step_budget",
|
| 451 |
+
"video_path",
|
| 452 |
+
"layout_id",
|
| 453 |
+
"style_id",
|
| 454 |
+
"language",
|
| 455 |
+
"target_mesh_keys",
|
| 456 |
+
"grasped",
|
| 457 |
+
"grasped_strict",
|
| 458 |
+
"lifted",
|
| 459 |
+
"in_cab",
|
| 460 |
+
"max_lift_m",
|
| 461 |
+
]
|
| 462 |
+
with csv_path.open("w", newline="") as f:
|
| 463 |
+
csv.DictWriter(f, fieldnames=fieldnames).writeheader()
|
| 464 |
+
|
| 465 |
+
successes: list[bool] = []
|
| 466 |
+
try:
|
| 467 |
+
_ep_list = ([int(x) for x in args.episode_indices.split(',') if x.strip()]
|
| 468 |
+
if args.episode_indices else None)
|
| 469 |
+
_n = len(_ep_list) if _ep_list else args.n_episodes
|
| 470 |
+
for local_episode_index in trange(_n, desc=f"{args.eval_task}/w{args.worker_id}"):
|
| 471 |
+
episode_index = (_ep_list[local_episode_index] if _ep_list
|
| 472 |
+
else args.episode_offset + local_episode_index)
|
| 473 |
+
replay_wrapper.set_episode_index(episode_index)
|
| 474 |
+
obs, info = env.reset()
|
| 475 |
+
reset_info = dict(info)
|
| 476 |
+
ep_meta_pickle = meta_pickle_dir / f"episode_{episode_index:06d}.pkl"
|
| 477 |
+
ep_meta, ep_summary = capture_ep_meta(env, ep_meta_pickle)
|
| 478 |
+
done = False
|
| 479 |
+
policy_steps = 0
|
| 480 |
+
reward_total = 0.0
|
| 481 |
+
last_info: dict[str, Any] = info
|
| 482 |
+
|
| 483 |
+
view_frames = {"wrist_view": []} # center is already covered by RecordVideo
|
| 484 |
+
stage = {"grasped": False, "grasped_strict": False, "lifted": False, "in_cab": False}
|
| 485 |
+
_max_dz = 0.0
|
| 486 |
+
_obj_z0 = None
|
| 487 |
+
|
| 488 |
+
while not done:
|
| 489 |
+
for _vk in view_frames:
|
| 490 |
+
_f = obs.get(f"video.{_vk}")
|
| 491 |
+
if _f is not None:
|
| 492 |
+
_f = np.asarray(_f)
|
| 493 |
+
view_frames[_vk].append(_f[0] if _f.ndim == 4 else _f)
|
| 494 |
+
policy_obs = dict(obs)
|
| 495 |
+
policy_obs["video.left_view"] = np.flip(obs["video.left_view"], axis=1)
|
| 496 |
+
policy_obs["video.right_view"] = np.flip(obs["video.right_view"], axis=1)
|
| 497 |
+
policy_obs["video.wrist_view"] = np.flip(obs["video.wrist_view"], axis=1)
|
| 498 |
+
if args.policy_seed is not None:
|
| 499 |
+
policy_obs[ACTION_SEED_KEY] = make_policy_action_seed(
|
| 500 |
+
args.policy_seed,
|
| 501 |
+
episode_index,
|
| 502 |
+
policy_steps,
|
| 503 |
+
)
|
| 504 |
+
action = client.get_action(policy_obs)
|
| 505 |
+
post_action = {}
|
| 506 |
+
for key, value in action.items():
|
| 507 |
+
post_action[key] = value[..., None] if getattr(value, "ndim", 0) == 1 else value
|
| 508 |
+
obs, reward, terminated, truncated, last_info = env.step(post_action)
|
| 509 |
+
try:
|
| 510 |
+
_inner = replay_wrapper.env.unwrapped
|
| 511 |
+
import robocasa.utils.object_utils as _OU
|
| 512 |
+
_z = _inner.sim.data.body_xpos[
|
| 513 |
+
_inner.sim.model.body_name2id(_inner.objects["obj"].root_body)][2]
|
| 514 |
+
if _obj_z0 is None:
|
| 515 |
+
_obj_z0 = _z
|
| 516 |
+
# RoboCasa's own definition: any gripper/object contact AND fingers closed.
|
| 517 |
+
# robosuite's _check_grasp additionally demands contact on BOTH fingerpads,
|
| 518 |
+
# which missed a grasp on an episode that completed the full task, so it is
|
| 519 |
+
# kept only as a strict secondary flag.
|
| 520 |
+
if _OU.check_obj_grasped(_inner, "obj"):
|
| 521 |
+
stage["grasped"] = True
|
| 522 |
+
if _inner._check_grasp(gripper=_inner.robots[0].gripper["right"],
|
| 523 |
+
object_geoms=_inner.objects["obj"].contact_geoms):
|
| 524 |
+
stage["grasped_strict"] = True
|
| 525 |
+
_max_dz = max(_max_dz, _z - _obj_z0)
|
| 526 |
+
if _z - _obj_z0 > 0.03:
|
| 527 |
+
stage["lifted"] = True
|
| 528 |
+
if _OU.obj_inside_of(_inner, "obj", _inner.cab):
|
| 529 |
+
stage["in_cab"] = True
|
| 530 |
+
except Exception as _e:
|
| 531 |
+
if policy_steps == 0:
|
| 532 |
+
print(f" stage logging unavailable: {_e}", flush=True)
|
| 533 |
+
reward_total += float(np.max(reward) if isinstance(reward, np.ndarray) else reward)
|
| 534 |
+
policy_steps += 1
|
| 535 |
+
done = bool(terminated or truncated)
|
| 536 |
+
|
| 537 |
+
if args.save_videos:
|
| 538 |
+
import imageio
|
| 539 |
+
ep_dir = video_dir / f"episode_{episode_index:06d}"
|
| 540 |
+
for _vk, _frames in view_frames.items():
|
| 541 |
+
if not _frames:
|
| 542 |
+
continue
|
| 543 |
+
ep_dir.mkdir(parents=True, exist_ok=True)
|
| 544 |
+
imageio.mimwrite(str(ep_dir / "wrist.mp4"), _frames, fps=20,
|
| 545 |
+
codec="libx264", macro_block_size=1,
|
| 546 |
+
ffmpeg_params=["-pix_fmt", "yuv420p"])
|
| 547 |
+
|
| 548 |
+
success = as_bool(last_info.get("is_success", False))
|
| 549 |
+
successes.append(success)
|
| 550 |
+
video_path = (
|
| 551 |
+
str(video_dir / f"{args.eval_task}-episode_{local_episode_index}.mp4")
|
| 552 |
+
if args.save_videos
|
| 553 |
+
else ""
|
| 554 |
+
)
|
| 555 |
+
row = {
|
| 556 |
+
"episode_index": episode_index,
|
| 557 |
+
"local_episode_index": local_episode_index,
|
| 558 |
+
"worker_id": args.worker_id,
|
| 559 |
+
"episode_seed": args.seed,
|
| 560 |
+
"canonical_episode_id": f"state_replay_episode_{episode_index:06d}",
|
| 561 |
+
"success": success,
|
| 562 |
+
"grasped": bool(stage["grasped"]),
|
| 563 |
+
"grasped_strict": bool(stage["grasped_strict"]),
|
| 564 |
+
"max_lift_m": round(float(_max_dz), 4),
|
| 565 |
+
"lifted": bool(stage["lifted"]),
|
| 566 |
+
"in_cab": bool(stage["in_cab"]),
|
| 567 |
+
"reward": reward_total,
|
| 568 |
+
"policy_steps": policy_steps,
|
| 569 |
+
"sim_step_budget": policy_steps * args.action_horizon,
|
| 570 |
+
"video_path": video_path,
|
| 571 |
+
"ep_meta_pickle": str(ep_meta_pickle),
|
| 572 |
+
"replay_state_npz": reset_info.get("replay_state_npz"),
|
| 573 |
+
"replay_model_xml_gz": reset_info.get("replay_model_xml_gz"),
|
| 574 |
+
"replay_state_json": reset_info.get("replay_state_json"),
|
| 575 |
+
"replay_state_flat_sha256": reset_info.get("replay_state_flat_sha256"),
|
| 576 |
+
"source_ep_meta_pickle": reset_info.get("source_ep_meta_pickle"),
|
| 577 |
+
"regenerated_ep_meta_pickle": reset_info.get("regenerated_ep_meta_pickle"),
|
| 578 |
+
"model_path": args.model_path,
|
| 579 |
+
"finetuned_task": args.finetuned_task,
|
| 580 |
+
"eval_task": args.eval_task,
|
| 581 |
+
"env_name": args.env_name,
|
| 582 |
+
"seed": args.seed,
|
| 583 |
+
**ep_summary,
|
| 584 |
+
}
|
| 585 |
+
write_jsonl(episodes_jsonl, row)
|
| 586 |
+
write_jsonl(
|
| 587 |
+
full_meta_jsonl,
|
| 588 |
+
{
|
| 589 |
+
"episode_index": episode_index,
|
| 590 |
+
"ep_meta_pickle": str(ep_meta_pickle),
|
| 591 |
+
"ep_meta": ep_meta,
|
| 592 |
+
},
|
| 593 |
+
)
|
| 594 |
+
with csv_path.open("a", newline="") as f:
|
| 595 |
+
writer = csv.DictWriter(f, fieldnames=fieldnames)
|
| 596 |
+
writer.writerow(
|
| 597 |
+
{
|
| 598 |
+
"episode_index": row["episode_index"],
|
| 599 |
+
"local_episode_index": row["local_episode_index"],
|
| 600 |
+
"worker_id": row["worker_id"],
|
| 601 |
+
"episode_seed": row["episode_seed"],
|
| 602 |
+
"success": int(success),
|
| 603 |
+
"reward": reward_total,
|
| 604 |
+
"policy_steps": policy_steps,
|
| 605 |
+
"sim_step_budget": policy_steps * args.action_horizon,
|
| 606 |
+
"video_path": video_path,
|
| 607 |
+
"layout_id": ep_summary.get("layout_id"),
|
| 608 |
+
"style_id": ep_summary.get("style_id"),
|
| 609 |
+
"language": ep_summary.get("language"),
|
| 610 |
+
"target_mesh_keys": ";".join(
|
| 611 |
+
str(obj.get("mesh_key"))
|
| 612 |
+
for obj in ep_summary.get("objects", [])
|
| 613 |
+
if obj.get("mesh_key")
|
| 614 |
+
),
|
| 615 |
+
}
|
| 616 |
+
)
|
| 617 |
+
finally:
|
| 618 |
+
env.close()
|
| 619 |
+
client.close()
|
| 620 |
+
|
| 621 |
+
summary = {
|
| 622 |
+
**run_info,
|
| 623 |
+
"finished_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
|
| 624 |
+
"n_episodes": len(successes),
|
| 625 |
+
"n_success": int(sum(successes)),
|
| 626 |
+
"success_rate": float(sum(successes) / max(len(successes), 1)),
|
| 627 |
+
}
|
| 628 |
+
(out_dir / "summary.json").write_text(json.dumps(summary, indent=2, sort_keys=True) + "\n")
|
| 629 |
+
print(json.dumps(summary, indent=2, sort_keys=True))
|
| 630 |
+
|
| 631 |
+
|
| 632 |
+
if __name__ == "__main__":
|
| 633 |
+
main()
|
eval/organize_videos.py
ADDED
|
@@ -0,0 +1,47 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
"""Put center and wrist side by side under one folder per global episode.
|
| 3 |
+
|
| 4 |
+
RecordVideo names its file by the worker-local index (agentview_center); the patched client
|
| 5 |
+
writes the wrist view under the global index. simulation_results.csv carries the mapping.
|
| 6 |
+
"""
|
| 7 |
+
import argparse, csv, os, shutil, sys
|
| 8 |
+
|
| 9 |
+
ap = argparse.ArgumentParser()
|
| 10 |
+
ap.add_argument("--root", required=True)
|
| 11 |
+
a = ap.parse_args()
|
| 12 |
+
|
| 13 |
+
csv_path = os.path.join(a.root, "simulation_results.csv")
|
| 14 |
+
if not os.path.exists(csv_path):
|
| 15 |
+
sys.exit(f"missing {csv_path}")
|
| 16 |
+
|
| 17 |
+
out = os.path.join(a.root, "rollouts")
|
| 18 |
+
os.makedirs(out, exist_ok=True)
|
| 19 |
+
moved = missing = 0
|
| 20 |
+
for r in csv.DictReader(open(csv_path)):
|
| 21 |
+
ep = int(r["episode_index"])
|
| 22 |
+
d = os.path.join(out, f"episode_{ep:06d}")
|
| 23 |
+
os.makedirs(d, exist_ok=True)
|
| 24 |
+
|
| 25 |
+
center = r.get("video_path", "")
|
| 26 |
+
if center and os.path.exists(center):
|
| 27 |
+
shutil.copy2(center, os.path.join(d, "center.mp4"))
|
| 28 |
+
moved += 1
|
| 29 |
+
else:
|
| 30 |
+
missing += 1
|
| 31 |
+
|
| 32 |
+
# the client wrote videos/episode_NNNNNN/wrist.mp4 inside that episode's worker dir
|
| 33 |
+
wd = os.path.dirname(os.path.dirname(center)) if center else ""
|
| 34 |
+
wrist = os.path.join(wd, "videos", f"episode_{ep:06d}", "wrist.mp4")
|
| 35 |
+
if os.path.exists(wrist):
|
| 36 |
+
shutil.copy2(wrist, os.path.join(d, "wrist.mp4"))
|
| 37 |
+
else:
|
| 38 |
+
missing += 1
|
| 39 |
+
|
| 40 |
+
with open(os.path.join(d, "info.txt"), "w") as f:
|
| 41 |
+
f.write(f"episode_index {ep}\n"
|
| 42 |
+
f"success {r['success']}\n"
|
| 43 |
+
f"target {r['target_mesh_keys'].split(';')[0]}\n"
|
| 44 |
+
f"language {r['language']}\n"
|
| 45 |
+
f"policy_steps {r['policy_steps']}\n"
|
| 46 |
+
f"layout/style {r['layout_id']}/{r['style_id']}\n")
|
| 47 |
+
print(f"{moved} episodes organised under {out} (missing files: {missing})")
|
eval/run_grasp_sample.sh
ADDED
|
@@ -0,0 +1,32 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env bash
|
| 2 |
+
# 40-episode sample (5 per object) with stage logging. Two models in parallel, 2 GPUs each,
|
| 3 |
+
# staying off GPU 0/1 and leaving 2,3 to the running training.
|
| 4 |
+
set -u
|
| 5 |
+
M=/data/minha/pose6daug/train_robocasa/myGR00T
|
| 6 |
+
PY=/data/nvidia/gripper_augmentator/conda-envs/mygr00t/bin/python
|
| 7 |
+
RPY=/lp-dev/jonghoon/robocasa_calib/envs/robocasa/bin/python
|
| 8 |
+
PP=/lp-dev/jonghoon/robocasa_calib/repos/robosuite_robocasa_dev:/lp-dev/jonghoon/robocasa_calib/repos/robocasa:$M
|
| 9 |
+
RS=/lp-dev/jonghoon/isaac-gr00t/eval_results/pnpcountertocab_mimicgen8_exact160_replay_policyseed12345_20260506
|
| 10 |
+
OUT=/home/nvidia/minha/pose6daug/evals/results/grasp_sample40
|
| 11 |
+
EPS="0,1,2,3,4,20,21,22,23,24,40,41,42,43,44,60,61,62,63,64,80,81,82,83,84,100,101,102,103,104,120,121,122,123,124,140,141,142,143,144"
|
| 12 |
+
|
| 13 |
+
one () { # tag, model, gpu, port
|
| 14 |
+
local tag=$1 model=$2 gpu=$3 port=$4
|
| 15 |
+
local d=$OUT/$tag; mkdir -p $d
|
| 16 |
+
( cd $M; PYTHONPATH=$M CUDA_VISIBLE_DEVICES=$gpu $PY scripts/inference_service.py --server --port $port --model_path $model --data_config single_panda_gripper --embodiment_tag new_embodiment --seed 12345 ) > $d/server.log 2>&1 &
|
| 17 |
+
echo $! > $d/server.pid
|
| 18 |
+
for i in $(seq 1 120); do
|
| 19 |
+
python3 -c "import socket,sys;s=socket.socket();sys.exit(0 if s.connect_ex(('127.0.0.1',$port))==0 else 1)" && break
|
| 20 |
+
sleep 5
|
| 21 |
+
done
|
| 22 |
+
echo "[grasp] $tag server ready $(date -u +%H:%M:%S)"
|
| 23 |
+
PYTHONPATH=$PP CUDA_VISIBLE_DEVICES=$gpu $RPY $M/scripts/eval_robocasa_replay_state_grasp.py --host 127.0.0.1 --port $port --model_path $model --output_dir $d --env_name PickPlaceCounterToCabinet --eval_task PnPCounterToCab --finetuned_task grasp-sample-$tag --replay_state_root $RS --episode_indices "$EPS" --worker_id 00 --seed 42 --max_episode_steps 750 --action_horizon 16 --robots PandaOmron --generative_textures --policy_seed 12345 > $d/client.log 2>&1
|
| 24 |
+
echo "[grasp] $tag done $(date -u +%H:%M:%S)"
|
| 25 |
+
kill $(cat $d/server.pid) 2>/dev/null
|
| 26 |
+
}
|
| 27 |
+
|
| 28 |
+
one mimicgen30k /data/minha/pose6daug/train_robocasa/runs/pnpcountertocab_mimicgen_natural_256_fixed_newemb_from60k_gbs64_30000steps_20260920_133919/checkpoint-30000 4 8851 &
|
| 29 |
+
sleep 10
|
| 30 |
+
one vace30k /data/minha/pose6daug/train_robocasa/runs/vace_batch64_30k_from_hf/checkpoint-30000 6 8861 &
|
| 31 |
+
wait
|
| 32 |
+
echo "[grasp] ALL DONE $(date -u +%H:%M:%S)"
|
tools/ACTION_LAYOUT.md
ADDED
|
@@ -0,0 +1,103 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Action-layout check for gr00t_views datasets
|
| 2 |
+
|
| 3 |
+
Found 2026-09-20 while investigating a 0/160 eval. Use it on any gr00t_views dataset before
|
| 4 |
+
training on it.
|
| 5 |
+
|
| 6 |
+
## The bug
|
| 7 |
+
|
| 8 |
+
RoboCasa's simulator emits a 12-d action as
|
| 9 |
+
|
| 10 |
+
```
|
| 11 |
+
[eef_pos(3), eef_rot(3), gripper(1), base(3), torso(1), base_mode(1)] robosuite order
|
| 12 |
+
```
|
| 13 |
+
|
| 14 |
+
A gr00t_views dataset's `meta/modality.json` declares
|
| 15 |
+
|
| 16 |
+
```
|
| 17 |
+
base_motion[0:4] control_mode[4:5] eef_pos[5:8] eef_rot[8:11] gripper_close[11:12]
|
| 18 |
+
```
|
| 19 |
+
|
| 20 |
+
which is a **different order**. A builder that copies simulator actions straight into the
|
| 21 |
+
parquet writes correct numbers under the wrong column names. Nothing errors, the loss
|
| 22 |
+
converges to a small value, and the resulting policy is useless: the slices land as
|
| 23 |
+
|
| 24 |
+
| modality.json reads | what is actually there |
|
| 25 |
+
|---|---|
|
| 26 |
+
| `base_motion[0:4]` | eef x, y, z and rot x |
|
| 27 |
+
| `control_mode[4:5]` | rot y |
|
| 28 |
+
| `eef_pos[5:8]` | rot z, **gripper**, 0 |
|
| 29 |
+
| `eef_rot[8:11]` | 0, 0, 0 |
|
| 30 |
+
| `gripper_close[11:12]` | base_mode, a constant |
|
| 31 |
+
|
| 32 |
+
So the model is trained to emit arm motion on the base-motion channel and a constant on the
|
| 33 |
+
gripper channel. At eval the robot drives its base away from the counter and never closes the
|
| 34 |
+
gripper. Measured: 0/160 on `pnpcountertocab_mimicgen8_exact160` for a checkpoint whose base
|
| 35 |
+
scored 15/160 on the same episodes.
|
| 36 |
+
|
| 37 |
+
## How the check works
|
| 38 |
+
|
| 39 |
+
No reference dataset needed. In this task the mobile base never moves, so the two layouts are
|
| 40 |
+
distinguishable by which dimensions are constant:
|
| 41 |
+
|
| 42 |
+
```
|
| 43 |
+
LeRobot dims 0-3 all zero, dim 4 constant, dim 11 two-valued (the gripper)
|
| 44 |
+
robosuite dims 0-5 continuous, dim 6 two-valued (the gripper), dims 7-10 zero, dim 11 constant
|
| 45 |
+
```
|
| 46 |
+
|
| 47 |
+
## Use
|
| 48 |
+
|
| 49 |
+
```bash
|
| 50 |
+
python check_action_layout.py --dataset <gr00t_views dataset> [--episodes 20]
|
| 51 |
+
```
|
| 52 |
+
|
| 53 |
+
Exit codes: `0` PASS (LeRobot), `1` FAIL (robosuite order), `2` not 12-d, `3` neither matched.
|
| 54 |
+
|
| 55 |
+
A `3` on a small sample can just mean the sampled episodes are degenerate -- re-run with a
|
| 56 |
+
larger `--episodes` before concluding anything.
|
| 57 |
+
|
| 58 |
+
```
|
| 59 |
+
$ python check_action_layout.py --dataset .../mimicgen_natural_256
|
| 60 |
+
dim 6 gripper -1.000 .. 1.000 uniq 2 <- two-valued
|
| 61 |
+
dim11 base_mode -1.000 .. -1.000 uniq 1 <- constant
|
| 62 |
+
FAIL action column is in RoboCasa/robosuite order but modality.json declares LeRobot order.
|
| 63 |
+
Repair: actions = actions[:, [7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6]]
|
| 64 |
+
|
| 65 |
+
$ python check_action_layout.py --dataset .../pickplace_target_human/PickPlaceCounterToCabinet
|
| 66 |
+
PASS action column is in LeRobot order, matching modality.json.
|
| 67 |
+
```
|
| 68 |
+
|
| 69 |
+
## Fix
|
| 70 |
+
|
| 71 |
+
In the builder, reorder before writing the parquet:
|
| 72 |
+
|
| 73 |
+
```python
|
| 74 |
+
# robosuite [eef_pos3, eef_rot3, gripper, base3, torso, base_mode]
|
| 75 |
+
# -> LeRobot [base3+torso, base_mode, eef_pos3, eef_rot3, gripper]
|
| 76 |
+
actions = all_actions[:, [7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6]]
|
| 77 |
+
```
|
| 78 |
+
|
| 79 |
+
Rebuild the dataset rather than patching parquet in place, unless you have confirmed nothing
|
| 80 |
+
else already consumed it.
|
| 81 |
+
|
| 82 |
+
## Which builders are affected
|
| 83 |
+
|
| 84 |
+
| builder | source of `action` | affected |
|
| 85 |
+
|---|---|---|
|
| 86 |
+
| `baseline/mimicgen/gr00t_build/render_to_gr00t.py` (line 180) | `demo_grp["actions"]` from the MimicGen HDF5, simulator order | **yes** |
|
| 87 |
+
| `train_robocasa/scripts/dataset_build/build_vace_objwise_gr00t_dataset.py` | `pd.read_parquet(src_parquet)` from the original dataset, already LeRobot order | no |
|
| 88 |
+
|
| 89 |
+
The rule of thumb: a builder that **re-derives** actions from a simulator rollout needs the
|
| 90 |
+
reorder; one that **copies** rows from an existing LeRobot dataset does not.
|
| 91 |
+
|
| 92 |
+
Note that `observation.state` is not affected in either builder -- `extract_obs_state()`
|
| 93 |
+
assembles it field by field (`base_pos(3) + base_quat(4) + eef_pos(3) + eef_quat(4) +
|
| 94 |
+
gripper_qpos(2)`) in the declared order, so only `action` was ever passed through raw.
|
| 95 |
+
|
| 96 |
+
## Datasets checked
|
| 97 |
+
|
| 98 |
+
| dataset | result |
|
| 99 |
+
|---|---|
|
| 100 |
+
| `baseline/mimicgen/gr00t_views/mimicgen_natural_256` | FAIL |
|
| 101 |
+
| `baseline/mimicgen/gr00t_views/mimicgen_per_target_32_256eps` | FAIL (same builder) |
|
| 102 |
+
| `robocasa_full/pickplace_target_human/PickPlaceCounterToCabinet` | PASS |
|
| 103 |
+
| any VACE gr00t_views set | expected PASS -- run the checker to confirm on that machine |
|
tools/check_action_layout.py
ADDED
|
@@ -0,0 +1,122 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
"""Check that a gr00t_views dataset's `action` column really is in the LeRobot order its
|
| 3 |
+
meta/modality.json claims.
|
| 4 |
+
|
| 5 |
+
RoboCasa's simulator emits actions as
|
| 6 |
+
[eef_pos(3), eef_rot(3), gripper(1), base(3), torso(1), base_mode(1)]
|
| 7 |
+
while a gr00t_views dataset declares
|
| 8 |
+
[base_motion(4), control_mode(1), eef_pos(3), eef_rot(3), gripper_close(1)]
|
| 9 |
+
|
| 10 |
+
Any builder that copies simulator actions through without reordering produces a dataset whose
|
| 11 |
+
numbers are right but whose columns are mislabelled. Training then puts arm motion on the
|
| 12 |
+
base-motion channel and a constant on the gripper channel, so the policy drives the robot away
|
| 13 |
+
and never closes the gripper. The loss still converges -- nothing errors -- which is why this
|
| 14 |
+
has to be checked explicitly.
|
| 15 |
+
|
| 16 |
+
The two layouts are told apart by which dimensions are constant, no reference dataset needed:
|
| 17 |
+
|
| 18 |
+
LeRobot dims 0-3 all zero, dim 4 constant, dim 11 two-valued (the gripper)
|
| 19 |
+
robosuite dims 0-5 continuous, dim 6 two-valued (the gripper), dims 7-10 zero, dim 11 constant
|
| 20 |
+
|
| 21 |
+
python check_action_layout.py --dataset <gr00t_views dataset> [--episodes 20]
|
| 22 |
+
"""
|
| 23 |
+
from __future__ import annotations
|
| 24 |
+
|
| 25 |
+
import argparse
|
| 26 |
+
import glob
|
| 27 |
+
import json
|
| 28 |
+
import os
|
| 29 |
+
import sys
|
| 30 |
+
|
| 31 |
+
import numpy as np
|
| 32 |
+
import pyarrow.parquet as pq
|
| 33 |
+
|
| 34 |
+
LEROBOT_NAMES = ["base_x", "base_y", "base_th", "torso", "ctrl_mode",
|
| 35 |
+
"eef_x", "eef_y", "eef_z", "rot_x", "rot_y", "rot_z", "gripper"]
|
| 36 |
+
ROBOSUITE_NAMES = ["eef_x", "eef_y", "eef_z", "rot_x", "rot_y", "rot_z",
|
| 37 |
+
"gripper", "base_x", "base_y", "base_th", "torso", "base_mode"]
|
| 38 |
+
# robosuite index -> LeRobot position; apply as actions[:, FIX] to repair a dataset.
|
| 39 |
+
FIX = [7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6]
|
| 40 |
+
|
| 41 |
+
|
| 42 |
+
def load_actions(dataset: str, n_episodes: int) -> np.ndarray:
|
| 43 |
+
paths = sorted(glob.glob(os.path.join(dataset, "data", "chunk-*", "*.parquet")))
|
| 44 |
+
if not paths:
|
| 45 |
+
sys.exit(f"no parquet under {dataset}/data/chunk-*/")
|
| 46 |
+
rows = []
|
| 47 |
+
for p in paths[:n_episodes]:
|
| 48 |
+
rows.append(np.stack(pq.read_table(p, columns=["action"]).column("action").to_pylist()))
|
| 49 |
+
return np.vstack(rows), len(paths), min(n_episodes, len(paths))
|
| 50 |
+
|
| 51 |
+
|
| 52 |
+
def classify(a: np.ndarray) -> tuple[str, list[str]]:
|
| 53 |
+
"""Which layout do the constant/continuous dimensions match?"""
|
| 54 |
+
const = [i for i in range(a.shape[1]) if len(np.unique(np.round(a[:, i], 4))) == 1]
|
| 55 |
+
zero = [i for i in const if abs(a[:, i]).max() < 1e-6]
|
| 56 |
+
twoval = [i for i in range(a.shape[1]) if len(np.unique(np.round(a[:, i], 4))) == 2]
|
| 57 |
+
notes = [f" all-zero dims : {zero}", f" constant dims : {const}",
|
| 58 |
+
f" two-valued : {twoval} (the gripper is two-valued)"]
|
| 59 |
+
lerobot = set([0, 1, 2, 3]).issubset(zero) and 4 in const and 11 in twoval
|
| 60 |
+
robosuite = set([7, 8, 9, 10]).issubset(zero) and 11 in const and 6 in twoval
|
| 61 |
+
if lerobot and not robosuite:
|
| 62 |
+
return "lerobot", notes
|
| 63 |
+
if robosuite and not lerobot:
|
| 64 |
+
return "robosuite", notes
|
| 65 |
+
return "unknown", notes
|
| 66 |
+
|
| 67 |
+
|
| 68 |
+
def main() -> int:
|
| 69 |
+
ap = argparse.ArgumentParser(description=__doc__,
|
| 70 |
+
formatter_class=argparse.RawDescriptionHelpFormatter)
|
| 71 |
+
ap.add_argument("--dataset", required=True, help="gr00t_views dataset directory")
|
| 72 |
+
ap.add_argument("--episodes", type=int, default=20, help="episodes to sample (default 20)")
|
| 73 |
+
a = ap.parse_args()
|
| 74 |
+
|
| 75 |
+
actions, n_total, n_read = load_actions(a.dataset, a.episodes)
|
| 76 |
+
print(f"dataset : {a.dataset}")
|
| 77 |
+
print(f"sampled : {n_read} of {n_total} episodes, {actions.shape[0]} frames x {actions.shape[1]} dims\n")
|
| 78 |
+
|
| 79 |
+
if actions.shape[1] != 12:
|
| 80 |
+
print(f"FAIL expected 12 action dims, found {actions.shape[1]}")
|
| 81 |
+
return 2
|
| 82 |
+
|
| 83 |
+
verdict, notes = classify(actions)
|
| 84 |
+
names = LEROBOT_NAMES if verdict == "lerobot" else ROBOSUITE_NAMES
|
| 85 |
+
for i in range(12):
|
| 86 |
+
col = actions[:, i]
|
| 87 |
+
uniq = len(np.unique(np.round(col, 4)))
|
| 88 |
+
tag = " <- constant" if uniq == 1 else (" <- two-valued" if uniq == 2 else "")
|
| 89 |
+
label = names[i] if verdict != "unknown" else "?"
|
| 90 |
+
print(f" dim{i:2d} {label:10s} {col.min():8.3f} .. {col.max():7.3f} "
|
| 91 |
+
f"|mean| {abs(col).mean():6.3f} uniq {uniq:5d}{tag}")
|
| 92 |
+
print()
|
| 93 |
+
print("\n".join(notes))
|
| 94 |
+
print()
|
| 95 |
+
|
| 96 |
+
modality = os.path.join(a.dataset, "meta", "modality.json")
|
| 97 |
+
if os.path.exists(modality):
|
| 98 |
+
act = json.load(open(modality)).get("action", {})
|
| 99 |
+
print(" modality.json action slices: "
|
| 100 |
+
+ ", ".join(f"{k} [{v['start']}:{v['end']}]" for k, v in act.items()))
|
| 101 |
+
print()
|
| 102 |
+
|
| 103 |
+
if verdict == "lerobot":
|
| 104 |
+
print("PASS action column is in LeRobot order, matching modality.json.")
|
| 105 |
+
return 0
|
| 106 |
+
if verdict == "robosuite":
|
| 107 |
+
print("FAIL action column is in RoboCasa/robosuite order but modality.json declares")
|
| 108 |
+
print(" LeRobot order. Training would read arm motion as base motion and a")
|
| 109 |
+
print(" constant as the gripper.")
|
| 110 |
+
print(f" Repair: actions = actions[:, {FIX}]")
|
| 111 |
+
print(" Fix the builder and rebuild; do not patch the parquet in place unless you")
|
| 112 |
+
print(" also confirm nothing else consumed it.")
|
| 113 |
+
return 1
|
| 114 |
+
print("UNKNOWN neither layout matched. Inspect the dimensions above by hand:")
|
| 115 |
+
print(" LeRobot expects dims 0-3 zero, dim 4 constant, dim 11 two-valued.")
|
| 116 |
+
print(" An all-constant or degenerate episode sample can also land here --")
|
| 117 |
+
print(" re-run with a larger --episodes before concluding.")
|
| 118 |
+
return 3
|
| 119 |
+
|
| 120 |
+
|
| 121 |
+
if __name__ == "__main__":
|
| 122 |
+
raise SystemExit(main())
|
train/smoke_test.sh
ADDED
|
@@ -0,0 +1,38 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env bash
|
| 2 |
+
# Verifies that GR00T 1.5 fine-tuning runs from THIS bundle: imports, dataset load,
|
| 3 |
+
# then a REAL 20-step single-GPU fine-tune on the 150-episode augmented dataset
|
| 4 |
+
# (per-GPU batch 2, no LoRA, same code path as the full run) writing a checkpoint
|
| 5 |
+
# into runs/smoke_<timestamp>/. Needs one GPU with ~25 GB free.
|
| 6 |
+
# ./smoke_test.sh <gpu_index> e.g. ./smoke_test.sh 3
|
| 7 |
+
set -euo pipefail
|
| 8 |
+
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"; B="$(dirname "$HERE")"
|
| 9 |
+
GPU="${1:?usage: smoke_test.sh <gpu_index>}"
|
| 10 |
+
PY=/data/nvidia/gripper_augmentator/conda-envs/mygr00t/bin/python
|
| 11 |
+
DS="${DATASET_PATH:-/lp-dev/jonghoon/robocasa_full/gr00t_views/PickPlaceCounterToCabinet_150eps_augmented_groot15_v3_100_clean}"
|
| 12 |
+
CK="${BASE_CKPT:-/lp-dev/jonghoon/myGR00T_outputs/pnpcountertocab_all502_gbs64_wandb_60k_save5k_20260427_193901/checkpoint-60000}"
|
| 13 |
+
pass(){ echo " PASS $*"; }; fail(){ echo " FAIL $*"; exit 1; }
|
| 14 |
+
export PYTHONPATH="$B/myGR00T" HF_HOME=/lp-dev/jonghoon/.cache/huggingface HF_HUB_OFFLINE=1 NO_ALBUMENTATIONS_UPDATE=1
|
| 15 |
+
|
| 16 |
+
echo "== static =="
|
| 17 |
+
[ -x "$PY" ] && pass "env python $PY" || fail "mygr00t env missing"
|
| 18 |
+
(cd "$B/myGR00T" && "$PY" -c "import gr00t, os; assert gr00t.__file__.startswith('$B/'), gr00t.__file__; from gr00t.experiment.data_config import DATA_CONFIG_MAP; assert 'single_panda_gripper' in DATA_CONFIG_MAP") && pass "gr00t imports from the bundle, data config single_panda_gripper registered" || fail "gr00t import"
|
| 19 |
+
[ -f "$DS/meta/modality.json" ] && pass "dataset $DS" || fail "dataset missing"
|
| 20 |
+
[ -f "$CK/model.safetensors.index.json" ] && pass "base checkpoint $CK" || fail "checkpoint missing"
|
| 21 |
+
(cd "$B/myGR00T" && "$PY" - <<PYEOF
|
| 22 |
+
from gr00t.data.dataset import LeRobotSingleDataset
|
| 23 |
+
from gr00t.data.schema import EmbodimentTag
|
| 24 |
+
from gr00t.experiment.data_config import DATA_CONFIG_MAP
|
| 25 |
+
dc = DATA_CONFIG_MAP["single_panda_gripper"]
|
| 26 |
+
ds = LeRobotSingleDataset("$DS", dc.modality_config(), transforms=dc.transform(), embodiment_tag=EmbodimentTag("new_embodiment"), video_backend="decord")
|
| 27 |
+
s = ds[0]; print({k: tuple(v.shape) for k, v in s.items() if hasattr(v, "shape")}); assert len(ds) > 1000
|
| 28 |
+
PYEOF
|
| 29 |
+
) && pass "LeRobotSingleDataset loads a sample (videos decoded)" || fail "dataset load"
|
| 30 |
+
|
| 31 |
+
echo "== 20-step fine-tune on GPU $GPU =="
|
| 32 |
+
RUN=smoke_$(date -u +%Y%m%d_%H%M%S); OUT="$B/runs/$RUN"
|
| 33 |
+
CUDA_VISIBLE_DEVICES="$GPU" GPUS="$GPU" NUM_GPUS=1 PER_GPU_BATCH=2 MAX_STEPS=20 SAVE_STEPS=20 DATALOADER_NUM_WORKERS=4 \
|
| 34 |
+
REPORT_TO=tensorboard RUN_NAME="$RUN" OUTPUT_DIR="$OUT" DATASET_PATH="$DS" BASE_CKPT="$CK" \
|
| 35 |
+
bash "$HERE/train_groot15_single_dataset.sh" > "$B/logs/$RUN.log" 2>&1 || { tail -30 "$B/logs/$RUN.log"; fail "fine-tune exited non-zero (log: $B/logs/$RUN.log)"; }
|
| 36 |
+
[ -f "$OUT/checkpoint-20/model.safetensors.index.json" ] || [ -f "$OUT/checkpoint-20/model.safetensors" ] && pass "checkpoint-20 written to $OUT" || fail "no checkpoint-20 in $OUT"
|
| 37 |
+
grep -q "train_loss\|'loss'" "$B/logs/$RUN.log" && pass "loss logged: $(grep -o "'loss': [0-9.]*" "$B/logs/$RUN.log" | tail -1)" || pass "run finished (no loss line found in log, check $B/logs/$RUN.log)"
|
| 38 |
+
echo "ALL PASSED (log: $B/logs/$RUN.log)"
|
train/train_groot15_single_dataset.sh
ADDED
|
@@ -0,0 +1,116 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env bash
|
| 2 |
+
set -euo pipefail
|
| 3 |
+
|
| 4 |
+
# Self-contained bundle version (2026-09-19): code = ../myGR00T (this bundle), outputs = ../runs.
|
| 5 |
+
# Only the conda env, the LeRobot dataset and the base checkpoint are referenced by absolute path.
|
| 6 |
+
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
| 7 |
+
BUNDLE="$(dirname "$HERE")"
|
| 8 |
+
|
| 9 |
+
# Set this to the dataset directory name under TRAIN_DATASET_BASE.
|
| 10 |
+
DATASETNAME="${DATASETNAME:-PickPlaceCounterToCabinet_150eps_augmented_groot15_v3_100_clean}"
|
| 11 |
+
|
| 12 |
+
MYGROOT_ROOT="${MYGROOT_ROOT:-$BUNDLE/myGR00T}"
|
| 13 |
+
PYTHON_BIN="${PYTHON_BIN:-/data/nvidia/gripper_augmentator/conda-envs/mygr00t/bin/python}"
|
| 14 |
+
CONDA_BIN="${CONDA_BIN:-/data/nvidia/gripper_augmentator/conda-envs/mygr00t/bin}"
|
| 15 |
+
TRAIN_DATASET_BASE="${TRAIN_DATASET_BASE:-/lp-dev/jonghoon/robocasa_full/gr00t_views}"
|
| 16 |
+
OUTPUT_BASE="${OUTPUT_BASE:-$BUNDLE/runs}"
|
| 17 |
+
BASE_CKPT="${BASE_CKPT:-/lp-dev/jonghoon/myGR00T_outputs/pnpcountertocab_all502_gbs64_wandb_60k_save5k_20260427_193901/checkpoint-60000}"
|
| 18 |
+
|
| 19 |
+
TASK_SLUG="${TASK_SLUG:-pnpcountertocab}"
|
| 20 |
+
RUN_ID="${RUN_ID:-$(date -u +%Y%m%d_%H%M%S)}"
|
| 21 |
+
MAX_STEPS="${MAX_STEPS:-30000}"
|
| 22 |
+
SAVE_STEPS="${SAVE_STEPS:-5000}"
|
| 23 |
+
RESUME="${RESUME:-0}"
|
| 24 |
+
GPUS="${GPUS:-0,1,2,3}"
|
| 25 |
+
PER_GPU_BATCH="${PER_GPU_BATCH:-16}"
|
| 26 |
+
DATALOADER_NUM_WORKERS="${DATALOADER_NUM_WORKERS:-8}"
|
| 27 |
+
DATA_CONFIG="${DATA_CONFIG:-single_panda_gripper}"
|
| 28 |
+
EMBODIMENT_TAG="${EMBODIMENT_TAG:-new_embodiment}"
|
| 29 |
+
BACKBONE_MODEL_TYPE="${BACKBONE_MODEL_TYPE:-eagle}"
|
| 30 |
+
BACKBONE_SELECT_LAYER="${BACKBONE_SELECT_LAYER:-12}"
|
| 31 |
+
REPORT_TO="${REPORT_TO:-tensorboard}" # wandb needs WANDB_API_KEY (or WANDB_API_KEY_FILE)
|
| 32 |
+
|
| 33 |
+
DATASET_PATH="${DATASET_PATH:-${TRAIN_DATASET_BASE}/${DATASETNAME}}"
|
| 34 |
+
RUN_NAME="${RUN_NAME:-${TASK_SLUG}_${DATASETNAME}_newemb_from60k_gbs64_${MAX_STEPS}steps_${RUN_ID}}"
|
| 35 |
+
OUTPUT_DIR="${OUTPUT_DIR:-${OUTPUT_BASE}/${RUN_NAME}}"
|
| 36 |
+
|
| 37 |
+
export CUDA_VISIBLE_DEVICES="${CUDA_VISIBLE_DEVICES:-$GPUS}"
|
| 38 |
+
export PATH="${CONDA_BIN}:${PATH}"
|
| 39 |
+
export WANDB_PROJECT="${WANDB_PROJECT:-gr00t-robocasa}"
|
| 40 |
+
export WANDB_ENTITY="${WANDB_ENTITY:-alphabet1}"
|
| 41 |
+
export WANDB_DIR="${WANDB_DIR:-$BUNDLE/logs/wandb}"
|
| 42 |
+
export WANDB_CACHE_DIR="${WANDB_CACHE_DIR:-$BUNDLE/logs/wandb/cache}"
|
| 43 |
+
export WANDB_CONFIG_DIR="${WANDB_CONFIG_DIR:-$BUNDLE/logs/wandb/config}"
|
| 44 |
+
export NO_ALBUMENTATIONS_UPDATE=1
|
| 45 |
+
# The mygr00t env's editable install of `gr00t` points at a deleted checkout; import the bundle copy instead.
|
| 46 |
+
export PYTHONPATH="$MYGROOT_ROOT${PYTHONPATH:+:$PYTHONPATH}"
|
| 47 |
+
export HF_HOME="${HF_HOME:-/lp-dev/jonghoon/.cache/huggingface}"
|
| 48 |
+
export HF_HUB_OFFLINE="${HF_HUB_OFFLINE:-1}"
|
| 49 |
+
export TOKENIZERS_PARALLELISM=false
|
| 50 |
+
|
| 51 |
+
# Optional, safer than hard-coding a secret in this script.
|
| 52 |
+
WANDB_API_KEY_FILE="${WANDB_API_KEY_FILE:-}"
|
| 53 |
+
if [[ -z "${WANDB_API_KEY:-}" && -n "$WANDB_API_KEY_FILE" && -f "$WANDB_API_KEY_FILE" ]]; then
|
| 54 |
+
WANDB_API_KEY="$(tr -d '[:space:]' < "$WANDB_API_KEY_FILE")"
|
| 55 |
+
export WANDB_API_KEY
|
| 56 |
+
fi
|
| 57 |
+
|
| 58 |
+
IFS=',' read -r -a GPU_LIST <<< "$CUDA_VISIBLE_DEVICES"
|
| 59 |
+
NUM_GPUS="${NUM_GPUS:-${#GPU_LIST[@]}}"
|
| 60 |
+
|
| 61 |
+
if [[ ! -d "$DATASET_PATH" ]]; then
|
| 62 |
+
echo "FATAL: dataset path does not exist: $DATASET_PATH" >&2
|
| 63 |
+
exit 1
|
| 64 |
+
fi
|
| 65 |
+
if [[ ! -f "$BASE_CKPT/model.safetensors.index.json" ]]; then
|
| 66 |
+
echo "FATAL: checkpoint is missing or incomplete: $BASE_CKPT" >&2
|
| 67 |
+
exit 1
|
| 68 |
+
fi
|
| 69 |
+
|
| 70 |
+
mkdir -p "$OUTPUT_DIR" "$WANDB_DIR" "$WANDB_CACHE_DIR" "$WANDB_CONFIG_DIR"
|
| 71 |
+
cd "$MYGROOT_ROOT"
|
| 72 |
+
|
| 73 |
+
exec > >(tee -a "$OUTPUT_DIR/train.log") 2>&1
|
| 74 |
+
|
| 75 |
+
echo "run_name=$RUN_NAME"
|
| 76 |
+
echo "output_dir=$OUTPUT_DIR"
|
| 77 |
+
echo "dataset_name=$DATASETNAME"
|
| 78 |
+
echo "dataset_path=$DATASET_PATH"
|
| 79 |
+
echo "base_ckpt=$BASE_CKPT"
|
| 80 |
+
echo "cuda_visible_devices=$CUDA_VISIBLE_DEVICES"
|
| 81 |
+
echo "num_gpus=$NUM_GPUS"
|
| 82 |
+
echo "per_gpu_batch=$PER_GPU_BATCH"
|
| 83 |
+
echo "global_batch=$((NUM_GPUS * PER_GPU_BATCH))"
|
| 84 |
+
echo "max_steps=$MAX_STEPS"
|
| 85 |
+
echo "save_steps=$SAVE_STEPS"
|
| 86 |
+
echo "resume=$RESUME"
|
| 87 |
+
date -u
|
| 88 |
+
|
| 89 |
+
RESUME_ARGS=()
|
| 90 |
+
if [[ "$RESUME" == "1" ]]; then
|
| 91 |
+
RESUME_ARGS+=(--resume)
|
| 92 |
+
fi
|
| 93 |
+
|
| 94 |
+
SEED_ARGS=()
|
| 95 |
+
if [[ -n "${DATA_SEED:-}" ]]; then
|
| 96 |
+
SEED_ARGS+=(--data-seed "$DATA_SEED")
|
| 97 |
+
fi
|
| 98 |
+
|
| 99 |
+
exec "$PYTHON_BIN" "$MYGROOT_ROOT/scripts/gr00t_finetune.py" \
|
| 100 |
+
--run-name "$RUN_NAME" \
|
| 101 |
+
--dataset-path "$DATASET_PATH" \
|
| 102 |
+
--output-dir "$OUTPUT_DIR" \
|
| 103 |
+
--data-config "$DATA_CONFIG" \
|
| 104 |
+
--dataloader-num-workers "$DATALOADER_NUM_WORKERS" \
|
| 105 |
+
--embodiment-tag "$EMBODIMENT_TAG" \
|
| 106 |
+
--backbone-model-type "$BACKBONE_MODEL_TYPE" \
|
| 107 |
+
--backbone-select-layer "$BACKBONE_SELECT_LAYER" \
|
| 108 |
+
--base-model-path "$BASE_CKPT" \
|
| 109 |
+
--num-gpus "$NUM_GPUS" \
|
| 110 |
+
--batch-size "$PER_GPU_BATCH" \
|
| 111 |
+
--max-steps "$MAX_STEPS" \
|
| 112 |
+
--save-steps "$SAVE_STEPS" \
|
| 113 |
+
--report-to "$REPORT_TO" \
|
| 114 |
+
"${SEED_ARGS[@]}" \
|
| 115 |
+
"${RESUME_ARGS[@]}" \
|
| 116 |
+
"$@"
|