yqi19/mp_yam_code

vfm-dexmimicgen

Rebase to IsaacLab External

[Updated 07/22/2026]

Installation

1. Install IsaacLab 's release 3.0.0-beta2

Activate your env_isaaclab

# e.g.
source <your_path_to>/env_isaaclab/bin/activate

## e.g. 
# source /home/yizhou/Projects/IsaacLab/env_isaaclab/bin/activate

2. Install repo

# install
python -m pip install -e .

## or 
# uv pip install -e .

Usage

1. List env

python scripts/list_envs.py

2.zero-agent

python scripts/zero_agent.py --task Template-Sdg-Mimic-Gen-v0 --viz kit

3. link asset

ln -s /path/to/dexmimicgen_custom_assets ./custom_assets

##  e.g. 
# ln -s /home/yizhou/Downloads/dexmimicgen_custom_assets ./custom_assets

Teleop

Install Isaac Teleop under any python env

# From PyPI
pip install 'isaacteleop[cloudxr,retargeters]~=1.3.131' --extra-index-url https://pypi.nvidia.com

Start

python -m isaacteleop.cloudxr --accept-eula

Open the Web Client

Activate CloudXR environment in another terminal in your IsaacLab env:

source ~/.cloudxr/run/cloudxr.env
## e.g.
# source /home/yizhou/.cloudxr/run/cloudxr.env

python scripts/teleop_se3_agent_bimanual_xr.py \
  --task Template-YAM-Play-v0 \
  --teleop_device motion_controllers \
  --enable_cameras \
  --viz kit \
  --num_envs 1 \
  --enable_gripper \
  --xr 

# python scripts/teleop_se3_agent_bimanual_xr.py \
#   --task Template-UR10-Play-v0 \
#   --teleop_device motion_controllers \
#   --enable_cameras \
#   --viz kit \
#   --num_envs 1 \
#   --xr \
#   --reverse_rotation_yz

---

YAM bimanual task suite (30 tasks)

Scripted manipulation tasks laid out the way ManiSkill lays out its tasks: one registered class per task file, with the environment, the solvers and the motion planner as separate layers. Built on RoboTwin 2.0 assets.

source/bimanual/yam/
  motion/     how the robot MOVES              arm.py · planner.py · recorder.py
  envs/       the world a task runs in         base_env.py · scene.py · config.py
  solvers/    scripted skills, one per file    pick_place · multi_pick · insert · stack ·
                                               dual_lift · push · handover · tool_use · pour ·
                                               pull · sort · shelf · articulate
  tasks/      ONE FILE PER TASK, registered by name
    configs/  one YAML per task (parameter overrides)
  registry.py · conditions.py
scripts/      runners, asset converters, verification

What this suite supports

30 scripted bimanual manipulation tasks on the YAM arm in Isaac Lab, built on RoboTwin 2.0 assets. Not a policy — every task ships a scripted solver, so it produces demonstrations and a pass/fail signal you can trust.

capability	what it means
Skill families	pick-and-place · multi-object · stacking · sorting · insertion · pushing (single and relay) · bimanual lift · handover · tool use (sweep, wipe) · pouring · constrained pull · articulated fixtures · deformable chain
Both arms	either arm alone, or both on one motion profile (move_both) — driving them in sequence parks the first in the second's path
Randomization	every object, fixture, container and marker takes xy_jitter / yaw_jitter; --seed reproduces an episode exactly. 52 of 56 spawn calls are randomized
Articulated props	SAPIEN mobility URDFs — hinged doors, sliding drawers, lids, toggles. Joint values are first-class state and can be success conditions
Chains / rope	a rope as a jointed chain, judged on its own geometry (end-to-end span) rather than on a placement
Configuration	class attribute → per-task YAML → --set key=value, resolved before the simulator starts
Asset pipeline	rigid → USD, hollow containers, articulated URDFs, plus procedurally generated meshes (ring, rope, pot lid)
Agentic verification	rendered contact sheets + review prompts for a vision agent, verdicts fed back into concrete parameter fixes, and task proposal from the asset library — scripts/yam_agent_loop.py
Verification	strict numeric conditions plus the visual check above; scripts/status.py regenerates the status table from run logs
Outputs	per-episode MP4 with an action/state HUD, contact sheets, single-frame scene renders

Environment

Isaac Sim 5.0 + Isaac Lab 2.2.0, Python 3.11, an NVIDIA GPU (the PhysX GPU pipeline is required).

Isaac Lab may already exist on the machine inside another project's virtualenv. If so, prefer an overlay venv — a fresh venv whose .pth file points at that interpreter's site-packages — over reinstalling Isaac Sim:

python -m venv .venv
echo "/path/to/existing/isaac/venv/lib/python3.11/site-packages" \
    > .venv/lib/python3.11/site-packages/_isaac_overlay.pth

Isaac Lab's source checkout is also needed for isaaclab_tasks / isaaclab_assets; add it the same way if it is not pip-installed.

Configure

Everything is driven by two environment variables and a relative path. Nothing below hardcodes a machine-specific location:

export ROBOTWIN_USD=./robotwin_usd          # converted USD assets (see Assets)
export ROBOTWIN_SRC=./robotwin_object/objects   # source meshes, only needed to convert
export OMNI_KIT_ACCEPT_EULA=YES

ROBOTWIN_USD defaults are read at import time, so export it before running anything. Unzip assets/robotwin_usd.zip from this repo's HF page to populate it.

Run

python scripts/yam_task.py --list
python scripts/yam_task.py --task grape_box --seed 3 \
    --video outputs/tasks/grape_box.mp4 \
    --kit_args="--/rtx/verifyDriverVersion/enabled=false"

Two flags are commonly required and easy to miss:

• OMNI_KIT_ACCEPT_EULA=YES — otherwise Omniverse blocks on an interactive EULA prompt and the run hangs forever with no output at all.
• --kit_args="--/rtx/verifyDriverVersion/enabled=false" — needed on drivers whose version string Omniverse misparses (e.g. 535.309.01 read as 535.53 and rejected as too old). The check is wrong, not the driver. Drop this flag if your driver passes the check.

Extras: --no-randomize (nominal poses, for reproducing a failure), --set gripper_effort=70 (override any task parameter), --first-frame out.png (build the scene, save one frame, exit without solving).

Config precedence: class attribute → source/bimanual/yam/tasks/configs/<task>.yaml → --set. Each task ships a YAML mirroring its parameters with every key commented out on purpose — an active key there overrides the Python and keeps overriding it, so a later code change silently does nothing.

Full architecture, task-authoring guide, the asset-gotcha table and the verification workflow: source/bimanual/yam/README.md. Every non-obvious failure and the measurement that settled it: DIAGNOSTICS.md — read this before debugging a new asset.

The one habit that matters

When a grasp or a joint misbehaves, measure the asset offline before touching a parameter. Most of the hard bugs here looked like tuning problems and were not:

# where is the graspable part, and is the mesh origin even on the object?
python -c "
import trimesh, numpy as np
m = trimesh.load('objects/083_brush/visual/base0.glb', force='mesh'); v = m.vertices
tall = int(np.argmax(m.extents)); oth = [a for a in range(3) if a != tall]
lo, hi = v[:,tall].min(), v[:,tall].max()
for i in range(8):                      # cross-section along the long axis
    s = v[(v[:,tall] >= lo+(hi-lo)*i/8) & (v[:,tall] < lo+(hi-lo)*(i+1)/8)]
    print(f'{i/8:.0%}', s[:,oth[0]].ptp(), s[:,oth[1]].ptp(), 'centroid', s[:,oth].mean(0))
"

That one command found the brush's 1 cm handle and the 3.7 cm offset between the mesh and the body origin that object_pos() reports — two separate bugs that both presented as gap 0.0 cm. The same check on the kettle found a 4.7 cm offset. For articulated assets, read the URDF's joint axis and bounding_box.json rather than inferring anything from the USD.

What actually works

Honest status, regenerated from the run logs — not a wish list. A task counts as passing only when its numeric conditions pass and the rendered contact sheet was inspected and agrees.

The numeric checks are deliberately strict, and three of them were caught lying during development (see DIAGNOSTICS.md): a pour that reported success with a failed grasp, and two joints that "moved" on gravity alone. Each was tightened rather than worked around, so several tasks moved from green to red on purpose.

Run python scripts/status.py to regenerate the table below from outputs/tasks/*.log.

Agent-in-the-loop verification

The numeric check only inspects final object poses. It will happily pass an episode where the cup landed upside-down, a gripper hung empty in the air, or a vessel was carried over the target without ever tipping — all three of those actually happened here. So every task is also reviewed visually, and only numeric=SUCCESS and visual=CONFIRMED counts as done.

python scripts/contact_sheet.py outputs/tasks/*.mp4   # episode -> one strip of frames
python scripts/yam_agent_loop.py evaluate             # contact sheets + per-task review prompts
python scripts/yam_agent_loop.py update               # verdicts -> concrete parameter fixes
python scripts/yam_agent_loop.py propose              # uncovered skill axes from the asset library

evaluate renders a sheet per episode and emits a prompt naming what to look for in that specific task (is the cup upright? is anything actually in the gripper? did the lid come off before the food went in?). update maps a verdict onto a fix through a symptom→cause table built from the failures in DIAGNOSTICS.md — e.g. "jaws closed on nothing" → measure the mesh origin offset, not the clamp force. propose reads the converted asset library and suggests skills not yet covered.

Reviewing this way is what caught the three false positives described above; the fix in each case was to tighten the numeric condition, so the visual pass and the numeric pass now agree.

18 of 30 passing.

task	status	last failure
bar_pull	pass
bell_coaster	fail
bottle_bin	pass
can_box	pass
cup_box	pass
cupboard_store	fail	object did not rise with the gripper (+0.000 m): the jaw closed beside or above
drawer_open	fail
drawer_store	fail
eraser_wipe	pass
fruit_basket	fail	can at (+0.160,-0.020,0.506) vs basket centre (+0.020,-0.240) half=0.096: dx=0.1
fruit_tray	pass
grape_box	pass
handover_basket	fail	rw_basket_dyn at (+0.058,+0.001,0.450) vs goal centre (+0.020,-0.230) half=0.145
laptop_close	fail	rw_laptop joint 0: robot moved it 0.0 (need 0.45), within limits=False
lid_food	pass
microwave_door	fail
peg_insert	pass
plank_dual_lift	pass
pot_dual_lift	fail
pour_kettle	pass
push_block	pass
push_relay	fail
ring_post	pass
rope_straighten	fail	rw_rope link7 z=-0.445 m above the table (need 0.04)
sort_cubes	pass
stack_blocks	pass
stack_cups	pass
sweep_debris	pass
switch_toggle	fail
weigh_fruit	pass

Relationship to RoboTwin's own 50 tasks

This suite uses RoboTwin's assets but is deliberately not a re-implementation of its task list. Where a task would otherwise duplicate one of theirs, it is varied on purpose — inverted, made compound, or given an extra constraint — so the two sets exercise different things.

RoboTwin task	here	what is deliberately different
Open Laptop	laptop_close	inverted — the lid is folded shut, a downward push instead of a lift
Stack Blocks Two	stack_blocks	blocks must land on a marked target region, not just on each other
Blocks Ranking RGB	sort_cubes	bimanual, each cube to a colour-matched pad on its own side of the table
Handover Block	handover_basket	a basket wider than the jaw, so it must be taken by the rim
Put Bottles Dustbin	bottle_bin	bottle lies on its side, so the jaw axis has to be chosen
Place Object Scale	weigh_fruit	—
Lift Pot	pot_dual_lift	—
Put Object Cabinet	cupboard_store	placement onto a raised shelf between walls
Open Microwave	microwave_door	—
Turn Switch	switch_toggle	—

No RoboTwin equivalent: pour_kettle, bar_pull, sweep_debris, eraser_wipe, push_relay, ring_post, peg_insert, drawer_store, push_block, grape_box, lid_food.

Assets

Object meshes come from RoboTwin 2.0 (MIT); custom_ring / custom_rope are generated by scripts in this repo.

# rigid objects -> USD (convex decomposition, so they can be dynamic)
python scripts/robotwin_convert.py --jobs 035_apple:base0
# containers -> USD with a HOLLOW cavity (triangle mesh, static only)
python scripts/robotwin_convert.py --collision none --suffix _mesh --jobs 110_basket:base0
# articulated fixtures (SAPIEN mobility URDF): SANITIZE FIRST, then convert
python scripts/sanitize_urdf_asset.py 044_microwave/7310
python scripts/robotwin_urdf_convert.py --fix_base --jobs 044_microwave/7310_clean

---

YAM Motion Planner — grape → box (PRM + real grasp)

Single-arm pick-and-place on Template-YAM-Play-v0: task-space PRM plan + real friction grasp (no kinematic attach), drop into a primitive box. Includes an N-episode benchmark that reports a success rate. Full details, flags, and output format: scripts/README_yam_motionplanner.md.

source "$HOME/miniconda3/etc/profile.d/conda.sh"; conda activate env_isaaclab
export OMNI_KIT_ACCEPT_EULA=YES OMNI_KIT_ALLOW_ROOT=1 PYTHONUNBUFFERED=1 CUDA_HOME=$CONDA_PREFIX

# one episode (grape -> box); positions are env-local metres (use --arg=-0.03,.. for negatives)
python scripts/yam_grasp_prm.py --headless --obj grape --basket \
  --obj_xy=0.00,0.10 --box_xy=0.06,-0.26 --episode 0 --video outputs/ep0.mp4

# N-episode benchmark + one stitched video with per-episode SUCCESS/FAIL and a final rate
OUT=outputs/grape_eps bash scripts/run_grape_episodes.sh
OUT=outputs/grape_eps python scripts/concat_episodes.py     # -> outputs/grape_eps/episodes.mp4

Key files: scripts/yam_grasp_prm.py (pick-place + per-frame debug overlay), scripts/yam_prm.py (PRM planner), scripts/run_grape_episodes.sh, scripts/concat_episodes.py. Robot/scene config: source/bimanual/tasks/manager_based/yam/{yam.py,yam_bimanual_env.py} (init pose, gripper, objects). Reference: ~52 s/episode; example 10-episode run = 6/10 (60%) success.