xhand cross-hand image Diffusion Policies

Image Diffusion Policies (LeRobot 0.4.2) that control several dexterous hands through one shared action space: a 41-D vector [palm position 3, palm rot6d 6, finger latent z 32], where z comes from a cross-hand finger codec. The question behind the experiments: does a policy trained on some hands transfer zero-shot to a hand it never saw?

Setup

  • Task: DexVerse Dexverse-GraspKettle-v0, floating right hand, teleoperated demos (50 episodes per hand).
  • Hands: sharpa, shadow, wuji (5-finger); allegro, leap (4-finger).
  • Policy: LeRobot diffusion, ResNet18 trained from scratch (pretrained_backbone_weights: None), two RGB cameras (third_person, eye_in_hand, 480Γ—640), 300 epochs, batch 64, seed 42, one snapshot every 50 epochs.
  • Arms (action vector); each arm folder has a CODEC.txt naming the exact codec / f_task file and sha256:
    • C: native joint vector zero-padded to 28 (no codec).
    • J: native floating-wrist joints 6 + E2* z 32 (38-D).
    • Z: palm pose (world) + z (41-D). Decode: fingers = Dec_hand(z), wrist = floating-wrist IK.
    • D: EE_common = palm @ inv(f_task(q_finger)) + z (41-D). Decode: palm = EE_common @ f_task(Dec_hand(z)).
    • suffix e2/e3 = E2* codec, v2 = codec v2 cyc05clf05f002, w3 = codec v3 wb_cl_l.
  • Selection rule: best = highest success total on the training hands, eps 0–9 per hand, over the snapshots at epochs 50–250; ties go to the later step. last = the final snapshot (epoch 300), which was not part of selection or evaluation.
  • Evaluation: DexVerse sim, DDPM 100 steps, n_action_steps 8, 3000-step cap (cv3dp counts successes within 1500 steps; no cv3dp episode succeeded later). Initial scenes come from the demo episodes. Held-out hands are unseen by the policy but are in the codec's training set (policy-unseen, not codec-unseen).

Layout

<experiment>/<arm>/best_<step>/pretrained_model/ and <experiment>/<arm>/last_<step>/pretrained_model/ (config, model.safetensors, pre/post-processor normalizers, train_config.json), plus <experiment>/<arm>/CODEC.txt. No training_state. In every arm best β‰  last.

experiment arms train hands eval hands state / action best last
pooled3hand (P3, hand one-hot in state) C, Z, D sharpa, shadow, wuji same (in-distribution) C 31/28, Z/D 44/41 C 148000, Z 059200, D 059200 177600
pooled3hand_nohot (P3-nohot) C, J, Z, D sharpa, shadow, wuji same; Z/D also zero-shot on allegro, leap (H1) C 28, J 38, Z/D 41 C 118400, J 088800, Z 059200, D 059200 177600
h2 (H2) Z, D shadow, allegro sharpa, wuji, leap (held-out) 41 Z 064800, D 108000 129600
cv2dp Ze2, Zv2 shadow, wuji sharpa (held-out) 41 Ze2 020950, Zv2 083800 125700
cv3dp Zw3, Ze3, Dw3, De3 shadow, wuji, allegro sharpa, leap (held-out) 41 Zw3 065500, Ze3 098250, Dw3 032750, De3 163750 196500

Pending (not uploaded yet):

  • cv3hm/{Zw3h, Ze3h}: FunctionalHammerStrike, train shadow + wuji + allegro, Zw3h = v3 wb_cl_l z, Ze3h = E2* z. Training is finishing; the selection eval follows.
  • cv3dp/Zc3: GraspKettle, cv3dp Z recipe with codec v2 cyc05clf05f002 z. Not trained yet.

Results (best snapshot)

P3 / P3-nohot, training hands, 50 episodes each (' = no one-hot; A = per-hand native DPs):

arm sharpa shadow wuji total /150
A (native, per hand) 26 42 15 83
C (one-hot) 49 40 43 132
C' 46 38 43 127
J 18 29 22 69
Z (one-hot) 17 29 31 77
Z' 27 27 29 83
D (one-hot) 18 34 19 71
D' 21 19 25 65

H1 (P3-nohot Z' / D' 059200, zero-shot): allegro 0/10, leap 0/10 for both. H2 (Z 064800 / D 108000, zero-shot): sharpa 0/10, wuji 0/10, leap 0/10 for both.

cv2dp (train shadow + wuji): sharpa (held-out) Ze2 0/50 vs Zv2 15/50 (exact McNemar p = 6.1e-5); training hands eps 0–9: 11/20 vs 10/20.

cv3dp (train shadow + wuji + allegro):

arm wrist / z sharpa (held-out) /50 leap (held-out) /50 shadow /50 wuji /50 allegro /50 train /150
Zw3 Z / v3 wb_cl_l 15 21 31 30 20 81
Ze3 Z / E2* 0 0 32 16 35 83
Dw3 D / v3 wb_cl_l 14 8 22 28 4 54
De3 D / E2* 0 0 30 28 40 98

Zw3 vs Ze3: p = 6.1e-5 (sharpa), 9.5e-7 (leap); training hands 81 vs 83 (p = 0.90). The D wrist (f_task) does not help. One training seed per arm.

How to load

from huggingface_hub import snapshot_download
from lerobot.policies.diffusion.modeling_diffusion import DiffusionPolicy
from lerobot.policies.factory import make_pre_post_processors

d = snapshot_download("Dexverse-CSI/xhand-crosshand-dp",
                      allow_patterns=["cv3dp/Zw3/best_065500/*", "cv3dp/Zw3/CODEC.txt"])
path = f"{d}/cv3dp/Zw3/best_065500/pretrained_model"
policy = DiffusionPolicy.from_pretrained(path)
pre, post = make_pre_post_processors(policy.config, pretrained_path=path)

The policy outputs the 41-D vector; turning it into hand joint targets needs the codec decoder and the wrist IK from the GitHub code (code/cv3dp_eval_pooled.py, code/p3h_policy_codec.py, code/ho_codec.py) and the codec file named in CODEC.txt.

Not included

  • training_state (optimizer / scheduler), intermediate snapshots: see the personal backup.
  • Per-hand native DPs (arm A): see Dexverse-CSI/dexverse-dp-image.
  • Datasets and DexVerse assets.

License: not yet specified (other).

Downloads last month

-

Downloads are not tracked for this model. How to track
Video Preview
loading