Instructions to use Dexverse-CSI/xhand-crosshand-dp with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- LeRobot
How to use Dexverse-CSI/xhand-crosshand-dp with LeRobot:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
xhand cross-hand image Diffusion Policies
Image Diffusion Policies (LeRobot 0.4.2) that control several dexterous hands through one shared action space:
a 41-D vector [palm position 3, palm rot6d 6, finger latent z 32], where z comes from a cross-hand finger codec.
The question behind the experiments: does a policy trained on some hands transfer zero-shot to a hand it never saw?
- Finger codecs and wrist maps:
Dexverse-CSI/xhand-finger-latent - pi0.5 versions (Z / D, no one-hot):
Dexverse-CSI/xhand-crosshand-pi05 - Per-hand native DPs (arm "A" of P3, a colleague's model):
Dexverse-CSI/dexverse-dp-image(not copied here) - Full personal backup (every snapshot, logs, training_state):
eunyoung927/xhand-crosshand-ckpts - Code: https://github.com/eunyoung030927/DexVerse (branch
dev, folderexperiments/cross_hand_latent_codec/)
Setup
- Task: DexVerse
Dexverse-GraspKettle-v0, floating right hand, teleoperated demos (50 episodes per hand). - Hands: sharpa, shadow, wuji (5-finger); allegro, leap (4-finger).
- Policy: LeRobot diffusion, ResNet18 trained from scratch (
pretrained_backbone_weights: None), two RGB cameras (third_person,eye_in_hand, 480Γ640), 300 epochs, batch 64, seed 42, one snapshot every 50 epochs. - Arms (action vector); each arm folder has a
CODEC.txtnaming the exact codec / f_task file and sha256:C: native joint vector zero-padded to 28 (no codec).J: native floating-wrist joints 6 + E2* z 32 (38-D).Z: palm pose (world) + z (41-D). Decode: fingers =Dec_hand(z), wrist = floating-wrist IK.D:EE_common = palm @ inv(f_task(q_finger))+ z (41-D). Decode: palm =EE_common @ f_task(Dec_hand(z)).- suffix
e2/e3= E2* codec,v2= codec v2cyc05clf05f002,w3= codec v3wb_cl_l.
- Selection rule: best = highest success total on the training hands, eps 0β9 per hand, over the snapshots at
epochs 50β250; ties go to the later step.
last= the final snapshot (epoch 300), which was not part of selection or evaluation. - Evaluation: DexVerse sim, DDPM 100 steps,
n_action_steps8, 3000-step cap (cv3dp counts successes within 1500 steps; no cv3dp episode succeeded later). Initial scenes come from the demo episodes. Held-out hands are unseen by the policy but are in the codec's training set (policy-unseen, not codec-unseen).
Layout
<experiment>/<arm>/best_<step>/pretrained_model/ and <experiment>/<arm>/last_<step>/pretrained_model/ (config, model.safetensors,
pre/post-processor normalizers, train_config.json), plus <experiment>/<arm>/CODEC.txt. No training_state. In every arm best β last.
| experiment | arms | train hands | eval hands | state / action | best | last |
|---|---|---|---|---|---|---|
pooled3hand (P3, hand one-hot in state) |
C, Z, D | sharpa, shadow, wuji | same (in-distribution) | C 31/28, Z/D 44/41 | C 148000, Z 059200, D 059200 | 177600 |
pooled3hand_nohot (P3-nohot) |
C, J, Z, D | sharpa, shadow, wuji | same; Z/D also zero-shot on allegro, leap (H1) | C 28, J 38, Z/D 41 | C 118400, J 088800, Z 059200, D 059200 | 177600 |
h2 (H2) |
Z, D | shadow, allegro | sharpa, wuji, leap (held-out) | 41 | Z 064800, D 108000 | 129600 |
cv2dp |
Ze2, Zv2 | shadow, wuji | sharpa (held-out) | 41 | Ze2 020950, Zv2 083800 | 125700 |
cv3dp |
Zw3, Ze3, Dw3, De3 | shadow, wuji, allegro | sharpa, leap (held-out) | 41 | Zw3 065500, Ze3 098250, Dw3 032750, De3 163750 | 196500 |
Pending (not uploaded yet):
cv3hm/{Zw3h, Ze3h}: FunctionalHammerStrike, train shadow + wuji + allegro, Zw3h = v3wb_cl_lz, Ze3h = E2* z. Training is finishing; the selection eval follows.cv3dp/Zc3: GraspKettle, cv3dp Z recipe with codec v2cyc05clf05f002z. Not trained yet.
Results (best snapshot)
P3 / P3-nohot, training hands, 50 episodes each (' = no one-hot; A = per-hand native DPs):
| arm | sharpa | shadow | wuji | total /150 |
|---|---|---|---|---|
| A (native, per hand) | 26 | 42 | 15 | 83 |
| C (one-hot) | 49 | 40 | 43 | 132 |
| C' | 46 | 38 | 43 | 127 |
| J | 18 | 29 | 22 | 69 |
| Z (one-hot) | 17 | 29 | 31 | 77 |
| Z' | 27 | 27 | 29 | 83 |
| D (one-hot) | 18 | 34 | 19 | 71 |
| D' | 21 | 19 | 25 | 65 |
H1 (P3-nohot Z' / D' 059200, zero-shot): allegro 0/10, leap 0/10 for both. H2 (Z 064800 / D 108000, zero-shot): sharpa 0/10, wuji 0/10, leap 0/10 for both.
cv2dp (train shadow + wuji): sharpa (held-out) Ze2 0/50 vs Zv2 15/50 (exact McNemar p = 6.1e-5); training hands eps 0β9: 11/20 vs 10/20.
cv3dp (train shadow + wuji + allegro):
| arm | wrist / z | sharpa (held-out) /50 | leap (held-out) /50 | shadow /50 | wuji /50 | allegro /50 | train /150 |
|---|---|---|---|---|---|---|---|
| Zw3 | Z / v3 wb_cl_l |
15 | 21 | 31 | 30 | 20 | 81 |
| Ze3 | Z / E2* | 0 | 0 | 32 | 16 | 35 | 83 |
| Dw3 | D / v3 wb_cl_l |
14 | 8 | 22 | 28 | 4 | 54 |
| De3 | D / E2* | 0 | 0 | 30 | 28 | 40 | 98 |
Zw3 vs Ze3: p = 6.1e-5 (sharpa), 9.5e-7 (leap); training hands 81 vs 83 (p = 0.90). The D wrist (f_task) does not help. One training seed per arm.
How to load
from huggingface_hub import snapshot_download
from lerobot.policies.diffusion.modeling_diffusion import DiffusionPolicy
from lerobot.policies.factory import make_pre_post_processors
d = snapshot_download("Dexverse-CSI/xhand-crosshand-dp",
allow_patterns=["cv3dp/Zw3/best_065500/*", "cv3dp/Zw3/CODEC.txt"])
path = f"{d}/cv3dp/Zw3/best_065500/pretrained_model"
policy = DiffusionPolicy.from_pretrained(path)
pre, post = make_pre_post_processors(policy.config, pretrained_path=path)
The policy outputs the 41-D vector; turning it into hand joint targets needs the codec decoder and the wrist IK from the
GitHub code (code/cv3dp_eval_pooled.py, code/p3h_policy_codec.py, code/ho_codec.py) and the codec file named in CODEC.txt.
Not included
training_state(optimizer / scheduler), intermediate snapshots: see the personal backup.- Per-hand native DPs (arm A): see
Dexverse-CSI/dexverse-dp-image. - Datasets and DexVerse assets.
License: not yet specified (other).