albilo commited on
Commit
8b760fb
Β·
verified Β·
1 Parent(s): 521ee33

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +77 -34
README.md CHANGED
@@ -9,40 +9,64 @@ tags:
9
 
10
  # DP3 floating-gripper garment policies β€” Hang_Trousers
11
 
12
- Inference-only checkpoints for five 3D Diffusion Policy variants trained on
13
- floating-gripper garment-hanging demonstrations in DexGarmentLab
14
- (`Hang_Trousers_FloatGrip`, 195 demos, 12,516 transitions).
 
15
 
16
- Code and full notes: **`floating-gripper-policy`** branch of
17
- <https://github.com/lei-shu/garmentlab>, under
18
- `DexGarmentLab/IL_Baselines/Diffusion_Policy_3D/` β€” see `EXPERIMENTS.md` for how
19
- these were trained and `SIM_EVAL_HANDOFF.md` for how to run them.
 
 
20
 
21
  ## The five variants
22
 
23
- Each differs only in the observation representation and the point encoder.
24
- `EE MAE` is held-out end-effector position error against demo actions β€” **not**
25
- task success, which has not been measured yet.
26
 
27
  | file | encoder | observation | edges | epoch | EE MAE |
28
  |---|---|---|---|---|---|
29
  | `trousers_pn_depth.ckpt` | PointNet | depth cloud 2048Γ—3 | β€” | 550 | **6.25 mm** |
30
  | `trousers_gnn_depth.ckpt` | mesh-GNN | depth cloud | kNN | 200 | 6.85 mm |
31
- | `trousers_pn_mesh.ckpt` | PointNet | decimated mesh + pothook | β€” | 900 | 6.94 mm |
32
- | `trousers_gnn_mesh.ckpt` | mesh-GNN | mesh nodes | **true garment faces** | 300 | 7.24 mm |
33
- | `trousers_gnn_knn.ckpt` | mesh-GNN | mesh nodes | kNN | 200 | 7.35 mm |
 
 
 
 
 
 
 
 
 
 
 
34
 
35
- Checkpoints were selected by **lowest held-out EE MAE**, not by validation loss β€”
36
- the two disagree for every run, and selecting on `val_loss` would be up to 3.3 mm
37
- worse (it is diffusion denoising MSE, not action quality). Note each figure is an
38
- argmin over many checkpoints on a 4-episode split, so it is optimistic.
39
 
40
- All five sit within ~1 mm of each other on a 4-episode validation split, so treat
41
- them as near-indistinguishable offline. They do beat an observation-free
42
- "hold position" baseline (48.9 mm) by 8Γ—, so they are genuinely using the
43
- observation.
 
 
 
 
 
 
 
 
 
 
 
 
44
 
45
- ## Loading
 
 
46
 
47
  ```python
48
  import torch, dill, hydra
@@ -54,19 +78,38 @@ policy = hydra.utils.instantiate(p["cfg"].policy)
54
  policy.load_state_dict(p["state_dicts"]["ema_model"])
55
  policy.cuda().eval()
56
 
57
- # obs: point_cloud (B, 3, 2048, 3) + agent_pos (B, 3, 8)
58
- # out: (B, 4, 8) = [left_xyz, right_xyz, grip_L, grip_R], absolute metres
59
- action = policy.predict_action(obs)["action"]
60
  ```
61
 
62
- These carry EMA weights only (~0.98 GiB each); the training checkpoints they came
63
- from were 3.91 GiB with optimizer state. The normalizer travels inside the
64
- `state_dict`, so actions come back in metres β€” do not rescale. Gripper bits are
65
- regressed continuously; threshold at 0.5.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
66
 
67
- `trousers_gnn_mesh` additionally needs `edges`, `n_nodes`, `n_edges` in the
68
- observation dict (see `SIM_EVAL_HANDOFF.md` Β§4a β€” the stock `RobotRunner` drops
69
- them and it will raise `KeyError: 'edges'`).
 
 
 
70
 
71
- Regenerate or re-upload with
72
- `python tools/export_checkpoints.py --out export/ --push <repo>`.
 
9
 
10
  # DP3 floating-gripper garment policies β€” Hang_Trousers
11
 
12
+ Five [3D Diffusion Policy](https://3d-diffusion-policy.github.io/) variants trained on
13
+ floating-gripper garment-hanging demonstrations in
14
+ [DexGarmentLab](https://huggingface.co/datasets/wayrise/DexGarmentLab)
15
+ (`Hang_Trousers_FloatGrip`, 195 demos, 12,516 transitions, Isaac Sim).
16
 
17
+ They differ **only** in the observation representation and the point encoder, which is
18
+ what makes them useful as an ablation: what does ground-truth cloth mesh state, and
19
+ true mesh connectivity, buy over a depth point cloud?
20
+
21
+ EMA weights only, ~0.98 GiB each. The source repo is currently private; everything
22
+ needed to *use* these checkpoints is below.
23
 
24
  ## The five variants
25
 
26
+ `EE MAE` is held-out end-effector position error against demonstration actions.
27
+ **It is not task success** β€” no policy here has been evaluated in closed loop yet.
 
28
 
29
  | file | encoder | observation | edges | epoch | EE MAE |
30
  |---|---|---|---|---|---|
31
  | `trousers_pn_depth.ckpt` | PointNet | depth cloud 2048Γ—3 | β€” | 550 | **6.25 mm** |
32
  | `trousers_gnn_depth.ckpt` | mesh-GNN | depth cloud | kNN | 200 | 6.85 mm |
33
+ | `trousers_pn_mesh.ckpt` | PointNet | decimated GT mesh + pothook | β€” | 900 | 6.94 mm |
34
+ | `trousers_gnn_mesh.ckpt` | mesh-GNN | GT mesh nodes | **true garment faces** | 300 | 7.24 mm |
35
+ | `trousers_gnn_knn.ckpt` | mesh-GNN | GT mesh nodes | kNN | 200 | 7.35 mm |
36
+
37
+ ### How to read this table
38
+
39
+ - **Down a column** (same encoder) isolates the representation. **Across a row** isolates
40
+ the encoder. Never use a row to make a claim about representations.
41
+ - `gnn_mesh` vs `gnn_knn` is the sharpest comparison: identical nodes, identical encoder,
42
+ identical parameter count β€” the *only* difference is whether edges come from true mesh
43
+ faces or from k-nearest-neighbours. The gap is 0.11 mm, i.e. nothing.
44
+ - Checkpoints were selected by lowest EE MAE, **not** by validation loss. The two disagree
45
+ for every run, and selecting on `val_loss` would be up to 3.3 mm worse β€” it is diffusion
46
+ denoising MSE averaged over noise levels, not a measure of sampled action quality.
47
 
48
+ ### Honest caveats
 
 
 
49
 
50
+ - All five are within ~1 mm of each other on a **4-episode** validation split. Treat them
51
+ as **near-indistinguishable**.
52
+ - Each figure is an argmin over 6–20 checkpoints on that same small split, so the numbers
53
+ are optimistic.
54
+ - They *are* learning: an observation-free "hold current position" baseline scores
55
+ 48.9 mm and constant-velocity 28.3 mm, so the policies beat trivial baselines by 4.5–8Γ—.
56
+ - Encoder capacity, normalizer range and padding were each checked and ruled out as
57
+ explanations for the mesh variants underperforming. The likely reason is that the depth
58
+ cloud already contains ~75 % garment points plus the randomised target hook in one
59
+ view-consistent frame, and this task's difficulty is not cloth-state estimation.
60
+
61
+ ## Usage
62
+
63
+ ```bash
64
+ hf download albilo/dp3-floatgrip --local-dir checkpoints_pretrained
65
+ ```
66
 
67
+ Needs `torch`, `diffusers`, `hydra-core`, `omegaconf`, `dill`, `einops`, `termcolor`, and
68
+ the `diffusion_policy_3d` package from DexGarmentLab's `IL_Baselines/Diffusion_Policy_3D`
69
+ on `PYTHONPATH` (the config in each checkpoint instantiates classes by dotted path).
70
 
71
  ```python
72
  import torch, dill, hydra
 
78
  policy.load_state_dict(p["state_dicts"]["ema_model"])
79
  policy.cuda().eval()
80
 
81
+ action = policy.predict_action(obs)["action"] # (B, 4, 8)
 
 
82
  ```
83
 
84
+ **Observation** β€” `n_obs_steps = 3`, so a history of 3 frames, oldest first:
85
+
86
+ | key | shape | notes |
87
+ |---|---|---|
88
+ | `point_cloud` | `(B, 3, 2048, 3)` float32 | xyz only, metres, ground removed, FPS-downsampled |
89
+ | `agent_pos` | `(B, 3, 8)` float32 | `[left_xyz, right_xyz, grip_L, grip_R]` |
90
+
91
+ `trousers_gnn_mesh` additionally requires `edges` `(B, 3, E, 2)`, `n_nodes` `(B, 3)` and
92
+ `n_edges` `(B, 3)`; without them it raises `KeyError: 'edges'`. The other four do not β€”
93
+ the kNN variants build their graph from coordinates at runtime.
94
+
95
+ **Action** β€” `(B, 4, 8)`: four future steps of
96
+ `[left_xyz, right_xyz, grip_L, grip_R]`, **absolute positions in metres**, not deltas.
97
+ There is no orientation (the grippers are attachment spheres; the demos never recorded
98
+ any). The normalizer travels inside the `state_dict`, so actions come back already in
99
+ metres β€” do not rescale. Gripper bits are regressed continuously; threshold at 0.5.
100
+ Execute all 4 actions before re-planning.
101
+
102
+ **Timing** β€” demonstrations were recorded every 10 physics steps with gripper motion
103
+ interpolated at 0.01 m/step, so one action β‰ˆ one recorded frame. Matching this matters:
104
+ a different velocity regime is out of distribution.
105
+
106
+ ## Training
107
 
108
+ 195 successful demonstrations, 12,516 transitions, one H100 per run. PointNet cells
109
+ 1000 epochs @ batch 256 (~1.4 h); mesh-GNN cells 300 epochs @ batch 64 (~4.4 h, gradient
110
+ checkpointing β€” the dense padded edge activations peak at 77 GiB without it). The
111
+ mesh-GNN is a MeshGraphNets-style encoder β†’ processor over the garment graph; mesh nodes
112
+ are decimated to 1234–1535 with topology taken from the garment USDs and vertex
113
+ correspondence preserved per frame.
114
 
115
+ `manifest.json` records which training epoch each file came from.