Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -9,40 +9,64 @@ tags:
|
|
| 9 |
|
| 10 |
# DP3 floating-gripper garment policies β Hang_Trousers
|
| 11 |
|
| 12 |
-
|
| 13 |
-
floating-gripper garment-hanging demonstrations in
|
| 14 |
-
(
|
|
|
|
| 15 |
|
| 16 |
-
|
| 17 |
-
|
| 18 |
-
|
| 19 |
-
|
|
|
|
|
|
|
| 20 |
|
| 21 |
## The five variants
|
| 22 |
|
| 23 |
-
|
| 24 |
-
|
| 25 |
-
task success, which has not been measured yet.
|
| 26 |
|
| 27 |
| file | encoder | observation | edges | epoch | EE MAE |
|
| 28 |
|---|---|---|---|---|---|
|
| 29 |
| `trousers_pn_depth.ckpt` | PointNet | depth cloud 2048Γ3 | β | 550 | **6.25 mm** |
|
| 30 |
| `trousers_gnn_depth.ckpt` | mesh-GNN | depth cloud | kNN | 200 | 6.85 mm |
|
| 31 |
-
| `trousers_pn_mesh.ckpt` | PointNet | decimated mesh + pothook | β | 900 | 6.94 mm |
|
| 32 |
-
| `trousers_gnn_mesh.ckpt` | mesh-GNN | mesh nodes | **true garment faces** | 300 | 7.24 mm |
|
| 33 |
-
| `trousers_gnn_knn.ckpt` | mesh-GNN | mesh nodes | kNN | 200 | 7.35 mm |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 34 |
|
| 35 |
-
|
| 36 |
-
the two disagree for every run, and selecting on `val_loss` would be up to 3.3 mm
|
| 37 |
-
worse (it is diffusion denoising MSE, not action quality). Note each figure is an
|
| 38 |
-
argmin over many checkpoints on a 4-episode split, so it is optimistic.
|
| 39 |
|
| 40 |
-
All five
|
| 41 |
-
|
| 42 |
-
|
| 43 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 44 |
|
| 45 |
-
|
|
|
|
|
|
|
| 46 |
|
| 47 |
```python
|
| 48 |
import torch, dill, hydra
|
|
@@ -54,19 +78,38 @@ policy = hydra.utils.instantiate(p["cfg"].policy)
|
|
| 54 |
policy.load_state_dict(p["state_dicts"]["ema_model"])
|
| 55 |
policy.cuda().eval()
|
| 56 |
|
| 57 |
-
|
| 58 |
-
# out: (B, 4, 8) = [left_xyz, right_xyz, grip_L, grip_R], absolute metres
|
| 59 |
-
action = policy.predict_action(obs)["action"]
|
| 60 |
```
|
| 61 |
|
| 62 |
-
|
| 63 |
-
|
| 64 |
-
|
| 65 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 66 |
|
| 67 |
-
|
| 68 |
-
|
| 69 |
-
|
|
|
|
|
|
|
|
|
|
| 70 |
|
| 71 |
-
|
| 72 |
-
`python tools/export_checkpoints.py --out export/ --push <repo>`.
|
|
|
|
| 9 |
|
| 10 |
# DP3 floating-gripper garment policies β Hang_Trousers
|
| 11 |
|
| 12 |
+
Five [3D Diffusion Policy](https://3d-diffusion-policy.github.io/) variants trained on
|
| 13 |
+
floating-gripper garment-hanging demonstrations in
|
| 14 |
+
[DexGarmentLab](https://huggingface.co/datasets/wayrise/DexGarmentLab)
|
| 15 |
+
(`Hang_Trousers_FloatGrip`, 195 demos, 12,516 transitions, Isaac Sim).
|
| 16 |
|
| 17 |
+
They differ **only** in the observation representation and the point encoder, which is
|
| 18 |
+
what makes them useful as an ablation: what does ground-truth cloth mesh state, and
|
| 19 |
+
true mesh connectivity, buy over a depth point cloud?
|
| 20 |
+
|
| 21 |
+
EMA weights only, ~0.98 GiB each. The source repo is currently private; everything
|
| 22 |
+
needed to *use* these checkpoints is below.
|
| 23 |
|
| 24 |
## The five variants
|
| 25 |
|
| 26 |
+
`EE MAE` is held-out end-effector position error against demonstration actions.
|
| 27 |
+
**It is not task success** β no policy here has been evaluated in closed loop yet.
|
|
|
|
| 28 |
|
| 29 |
| file | encoder | observation | edges | epoch | EE MAE |
|
| 30 |
|---|---|---|---|---|---|
|
| 31 |
| `trousers_pn_depth.ckpt` | PointNet | depth cloud 2048Γ3 | β | 550 | **6.25 mm** |
|
| 32 |
| `trousers_gnn_depth.ckpt` | mesh-GNN | depth cloud | kNN | 200 | 6.85 mm |
|
| 33 |
+
| `trousers_pn_mesh.ckpt` | PointNet | decimated GT mesh + pothook | β | 900 | 6.94 mm |
|
| 34 |
+
| `trousers_gnn_mesh.ckpt` | mesh-GNN | GT mesh nodes | **true garment faces** | 300 | 7.24 mm |
|
| 35 |
+
| `trousers_gnn_knn.ckpt` | mesh-GNN | GT mesh nodes | kNN | 200 | 7.35 mm |
|
| 36 |
+
|
| 37 |
+
### How to read this table
|
| 38 |
+
|
| 39 |
+
- **Down a column** (same encoder) isolates the representation. **Across a row** isolates
|
| 40 |
+
the encoder. Never use a row to make a claim about representations.
|
| 41 |
+
- `gnn_mesh` vs `gnn_knn` is the sharpest comparison: identical nodes, identical encoder,
|
| 42 |
+
identical parameter count β the *only* difference is whether edges come from true mesh
|
| 43 |
+
faces or from k-nearest-neighbours. The gap is 0.11 mm, i.e. nothing.
|
| 44 |
+
- Checkpoints were selected by lowest EE MAE, **not** by validation loss. The two disagree
|
| 45 |
+
for every run, and selecting on `val_loss` would be up to 3.3 mm worse β it is diffusion
|
| 46 |
+
denoising MSE averaged over noise levels, not a measure of sampled action quality.
|
| 47 |
|
| 48 |
+
### Honest caveats
|
|
|
|
|
|
|
|
|
|
| 49 |
|
| 50 |
+
- All five are within ~1 mm of each other on a **4-episode** validation split. Treat them
|
| 51 |
+
as **near-indistinguishable**.
|
| 52 |
+
- Each figure is an argmin over 6β20 checkpoints on that same small split, so the numbers
|
| 53 |
+
are optimistic.
|
| 54 |
+
- They *are* learning: an observation-free "hold current position" baseline scores
|
| 55 |
+
48.9 mm and constant-velocity 28.3 mm, so the policies beat trivial baselines by 4.5β8Γ.
|
| 56 |
+
- Encoder capacity, normalizer range and padding were each checked and ruled out as
|
| 57 |
+
explanations for the mesh variants underperforming. The likely reason is that the depth
|
| 58 |
+
cloud already contains ~75 % garment points plus the randomised target hook in one
|
| 59 |
+
view-consistent frame, and this task's difficulty is not cloth-state estimation.
|
| 60 |
+
|
| 61 |
+
## Usage
|
| 62 |
+
|
| 63 |
+
```bash
|
| 64 |
+
hf download albilo/dp3-floatgrip --local-dir checkpoints_pretrained
|
| 65 |
+
```
|
| 66 |
|
| 67 |
+
Needs `torch`, `diffusers`, `hydra-core`, `omegaconf`, `dill`, `einops`, `termcolor`, and
|
| 68 |
+
the `diffusion_policy_3d` package from DexGarmentLab's `IL_Baselines/Diffusion_Policy_3D`
|
| 69 |
+
on `PYTHONPATH` (the config in each checkpoint instantiates classes by dotted path).
|
| 70 |
|
| 71 |
```python
|
| 72 |
import torch, dill, hydra
|
|
|
|
| 78 |
policy.load_state_dict(p["state_dicts"]["ema_model"])
|
| 79 |
policy.cuda().eval()
|
| 80 |
|
| 81 |
+
action = policy.predict_action(obs)["action"] # (B, 4, 8)
|
|
|
|
|
|
|
| 82 |
```
|
| 83 |
|
| 84 |
+
**Observation** β `n_obs_steps = 3`, so a history of 3 frames, oldest first:
|
| 85 |
+
|
| 86 |
+
| key | shape | notes |
|
| 87 |
+
|---|---|---|
|
| 88 |
+
| `point_cloud` | `(B, 3, 2048, 3)` float32 | xyz only, metres, ground removed, FPS-downsampled |
|
| 89 |
+
| `agent_pos` | `(B, 3, 8)` float32 | `[left_xyz, right_xyz, grip_L, grip_R]` |
|
| 90 |
+
|
| 91 |
+
`trousers_gnn_mesh` additionally requires `edges` `(B, 3, E, 2)`, `n_nodes` `(B, 3)` and
|
| 92 |
+
`n_edges` `(B, 3)`; without them it raises `KeyError: 'edges'`. The other four do not β
|
| 93 |
+
the kNN variants build their graph from coordinates at runtime.
|
| 94 |
+
|
| 95 |
+
**Action** β `(B, 4, 8)`: four future steps of
|
| 96 |
+
`[left_xyz, right_xyz, grip_L, grip_R]`, **absolute positions in metres**, not deltas.
|
| 97 |
+
There is no orientation (the grippers are attachment spheres; the demos never recorded
|
| 98 |
+
any). The normalizer travels inside the `state_dict`, so actions come back already in
|
| 99 |
+
metres β do not rescale. Gripper bits are regressed continuously; threshold at 0.5.
|
| 100 |
+
Execute all 4 actions before re-planning.
|
| 101 |
+
|
| 102 |
+
**Timing** β demonstrations were recorded every 10 physics steps with gripper motion
|
| 103 |
+
interpolated at 0.01 m/step, so one action β one recorded frame. Matching this matters:
|
| 104 |
+
a different velocity regime is out of distribution.
|
| 105 |
+
|
| 106 |
+
## Training
|
| 107 |
|
| 108 |
+
195 successful demonstrations, 12,516 transitions, one H100 per run. PointNet cells
|
| 109 |
+
1000 epochs @ batch 256 (~1.4 h); mesh-GNN cells 300 epochs @ batch 64 (~4.4 h, gradient
|
| 110 |
+
checkpointing β the dense padded edge activations peak at 77 GiB without it). The
|
| 111 |
+
mesh-GNN is a MeshGraphNets-style encoder β processor over the garment graph; mesh nodes
|
| 112 |
+
are decimated to 1234β1535 with topology taken from the garment USDs and vertex
|
| 113 |
+
correspondence preserved per frame.
|
| 114 |
|
| 115 |
+
`manifest.json` records which training epoch each file came from.
|
|
|