dit_block_3 / README.md
anschrur's picture
add model card
01c3013 verified
|
Raw History Blame Contribute Delete
6.73 kB
---
pipeline_tag: robotics
tags:
- imitation-learning
- flow-matching
- diffusion-policy
- so-101
- robot-imitation-glue
---
# dit_block_3 β€” DiT flow-matching policy, cup-into-ring
Single-camera Diffusion-Transformer flow policy for the SO-10x task *pick up the cup, place it in the ring*.
Trained on the same clean wrist retake as `dit_block_2`, grown from 30 to 101 teleoperated episodes, at a
matched gradient-step budget.
**This is not a `from_pretrained` checkpoint.** It is one raw `torch.save` dict written by
`robot_imitation_glue.baseline.train_dit`. lerobot's policy registry cannot open it β€” there is no
`config.json` / `model.safetensors` here. Load it through `DiTAgent` (below).
## File
| file | bytes | md5 |
|---|---|---|
| `policy.ckpt` | 66,705,733 | `036a435230f9bff3f1f5e7ce430ac2a6` |
Contents: `config`, `image_size`, `crop_ratio`, `model_state_dict` (16,660,576 params),
`proprio_normalizer_state_dict`, `action_normalizer_state_dict`. The two normalisers ship with the weights β€”
the model is unusable without them, and nothing else (no dataset paths, no credentials) is inside the file.
Byte size is identical to `dit_block_2`'s because the architecture is unchanged; only the numbers differ.
## Task interface
- **State / action**: 6-D absolute Feetech positions, dataset order
`shoulder_pan, shoulder_lift, elbow_flex, wrist_flex, wrist_roll, gripper`. Dims 0-4 are **degrees**; dim 5 is
**% open** (the gripper servo is hard-wired to `RANGE_0_100`). Not one unit β€” do not scale the vector.
- **Vision**: one camera, `observation.images.front`, recorded 480Γ—640 AV1 β†’ resized to 128Γ—128 with a 0.9 centre crop.
- **Chunk**: `prediction_horizon = 48` counts **frames**, and the dataset is **30 fps**, so one chunk is 1.6 s of
motion. A control loop must therefore run at 30 fps; at 10 fps the same chunk stretches to 4.8 s.
## Architecture
| | |
|---|---|
| hidden dim | 256 |
| layers / heads | 4 / 4 |
| camera encoder | resnet18, ImageNet-pretrained, frozen statistics shipped as `encoders.0.mean/std` |
| prediction | flow matching, Euler sampler (10 steps is what the deployment script uses) |
| params | 16,660,576 |
Identical to `anschrur/dit_block_2` in every architectural respect, so the two are drop-in substitutes for each
other at the deployment script.
## Training
| | |
|---|---|
| dataset | [`anschrur/record-test_2_20261115_115045`](https://huggingface.co/datasets/anschrur/record-test_2_20261115_115045) β€” 101 episodes, 28,498 frames, 30 fps, 1 task |
| optimiser steps | 41,385 = 93 epochs Γ— floor(28,498 / 64), `drop_last=True` |
| batch / optimiser | 64 / AdamW, lr 1e-4, weight decay 1e-4 |
| schedule | 500-step linear warmup β†’ cosine to 1 % of base (final lr 1e-6), grad-clip 10, seed 42 |
| final train loss | 0.034032 (epoch 93/93); 0.091574 @ 20, 0.059593 @ 40, 0.043904 @ 60, 0.034347 @ 80 |
| hardware / wall clock | 1Γ— RTX 5090 (imec iLab.t GPULab cluster 9), 1 h 46 min |
| written | 2026-11-15 17:04 UTC, `/project_ghent/dit/outputs/dit_cup_3/policy.ckpt` (GPULab job `dit`, `eeb9465c`) |
The epoch count (93) is not inherited β€” it solves for the step total. Steps = epochs Γ— floor(frames / batch),
because the LR schedule is built from `epochs Γ— len(dataloader)` and the loader drops the last partial batch.
Holding steps at ~41 k makes this run a like-for-like exchange with `dit_block_2` (41,200 steps) on the same
compute budget; 2.2Γ— the frames therefore means each frame is visited 93 rather than 200 times.
`--save-every 20` rewrites the *same* `policy.ckpt` path, so these are the epoch-93 weights only β€” five writes,
one surviving file, no intermediate snapshots and no epoch sweep to plot. The dataset declares
`splits {train: 0:101}`, so `train_loss` is a training-set number, not an evaluation result.
## What changed against `anschrur/dit_block_2`
Same dataset directory, same wrist-clean labels, same architecture and hyperparameters. The dataset grew from
30 episodes / 13,218 frames to 101 episodes / 28,498 frames, and the epoch count dropped 200 β†’ 93 to keep the
step totals matched. Action normaliser (degrees for dims 0-4, % open for dim 5), measured out of the file:
| dim | `dit_block_2` mean / std | `dit_block_3` mean / std |
|---|---|---|
| shoulder_pan | 0.79 / 16.15 | βˆ’1.61 / 16.72 |
| shoulder_lift | 79.98 / 51.82 | 86.14 / 45.21 |
| elbow_flex | βˆ’71.29 / 43.93 | βˆ’73.83 / 36.81 |
| wrist_flex | 73.85 / 13.39 | 70.36 / 14.21 |
| **wrist_roll** | **4.37 / 20.51** | **βˆ’4.31 / 19.38** |
| gripper | 6.85 / 11.46 | 8.34 / 12.70 |
`wrist_roll` stays near zero mean with std β‰ˆ 20 β€” that is the label-fix signature from `dit_block_2` surviving
into the 71 new episodes, and it is the one row here that is a direct quality check rather than a distribution
shift. The other five moved because the added episodes reach more of the workspace; every std is in the same
band, so no joint collapsed or exploded.
**Do not read the loss as a regression.** `dit_block_2` finished at 0.018505 and this at 0.034032, but the two
numbers are over different frame sets at different visit counts: 13,218 frames seen 200Γ— versus 28,498 frames
seen 93Γ—. The lower figure is the run that memorised its training set harder, not the better policy. The
comparable statement is shape, not level β€” and the flattening here (0.034347 @ 80 β†’ 0.034032 @ 93) says the
optimiser was near exhausted by epoch 80, so 93 epochs is not leaving much on the table.
## Loading
```python
from robot_imitation_glue.baseline.dit_agent import DiTAgent # plain clone on sys.path is enough
agent = DiTAgent(
checkpoint_path="policy.ckpt",
camera_key_mapping={"front": "observation.images.front"}, # live camera key -> policy key
state_key="observation.state",
device="cpu",
n_sampling_steps=10,
)
```
Closed-loop on an arm β€” the repo is public, so this needs no token:
```bash
python dit_on_robot.py --repo-id anschrur/dit_block_3 --fps 30 --dry-run # arm must NOT move on this one
```
`dit_on_robot.py` (thesis glue, not published here) wires `DiTAgent` to lerobot's `SOFollower` with a wrap-aware
per-step delta guard, because lerobot's own `max_relative_target` clamp mishandles the Β±180Β° `wrist_roll` boundary.
## Not measured
No open-loop action MSE, no evaluation episodes, no success rate, and no head-to-head against `dit_block_2`
beyond the normaliser stats above. Whether 2.2Γ— the demonstrations buys closed-loop behaviour at matched step
count is exactly what the next rollout on the arm has to answer β€” `dit_block_2` froze on the first real run, and
that failure mode is not yet attributed to data volume, so this checkpoint inherits the question rather than
settling it.