README.md

---
license: cc-by-nc-4.0
library_name: lerobot
pipeline_tag: robotics
tags:
  - robotics
  - lerobot
  - flow-matching
  - diffusion-transformer
  - imitation-learning
  - ur7e
  - end-effector
---

# Flow-Matching DiT · carrot-in-pot · EEF-delta (state 16 / action 7) — checkpoint 60k

Text-conditioned flow-matching policy (`multi_task_dit`, CLIP text encoder, Euler ODE) trained on
the **real** UR7e *"Put carrot in pot"* demonstrations (54 GELLO-teleop takes, 30 fps) in the
**EEF-delta action space** (`eef_delta_v1`). This is the **60k-step checkpoint** (best open-loop
chunk-30 error of 10k..100k; kanu kept only 100k).

## Observation / action space (`eef_delta_v1`)
- `observation.state` **16-D** = `[q1..q6 (rad, UR order), tcp_x, tcp_y, tcp_z (m, base_link),
  r11, r21, r31, r12, r22, r32 (first two columns of the TCP rotation), grip_pos (0=open..1=closed)]`.
  TCP = `ur_kin.fk(q)` + 0.174 m along flange +Z.
- `action` **7-D** = `[dx, dy, dz, drx, dry, drz, grip_cmd]` — achieved per-frame TCP motion
  (`dp = p_{t+1}-p_t`, `drot = so3_log(R_{t+1} R_t^T)`, base frame), gripper absolute 0..1.
  Deploy: `p_target = p_live + dp`, `R_target = so3_exp(drot) R_live`, analytic IK, branch locking.
- Cameras: `cam1` (scene), `cam2` (wrist), native 720×1280 fed in; the policy resizes internally
  to `image_resize_shape = [224, 224]` — **do not pre-resize** (double-resize silently degrades).
- Task string: `"Put carrot in pot"` (CLIP-conditioned; send it every tick).
- `chunk_size 32`, `n_action_steps 24`, `n_obs_steps 2`, `num_integration_steps` 100 at train
  time (10 is the usual serving override), DiT hidden 512 × 6 layers.

## Training
- Dataset: `carrot_in_pot_eef_lerobot_v3` — local LeRobot v3 re-export of
  [`Bigenlight/carrot_in_pot_lerobot_v3`](https://huggingface.co/datasets/Bigenlight/carrot_in_pot_lerobot_v3)
  (54 ep / 17,085 frames; joints de-lagged by per-take τ≈0.90 s + linear re-interpolation).
  Not yet on the Hub.
- `lerobot-train`, batch 8, seed 1000, 100k steps (10 h 15 m on one RTX A4000, kanu), job `fm_carrot_eef`.

## Held-out results (open-loop, chunk-30 cumulative error, episodes 48–53)
| 10k | 20k | 30k | **60k (this)** | 100k | zero-motion |
|---|---|---|---|---|---|
| 34.8 mm | 32.8 mm | ~33 mm | **32.0 mm** | 32.6 mm | 65.6 mm |

lerobot's `eval_loss` rose monotonically 0.096→0.233 over the run while the open-loop metric
kept improving — it was ignored for checkpoint selection.

## Status
Real-robot closed-loop evaluation: **not yet run**. The shipped ZMQ servers
(`gello_policy/policy_server/fm_server.py`) are joint-space 7/7 and refuse this checkpoint's
16-D state; serving needs the EEF-mode deploy node from branch `feat/carrot-eef-il`.