README.md

---
library_name: lerobot
pipeline_tag: robotics
base_model: lerobot-data-collection/folding_final
datasets:
  - lerobot-data-collection/level2_final_quality3_t_0_hil_data_c
tags:
  - lerobot
  - pi05
  - robotics
  - real-time-chunking
  - rtc
---

# pi0.5 folding — training-time RTC, max delay 10 (step 3000)

π₀.₅ fine-tuned with **training-time real-time chunking** so that inference can run with
`--inference.rtc.mode=trained`: during training the last `d` actions of the previous chunk are
held clean and the remainder is denoised, with `d` drawn uniformly from 0…10 per example. The
model therefore learns to complete a chunk whose prefix is already committed, which is what makes
delay-compensated inference sound rather than approximate.

This is step 3000 of a planned 5000-step run. The interactive session hosting the job was lost at
around step 3800, so this is the last checkpoint written and the cosine schedule had not annealed
(LR was 1.2e-05 at the point of loss). Loss was flat from step 2100 onward, so the missing 2000
steps are unlikely to matter much for behaviour, but it is not a fully decayed model.

| | |
| --- | --- |
| base checkpoint | [`lerobot-data-collection/folding_final`](https://huggingface.co/lerobot-data-collection/folding_final) @ `695abe40dbf3aac04efda59c1501d748681fa0fb` |
| dataset | [`lerobot-data-collection/level2_final_quality3_t_0_hil_data_c`](https://huggingface.co/datasets/lerobot-data-collection/level2_final_quality3_t_0_hil_data_c) @ `2496db53d330c360f910d095e13698d968c56fc6` — 1319 episodes, 3,414,338 frames |
| code | `huggingface/lerobot` @ `4dbb965455ed8c260e5ba947e55558b4d4c14b5d` (training-time RTC, [PR #4056](https://github.com/huggingface/lerobot/pull/4056)) |
| `rtc_training_max_delay` | 10 |
| `chunk_size` | 30 |
| relative actions | enabled (`relative_exclude_joints: ['gripper']`) |
| hardware | 8×H100 80GB, one node, 45.96 GB per GPU |
| batch | 12 per GPU × 8 = 96 |
| optimizer | AdamW, LR 3.75e-05, weight decay 0.01, cosine decay with 500 warm-up steps over 5000 |
| throughput | 1.22 s/step, ~78 samples/s |

## Training command

Run under `accelerate` on a single 8×GPU node. `--policy.path` points at a local copy of the base
snapshot, because `folding_final` predates the `delta_actions_processor` →
`relative_actions_processor` registry rename and that one string has to be patched in
`policy_preprocessor.json` before it will load.

```bash
accelerate launch \
  --num_machines=1 \
  --num_processes=8 \
  --main_process_port=29827 \
  lerobot-train \
  --policy.path=/path/to/folding_final_snapshot_copy \
  --policy.rtc_training_max_delay=10 \
  --policy.device=cuda \
  --policy.dtype=bfloat16 \
  --policy.gradient_checkpointing=true \
  --policy.compile_model=false \
  --policy.push_to_hub=false \
  --policy.scheduler_warmup_steps=500 \
  --policy.scheduler_decay_steps=5000 \
  --dataset.repo_id=lerobot-data-collection/level2_final_quality3_t_0_hil_data_c \
  --dataset.root=/path/to/level2_final_quality3_t_0_hil_data_c_snapshot \
  --dataset.use_imagenet_stats=false \
  --output_dir=/path/to/output \
  --job_name=folding-final-rtc-d10-5k \
  --batch_size=12 \
  --steps=5000 \
  --num_workers=8 \
  --save_freq=1000 \
  --log_freq=100 \
  --seed=1000 \
  --wandb.enable=true \
  --wandb.project=folding-rtc \
  --wandb.disable_artifact=true
```

`--policy.compile_model=false` is required rather than cosmetic: the base checkpoint carries
`compile_model: true` with `compile_mode: max-autotune`, and triton fails to lower it
(`KeyError: 'cubin'`) on this cluster. `--policy.dtype=bfloat16` with no AMP keeps AdamW state in
bf16, which is why a full 4.1B-parameter fine-tune fits in 46 GB.

## Loss

| step | 100 | 600 | 1100 | 1600 | 2100 | 2600 | 3100 | 3600 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| loss | 0.428 | 0.036 | 0.029 | 0.026 | 0.026 | 0.023 | 0.023 | 0.021 |
| grad norm | 5.71 | 0.38 | 0.27 | 0.25 | 0.24 | 0.25 | 0.25 | 0.25 |

## Inference

```bash
lerobot-rollout \
  --strategy.type=base \
  --policy.path=nepyope/rtc \
  --inference.type=rtc \
  --inference.rtc.mode=trained \
  --inference.rtc.execution_horizon=10 \
  --inference.queue_threshold=10 \
  --robot.type=bi_openarm_follower \
  --task="Fold the T-shirt properly" \
  --fps=30 \
  --interpolation_multiplier=3
```

`execution_horizon` must satisfy `rtc_training_max_delay <= horizon <= chunk_size -
rtc_training_max_delay`, so the valid window here is **10 to 20**, and `queue_threshold` has the
same lower bound of 10. Measured rollout delay on the reference hardware is ~7 frames, so 10 is the
reactive end of the range; raise it if calls overrun and chunks get discarded. A one-off spike past
the trained delay discards that chunk and retries — only a sustained overrun is fatal.

The relative-actions processor must stay enabled: RTC inference requires an enabled
`RelativeActionsProcessorStep` to reconstruct absolute targets from the committed prefix.