Franka 5-task co-training — AdaSpeVLA with the pi0.5 action expert
One AdaSpeVLA policy (cubic B-spline action chunks with learned knots and an adaptive query-interval / timing head)
trained jointly on five Franka tasks. The control points are generated by the pretrained π0.5 action expert
(Gemma-300M + adaRMS, loaded from pi05_base; generator="pi05_expert") on the π0.5 PaliGemma backbone.
The task is selected by the language prompt.
Contents
Root of the repo = one OpenPI/Orbax inference checkpoint (timing stage = full model):
params/: EMA parameters; backbone, action expert, knot head from the 40k-step action stage (bfloat16, frozen in the timing stage), timing head (freq_head) in float32. No optimizer state.assets/cotrain/franka5/norm_stats.json: joint normalization statistics of the five tasks (tasks weighted equally; computed with OpenPI's RunningStats on the training episodes). Use these, not the single-task stats.
60k/: the same run continued to 60,000 action steps (same schedule: the LR is at its floor of 2.5e-6 after 30k),
with its own timing head trained from 60k; same layout and norm stats. Evaluation below.
Data
| Task | Dataset | Train / total episodes | Held-out episodes | Prompt |
|---|---|---|---|---|
| bread | dddddsds31341/bread-lerobot @ a45b8ba |
90 / 100 | [1, 3, 7, 17, 25, 29, 47, 58, 77, 81] | Pick up both slices of bread from the toaster and place them on the plate. |
| test_tube | dddddsds31341/test-tube-lerobot @ 29d821a |
89 / 99 | [1, 3, 7, 17, 25, 28, 47, 57, 76, 80] | Place the test tube into the test tube rack. |
| tennis | dddddsds31341/tennis-balls-lerobot @ 04a27cd |
90 / 100 | [1, 3, 7, 17, 25, 29, 47, 58, 77, 81] | Place all tennis balls into the green basket. |
| insert_tube | dddddsds31341/insert-tube-lerobot @ 70bcd1f |
92 / 102 | [1, 4, 7, 17, 25, 29, 48, 59, 79, 82] | Insert the gray tube into the empty black holder. |
| wipe_whiteboard | dddddsds31341/wipe-whiteboard-lerobot @ 2c8cfa8 |
102 / 112 | [1, 4, 8, 19, 28, 32, 53, 66, 87, 91] | Erase the writing from the whiteboard. |
Tasks are sampled uniformly (every task equally often; smaller tasks are cycled more), as in the original AdaSpeVLA 5-task recipe.
Training
- Code:
cognaclee/Frank-Insert-Wipe@037c782, foldercode/adaspe_train(OpenPI215abfb), imported unchanged; the mixture, the joint norm stats and per-task labels are added around it. - Recipe (2026-11-10 revision): 16-step chunk (1.5 s at 10 Hz), 11 control points, query interval 0.2–1.5 s, timing / knot labels from measured motion and the commanded gripper state, e_crit-normalized knot density. Each task keeps its own calibration (motion scale, e_crit): every sample carries its task's scale × e_crit, which is exactly equivalent to the single-task labels.
- Action expert: π0.5 flow convention (x_t = t·noise + (1−t)·data, t ~ Beta(1.5,1)), 10 ODE steps; knot information
enters each control-point token through the zero-initialized
cp_knot_proj. - Action stage 40,000 steps (OpenPI defaults: AdamW, cosine LR warmup 1k, peak 2.5e-5, decay 30k; EMA 0.99;
batch 32; 4 GPUs, FSDP), full fine-tuning from
pi05_base. Timing stage: 10,000 steps on top of the 40k checkpoint, only the timing head trained.
Open-loop evaluation (1,024 frames per task; normalized actions over the 16-step chunk)
R² relative to predicting the per-dimension mean.
Held-out episodes (never seen in training):
| Task | R² x | R² y | R² z | R² rx | R² ry | R² rz | R² grip | gripper sign acc. | interval MAE / corr. |
|---|---|---|---|---|---|---|---|---|---|
| bread | 0.18 | 0.93 | 0.82 | 0.14 | 0.30 | 0.23 | 0.75 | 94.8% | 0.315 s / 0.55 |
| test_tube | 0.43 | 0.84 | 0.77 | 0.16 | 0.13 | 0.20 | 0.64 | 92.1% | 0.343 s / 0.52 |
| tennis | 0.59 | 0.77 | 0.69 | 0.28 | 0.44 | 0.41 | 0.77 | 94.5% | 0.387 s / 0.52 |
| insert_tube | 0.18 | 0.85 | 0.86 | 0.36 | 0.33 | 0.31 | 0.72 | 95.4% | 0.345 s / 0.57 |
| wipe_whiteboard | 0.56 | 0.61 | 0.28 | 0.37 | 0.39 | 0.11 | 0.68 | 94.0% | 0.408 s / 0.32 |
Training frames:
| Task | R² x | R² y | R² z | R² rx | R² ry | R² rz | R² grip | gripper sign acc. | interval MAE / corr. |
|---|---|---|---|---|---|---|---|---|---|
| bread | 0.90 | 0.99 | 0.97 | 0.89 | 0.86 | 0.85 | 0.97 | 99.2% | 0.283 s / 0.69 |
| test_tube | 0.91 | 0.97 | 0.95 | 0.84 | 0.86 | 0.87 | 0.94 | 98.6% | 0.284 s / 0.69 |
| tennis | 0.92 | 0.96 | 0.95 | 0.89 | 0.90 | 0.91 | 0.96 | 98.8% | 0.372 s / 0.52 |
| insert_tube | 0.88 | 0.97 | 0.98 | 0.93 | 0.93 | 0.91 | 0.96 | 99.4% | 0.325 s / 0.64 |
| wipe_whiteboard | 0.96 | 0.98 | 0.93 | 0.95 | 0.96 | 0.91 | 0.96 | 99.3% | 0.348 s / 0.57 |
Continued to 60k (60k/), held-out episodes:
| Task | R² x | R² y | R² z | R² rx | R² ry | R² rz | R² grip | gripper sign acc. | interval MAE / corr. |
|---|---|---|---|---|---|---|---|---|---|
| bread | 0.21 | 0.93 | 0.82 | 0.14 | 0.33 | 0.24 | 0.75 | 94.9% | 0.316 s / 0.55 |
| test_tube | 0.45 | 0.84 | 0.77 | 0.16 | 0.14 | 0.23 | 0.64 | 92.2% | 0.338 s / 0.52 |
| tennis | 0.59 | 0.77 | 0.69 | 0.28 | 0.44 | 0.41 | 0.76 | 94.3% | 0.391 s / 0.52 |
| insert_tube | 0.21 | 0.85 | 0.86 | 0.34 | 0.34 | 0.31 | 0.73 | 95.3% | 0.346 s / 0.57 |
| wipe_whiteboard | 0.57 | 0.61 | 0.27 | 0.37 | 0.40 | 0.11 | 0.68 | 93.8% | 0.402 s / 0.33 |
60k, training frames:
| Task | R² x | R² y | R² z | R² rx | R² ry | R² rz | R² grip | gripper sign acc. | interval MAE / corr. |
|---|---|---|---|---|---|---|---|---|---|
| bread | 0.92 | 0.99 | 0.98 | 0.91 | 0.89 | 0.88 | 0.97 | 99.4% | 0.283 s / 0.70 |
| test_tube | 0.93 | 0.98 | 0.96 | 0.88 | 0.89 | 0.89 | 0.96 | 99.0% | 0.274 s / 0.71 |
| tennis | 0.94 | 0.97 | 0.96 | 0.91 | 0.93 | 0.93 | 0.97 | 99.2% | 0.376 s / 0.51 |
| insert_tube | 0.91 | 0.98 | 0.99 | 0.94 | 0.95 | 0.94 | 0.97 | 99.4% | 0.329 s / 0.64 |
| wipe_whiteboard | 0.97 | 0.98 | 0.94 | 0.96 | 0.97 | 0.93 | 0.97 | 99.3% | 0.344 s / 0.58 |
Bread, same held-out frames, single-task model with the same generator and recipe (Gospe1/Franka-Bread-AdaSpeVLA-Expert,
20k steps on bread only):
| Task | R² x | R² y | R² z | R² rx | R² ry | R² rz | R² grip | gripper sign acc. | interval MAE / corr. |
|---|---|---|---|---|---|---|---|---|---|
| bread (single-task, 20k) | 0.18 | 0.92 | 0.82 | 0.14 | 0.32 | 0.20 | 0.75 | 95.1% | 0.300 s / 0.56 |
Real-robot results are not included.
Use
Needs code/adaspe_train from cognaclee/Frank-Insert-Wipe @ 037c782 and OpenPI 215abfb.
Model config: configs.model_config("timing", e_crit=1.0, motion_scale=(1.0,) * 6, generator="pi05_expert")
(e_crit / motion_scale only matter for training labels). Normalize observations / unnormalize actions with
assets/cotrain/franka5/norm_stats.json; the prompt selects the task (table above). Load params/ by mapping the
saved keys onto the model's own keys (the plugin's int layer keys come back as strings from Orbax, so
cfg.load(restore_params(...)) fails with a pytree-structure error):
import flax.nnx as nnx, flax.traverse_util as tu, jax, numpy as np
import openpi.models.model as _model
def load(cfg, params_path):
model = nnx.eval_shape(lambda: cfg.create(jax.random.key(0)))
graphdef, state = nnx.split(model)
name = lambda k: "/".join(map(str, k))
expected = tu.flatten_dict(state.to_pure_dict())
loaded = {name(k): v for k, v in tu.flatten_dict(_model.restore_params(params_path, restore_type=np.ndarray)).items()}
assert {name(k) for k in expected} == set(loaded)
state.replace_by_pure_dict(tu.unflatten_dict({k: jax.numpy.asarray(loaded[name(k)], dtype=s.dtype) for k, s in expected.items()}))
return nnx.merge(graphdef, state)
model.sample_spline(rng, observation) returns dense_actions (16 normalized commands), query_interval and
execute_steps: execute dense_actions[:, :execute_steps] (one 0.1 s body-frame delta command each), then observe and
query again; or execute a fixed number of commands to ignore the timing head. Validate observation ordering, units,
normalization and workspace limits in a controlled setting before running on hardware.