Franka 5-task co-training — AdaSpeVLA with the pi0.5 action expert

One AdaSpeVLA policy (cubic B-spline action chunks with learned knots and an adaptive query-interval / timing head) trained jointly on five Franka tasks. The control points are generated by the pretrained π0.5 action expert (Gemma-300M + adaRMS, loaded from pi05_base; generator="pi05_expert") on the π0.5 PaliGemma backbone. The task is selected by the language prompt.

Contents

Root of the repo = one OpenPI/Orbax inference checkpoint (timing stage = full model):

  • params/: EMA parameters; backbone, action expert, knot head from the 40k-step action stage (bfloat16, frozen in the timing stage), timing head (freq_head) in float32. No optimizer state.
  • assets/cotrain/franka5/norm_stats.json: joint normalization statistics of the five tasks (tasks weighted equally; computed with OpenPI's RunningStats on the training episodes). Use these, not the single-task stats.

60k/: the same run continued to 60,000 action steps (same schedule: the LR is at its floor of 2.5e-6 after 30k), with its own timing head trained from 60k; same layout and norm stats. Evaluation below.

Data

Task Dataset Train / total episodes Held-out episodes Prompt
bread dddddsds31341/bread-lerobot @ a45b8ba 90 / 100 [1, 3, 7, 17, 25, 29, 47, 58, 77, 81] Pick up both slices of bread from the toaster and place them on the plate.
test_tube dddddsds31341/test-tube-lerobot @ 29d821a 89 / 99 [1, 3, 7, 17, 25, 28, 47, 57, 76, 80] Place the test tube into the test tube rack.
tennis dddddsds31341/tennis-balls-lerobot @ 04a27cd 90 / 100 [1, 3, 7, 17, 25, 29, 47, 58, 77, 81] Place all tennis balls into the green basket.
insert_tube dddddsds31341/insert-tube-lerobot @ 70bcd1f 92 / 102 [1, 4, 7, 17, 25, 29, 48, 59, 79, 82] Insert the gray tube into the empty black holder.
wipe_whiteboard dddddsds31341/wipe-whiteboard-lerobot @ 2c8cfa8 102 / 112 [1, 4, 8, 19, 28, 32, 53, 66, 87, 91] Erase the writing from the whiteboard.

Tasks are sampled uniformly (every task equally often; smaller tasks are cycled more), as in the original AdaSpeVLA 5-task recipe.

Training

  • Code: cognaclee/Frank-Insert-Wipe @ 037c782, folder code/adaspe_train (OpenPI 215abfb), imported unchanged; the mixture, the joint norm stats and per-task labels are added around it.
  • Recipe (2026-11-10 revision): 16-step chunk (1.5 s at 10 Hz), 11 control points, query interval 0.2–1.5 s, timing / knot labels from measured motion and the commanded gripper state, e_crit-normalized knot density. Each task keeps its own calibration (motion scale, e_crit): every sample carries its task's scale × e_crit, which is exactly equivalent to the single-task labels.
  • Action expert: π0.5 flow convention (x_t = t·noise + (1−t)·data, t ~ Beta(1.5,1)), 10 ODE steps; knot information enters each control-point token through the zero-initialized cp_knot_proj.
  • Action stage 40,000 steps (OpenPI defaults: AdamW, cosine LR warmup 1k, peak 2.5e-5, decay 30k; EMA 0.99; batch 32; 4 GPUs, FSDP), full fine-tuning from pi05_base. Timing stage: 10,000 steps on top of the 40k checkpoint, only the timing head trained.

Open-loop evaluation (1,024 frames per task; normalized actions over the 16-step chunk)

R² relative to predicting the per-dimension mean.

Held-out episodes (never seen in training):

Task R² x R² y R² z R² rx R² ry R² rz R² grip gripper sign acc. interval MAE / corr.
bread 0.18 0.93 0.82 0.14 0.30 0.23 0.75 94.8% 0.315 s / 0.55
test_tube 0.43 0.84 0.77 0.16 0.13 0.20 0.64 92.1% 0.343 s / 0.52
tennis 0.59 0.77 0.69 0.28 0.44 0.41 0.77 94.5% 0.387 s / 0.52
insert_tube 0.18 0.85 0.86 0.36 0.33 0.31 0.72 95.4% 0.345 s / 0.57
wipe_whiteboard 0.56 0.61 0.28 0.37 0.39 0.11 0.68 94.0% 0.408 s / 0.32

Training frames:

Task R² x R² y R² z R² rx R² ry R² rz R² grip gripper sign acc. interval MAE / corr.
bread 0.90 0.99 0.97 0.89 0.86 0.85 0.97 99.2% 0.283 s / 0.69
test_tube 0.91 0.97 0.95 0.84 0.86 0.87 0.94 98.6% 0.284 s / 0.69
tennis 0.92 0.96 0.95 0.89 0.90 0.91 0.96 98.8% 0.372 s / 0.52
insert_tube 0.88 0.97 0.98 0.93 0.93 0.91 0.96 99.4% 0.325 s / 0.64
wipe_whiteboard 0.96 0.98 0.93 0.95 0.96 0.91 0.96 99.3% 0.348 s / 0.57

Continued to 60k (60k/), held-out episodes:

Task R² x R² y R² z R² rx R² ry R² rz R² grip gripper sign acc. interval MAE / corr.
bread 0.21 0.93 0.82 0.14 0.33 0.24 0.75 94.9% 0.316 s / 0.55
test_tube 0.45 0.84 0.77 0.16 0.14 0.23 0.64 92.2% 0.338 s / 0.52
tennis 0.59 0.77 0.69 0.28 0.44 0.41 0.76 94.3% 0.391 s / 0.52
insert_tube 0.21 0.85 0.86 0.34 0.34 0.31 0.73 95.3% 0.346 s / 0.57
wipe_whiteboard 0.57 0.61 0.27 0.37 0.40 0.11 0.68 93.8% 0.402 s / 0.33

60k, training frames:

Task R² x R² y R² z R² rx R² ry R² rz R² grip gripper sign acc. interval MAE / corr.
bread 0.92 0.99 0.98 0.91 0.89 0.88 0.97 99.4% 0.283 s / 0.70
test_tube 0.93 0.98 0.96 0.88 0.89 0.89 0.96 99.0% 0.274 s / 0.71
tennis 0.94 0.97 0.96 0.91 0.93 0.93 0.97 99.2% 0.376 s / 0.51
insert_tube 0.91 0.98 0.99 0.94 0.95 0.94 0.97 99.4% 0.329 s / 0.64
wipe_whiteboard 0.97 0.98 0.94 0.96 0.97 0.93 0.97 99.3% 0.344 s / 0.58

Bread, same held-out frames, single-task model with the same generator and recipe (Gospe1/Franka-Bread-AdaSpeVLA-Expert, 20k steps on bread only):

Task R² x R² y R² z R² rx R² ry R² rz R² grip gripper sign acc. interval MAE / corr.
bread (single-task, 20k) 0.18 0.92 0.82 0.14 0.32 0.20 0.75 95.1% 0.300 s / 0.56

Real-robot results are not included.

Use

Needs code/adaspe_train from cognaclee/Frank-Insert-Wipe @ 037c782 and OpenPI 215abfb. Model config: configs.model_config("timing", e_crit=1.0, motion_scale=(1.0,) * 6, generator="pi05_expert") (e_crit / motion_scale only matter for training labels). Normalize observations / unnormalize actions with assets/cotrain/franka5/norm_stats.json; the prompt selects the task (table above). Load params/ by mapping the saved keys onto the model's own keys (the plugin's int layer keys come back as strings from Orbax, so cfg.load(restore_params(...)) fails with a pytree-structure error):

import flax.nnx as nnx, flax.traverse_util as tu, jax, numpy as np
import openpi.models.model as _model

def load(cfg, params_path):
    model = nnx.eval_shape(lambda: cfg.create(jax.random.key(0)))
    graphdef, state = nnx.split(model)
    name = lambda k: "/".join(map(str, k))
    expected = tu.flatten_dict(state.to_pure_dict())
    loaded = {name(k): v for k, v in tu.flatten_dict(_model.restore_params(params_path, restore_type=np.ndarray)).items()}
    assert {name(k) for k in expected} == set(loaded)
    state.replace_by_pure_dict(tu.unflatten_dict({k: jax.numpy.asarray(loaded[name(k)], dtype=s.dtype) for k, s in expected.items()}))
    return nnx.merge(graphdef, state)

model.sample_spline(rng, observation) returns dense_actions (16 normalized commands), query_interval and execute_steps: execute dense_actions[:, :execute_steps] (one 0.1 s body-frame delta command each), then observe and query again; or execute a fixed number of commands to ignore the timing head. Validate observation ordering, units, normalization and workspace limits in a controlled setting before running on hardware.

Downloads last month

-

Downloads are not tracked for this model. How to track
Video Preview
loading

Datasets used to train Gospe1/Franka-5Task-AdaSpeVLA-Expert