--- license: apache-2.0 tags: - robotics - vla - gr00t - libero - robocasa - action-chunking - inference-speedup --- # Faster-VLA speed module (per-chunk speed-up predictor for frozen GR00T-N1.5) A small MLP that predicts, for each action chunk of a **frozen** VLA policy, how much faster the chunk can be executed (speed-up `K ∈ [1, 3]`). The policy is never retrained. At run time the policy's predicted chunk is compressed post hoc by `K` (consecutive delta actions merged, exact block rule) and the gripper is time-scaled by the same factor. Code, label toolchain and evaluation: (see `docs/TESTING.md`). ## Inputs / output | input | shape | source at run time | |---|---|---| | image features | `V × 768` | frozen DINOv3 ViT-S/16 (`facebook/dinov3-vits16-pretrain-lvd1689m`, gated on HF) of every camera the policy sees (LIBERO: front + wrist, RoboCasa: left + right + wrist) | | state | LIBERO 8, RoboCasa 9 | proprioception (LIBERO: eef pos, axis-angle, finger joints; RoboCasa: relative eef pos / quat, finger joints) | | predicted chunk | `16 × 7` | the frozen policy's own predicted action chunk | | **output** | scalar | `K = 1 + 2·sigmoid(z)` | No task identity is given to the module. ## Training labels (no rollouts used) `K_max` per demonstration chunk from measured, real-robot-applicable factors: - tolerance = min(demonstration spread, geometric margin) - geometric margin: grasp / place margin near a contact, wrist-camera clearance elsewhere - the largest `K` whose calibrated extra controller tracking error (position + camera lever arm × orientation) stays within the tolerance - capped by the Franka joint velocity / acceleration / torque limits - v17b: careful before every contact; with an open hand, the tolerance is also bounded by the side clearance (wrist-camera distance to everything except the object about to be grasped) - final transform: labels below 2 are lifted toward 1.5 (`K' = 1.5 + 0.5 (K − 1)`, order kept) and capped at 2.5 Simulator ground truth was used only to validate the tools. Hyper-parameters were chosen on held-out label error (pre-registered grid), never on rollouts. ## Files | path | base policy | labels | notes | |---|---|---|---| | `libero/v17b/` + `ensemble/seed{1..4}/` | [prehj/GR00T-N1.5-libero-baseline-bs32-60k](https://huggingface.co/prehj/GR00T-N1.5-libero-baseline-bs32-60k) | v17b | **final**; K = min over the five members, no floor | | `robocasa/v17b/` + `ensemble/seed{1..4}/` | [prehj/GR00T-N1.5-robocasa-baseline](https://huggingface.co/prehj/GR00T-N1.5-robocasa-baseline) | v17b | **final** | | `robocasa365/v17b/` + `ensemble/seed{1..4}/` | `robocasa/robocasa365_checkpoints` gr00t_n1-5/multitask_learning/checkpoint-120000 | v17b | **final**; also compresses mobile-base chunks | | `*/v16b` (+ ensembles) | | v16b | previous version (labels as measured, no lift / cap) | | `libero/v13`, `robocasa/v13` (+ ensembles), `libero/v11`, `libero/v10`, `robocasa/v12`, `robocasa/v11` | | | earlier versions | Each folder: `model.pt`, `norm.npz`, `metrics.json` (+ `val_pred.parquet` for the main model). `SHA256SUMS` lists the `model.pt` hashes. ## Results (closed loop, final v17b, no speed floor) Paired on the same initial states, task-stratified bootstrap. Speed-up = env steps of the reference / env steps of the arm, on episodes both solve. | benchmark | 1x | uniform 1.5x | ours (v17b) | speed-up vs 1x | Δ vs uniform 1.5x [95% CI] | speed-up vs uniform 1.5x [95% CI] | |---|---|---|---|---|---|---| | LIBERO (40 tasks x 50) | 96.2% | 94.4% | 93.0% | 1.64 | −1.4 [−2.7, +0.0] | 1.21 [1.19, 1.22] | | RoboCasa (24 tasks x 50) | 66.4% | 66.4% | 65.3% | 1.67 | −1.1 [−3.8, +1.6] | 1.20 [1.13, 1.27] | | RoboCasa365 (18 tasks x 50) | 51.8% | 53.4% | 54.6% | 1.52 | +1.1 [−2.2, +4.6] | 1.15 [1.06, 1.24] | At the success rate of uniform 1.5x, the per-chunk module is 1.15–1.21x faster than uniform 1.5x. ## Usage ```bash pip install huggingface_hub huggingface-cli download prehj/faster-vla-speed-module --local-dir speed_module # K server (from the git repo): one model, or a comma-separated list for the ensemble M=speed_module/libero/v17b; ENS=$M/model.pt; for s in 1 2 3 4; do ENS=$ENS,$M/ensemble/seed$s/model.pt; done python tools/qmodule/server.py --bench libero --model $ENS --k-min 1.0 --port 9800 # no floor (final setting) curl -X POST localhost:9800/k -d '{"images": [...], "shape": [...], "state": [...], "pred": [[...]]}' ``` The full closed-loop protocol (policy server + K server + LIBERO / RoboCasa clients) is in `docs/TESTING.md` of the git repo. ## Limitations - **Policies:** trained for GR00T-N1.5 fine-tuned on LIBERO / RoboCasa. The module conditions on that policy's predicted chunks. - **LIBERO:** the remaining loss is concentrated in stove + moka pot (`libero_10_2`, `libero_10_8`). - **RoboCasa:** the wrist-clearance tool still under-detects near drawer / door fronts. - An execution floor (K >= 1.5) used in earlier versions overrode 40-56% of the labels and was dropped; the final lift keeps the order of the tool labels instead of overriding them. - The side clearance uses monocular depth (Depth Anything 3); at close range it reads obstacles too close (median 6 mm vs 11 mm on simulator depth), so some chunks near a grasp are slower than necessary.