prehj's picture
v17b final speed modules (LIBERO, RoboCasa, RoboCasa365) + model card
ed3baa9 verified
|
Raw History Blame Contribute Delete
5.37 kB
---
license: apache-2.0
tags:
- robotics
- vla
- gr00t
- libero
- robocasa
- action-chunking
- inference-speedup
---
# Faster-VLA speed module (per-chunk speed-up predictor for frozen GR00T-N1.5)
A small MLP that predicts, for each action chunk of a **frozen** VLA policy, how much faster the chunk can be executed
(speed-up `K ∈ [1, 3]`). The policy is never retrained. At run time the policy's predicted chunk is compressed post hoc
by `K` (consecutive delta actions merged, exact block rule) and the gripper is time-scaled by the same factor.
Code, label toolchain and evaluation: <https://github.com/rakybond007/faster_vla_harness> (see `docs/TESTING.md`).
## Inputs / output
| input | shape | source at run time |
|---|---|---|
| image features | `V × 768` | frozen DINOv3 ViT-S/16 (`facebook/dinov3-vits16-pretrain-lvd1689m`, gated on HF) of every camera the policy sees (LIBERO: front + wrist, RoboCasa: left + right + wrist) |
| state | LIBERO 8, RoboCasa 9 | proprioception (LIBERO: eef pos, axis-angle, finger joints; RoboCasa: relative eef pos / quat, finger joints) |
| predicted chunk | `16 × 7` | the frozen policy's own predicted action chunk |
| **output** | scalar | `K = 1 + 2·sigmoid(z)` |
No task identity is given to the module.
## Training labels (no rollouts used)
`K_max` per demonstration chunk from measured, real-robot-applicable factors:
- tolerance = min(demonstration spread, geometric margin)
- geometric margin: grasp / place margin near a contact, wrist-camera clearance elsewhere
- the largest `K` whose calibrated extra controller tracking error (position + camera lever arm × orientation) stays within the tolerance
- capped by the Franka joint velocity / acceleration / torque limits
- v17b: careful before every contact; with an open hand, the tolerance is also bounded by the side clearance (wrist-camera distance to everything except the object about to be grasped)
- final transform: labels below 2 are lifted toward 1.5 (`K' = 1.5 + 0.5 (K − 1)`, order kept) and capped at 2.5
Simulator ground truth was used only to validate the tools. Hyper-parameters were chosen on held-out label error
(pre-registered grid), never on rollouts.
## Files
| path | base policy | labels | notes |
|---|---|---|---|
| `libero/v17b/` + `ensemble/seed{1..4}/` | [prehj/GR00T-N1.5-libero-baseline-bs32-60k](https://huggingface.co/prehj/GR00T-N1.5-libero-baseline-bs32-60k) | v17b | **final**; K = min over the five members, no floor |
| `robocasa/v17b/` + `ensemble/seed{1..4}/` | [prehj/GR00T-N1.5-robocasa-baseline](https://huggingface.co/prehj/GR00T-N1.5-robocasa-baseline) | v17b | **final** |
| `robocasa365/v17b/` + `ensemble/seed{1..4}/` | `robocasa/robocasa365_checkpoints` gr00t_n1-5/multitask_learning/checkpoint-120000 | v17b | **final**; also compresses mobile-base chunks |
| `*/v16b` (+ ensembles) | | v16b | previous version (labels as measured, no lift / cap) |
| `libero/v13`, `robocasa/v13` (+ ensembles), `libero/v11`, `libero/v10`, `robocasa/v12`, `robocasa/v11` | | | earlier versions |
Each folder: `model.pt`, `norm.npz`, `metrics.json` (+ `val_pred.parquet` for the main model). `SHA256SUMS` lists the
`model.pt` hashes.
## Results (closed loop, final v17b, no speed floor)
Paired on the same initial states, task-stratified bootstrap. Speed-up = env steps of the reference / env steps of the
arm, on episodes both solve.
| benchmark | 1x | uniform 1.5x | ours (v17b) | speed-up vs 1x | Δ vs uniform 1.5x [95% CI] | speed-up vs uniform 1.5x [95% CI] |
|---|---|---|---|---|---|---|
| LIBERO (40 tasks x 50) | 96.2% | 94.4% | 93.0% | 1.64 | −1.4 [−2.7, +0.0] | 1.21 [1.19, 1.22] |
| RoboCasa (24 tasks x 50) | 66.4% | 66.4% | 65.3% | 1.67 | −1.1 [−3.8, +1.6] | 1.20 [1.13, 1.27] |
| RoboCasa365 (18 tasks x 50) | 51.8% | 53.4% | 54.6% | 1.52 | +1.1 [−2.2, +4.6] | 1.15 [1.06, 1.24] |
At the success rate of uniform 1.5x, the per-chunk module is 1.15–1.21x faster than uniform 1.5x.
## Usage
```bash
pip install huggingface_hub
huggingface-cli download prehj/faster-vla-speed-module --local-dir speed_module
# K server (from the git repo): one model, or a comma-separated list for the ensemble
M=speed_module/libero/v17b; ENS=$M/model.pt; for s in 1 2 3 4; do ENS=$ENS,$M/ensemble/seed$s/model.pt; done
python tools/qmodule/server.py --bench libero --model $ENS --k-min 1.0 --port 9800 # no floor (final setting)
curl -X POST localhost:9800/k -d '{"images": [...], "shape": [...], "state": [...], "pred": [[...]]}'
```
The full closed-loop protocol (policy server + K server + LIBERO / RoboCasa clients) is in `docs/TESTING.md` of the
git repo.
## Limitations
- **Policies:** trained for GR00T-N1.5 fine-tuned on LIBERO / RoboCasa. The module conditions on that policy's predicted chunks.
- **LIBERO:** the remaining loss is concentrated in stove + moka pot (`libero_10_2`, `libero_10_8`).
- **RoboCasa:** the wrist-clearance tool still under-detects near drawer / door fronts.
- An execution floor (K >= 1.5) used in earlier versions overrode 40-56% of the labels and was dropped; the final lift
keeps the order of the tool labels instead of overriding them.
- The side clearance uses monocular depth (Depth Anything 3); at close range it reads obstacles too close (median 6 mm vs
11 mm on simulator depth), so some chunks near a grasp are slower than necessary.