File size: 11,982 Bytes
c3c3651 67c4f10 c3c3651 67c4f10 c3c3651 67c4f10 f98e714 67c4f10 f98e714 67c4f10 f98e714 67c4f10 f98e714 67c4f10 f98e714 67c4f10 f98e714 67c4f10 f98e714 67c4f10 f98e714 67c4f10 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 | ---
language:
- en
license: apache-2.0
tags:
- robotics
- flow-matching
- world-models
- imitation-learning
- libero
- action-chunking
- pytorch
- safetensors
pipeline_tag: robotics
---
# PARVULUS: 10M-Scale World Action Model for Robot Manipulation
**PARVULUS** is an ultra-efficient 10M-parameter Flow Matching World Action Model (WAM) designed for continuous robot manipulation. Trained on the canonical [LIBERO](https://github.com/Lifelong-Robot-Learning/LIBERO) benchmark, Parvulus achieves a **90.88% average success rate across all 40 tasks** while running at real-time control frequencies (>100 Hz in C++/CUDA, sub-40ms with PyTorch KV-caching).
---
## 1. Highlights
- **9.98M parameters** (~20 MB in BF16/FP16, ~38 MB in FP32) for standalone robot control.
- **90.88% Β± 0.37%** average success across all 40 canonical LIBERO tasks (N = 6,000 simulation rollouts).
- Pinned `google-t5/t5-small` text representations with a trained 2-layer projection head.
- **Multiple Execution Modes**:
- **PyTorch KV-Cached**: β 38.8 ms (206 Hz action bandwidth).
- **Standalone C++ / CUDA Engine**: **7.28 ms** (>1,000 Hz action bandwidth, pure CUDA/cuBLAS).
- **Intel Iris Xe Hybrid**: β 106.6 ms on low-power laptop integrated graphics (`xpu` vision + CPU ODE).
- Includes optional auxiliary training decoders (`parvulus_auxiliary.safetensors`, 29.15M params) for auto-regressive visual self-rollout, motion change estimation, and self-reconstruction from noisy images.
---
## 2. Benchmark Performance
Evaluated across the 4 canonical LIBERO suites (N = 6,000 physical rollouts, 10 seeds, receding-horizon chunking H = 8, C = 8):
| Benchmark Suite | Canonical Tasks | Rollout Trials | Success Rate |
| :--- | :---: | :---: | :---: |
| **LIBERO-Spatial** | 10 | 1,500 | **92.67%** |
| **LIBERO-Object** | 10 | 1,500 | **91.13%** |
| **LIBERO-Goal** | 10 | 1,500 | **89.47%** |
| **LIBERO-10 (Long)** | 10 | 1,500 | **90.27%** |
| **Full Benchmark Average** | **40** | **6,000** | **90.88% Β± 0.37%** |
---
## 3. Repository Structure
```
.
βββ parvulus.safetensors # Action-only core policy weights (9.98M params, ~39 MB FP32)
βββ parvulus_full.safetensors # Full model weights: action policy + auxiliary heads (39.12M params, ~156 MB)
βββ parvulus_auxiliary.safetensors # Auxiliary world model heads only (video, change, self-recon; 29.15M params)
βββ model.safetensors # HuggingFace default entrypoint (alias of parvulus.safetensors)
β
βββ config.json # Model hyperparameters and pinned T5 commit revision
βββ action_scale.json # 7-DoF per-channel min/max normalization boundaries
βββ pyproject.toml # Standard package configuration
βββ requirements.txt # Minimal Python dependencies
βββ prepare_runtime.py # Setup script: builds C engine, generates binaries, prepares text encoder
β
βββ parvulus_runtime.py # High-performance Python runtime for standalone C engine
βββ infer.py # High-level ParvulusInference client (supports CUDA, C engine, XPU, Hybrid)
βββ parvulus.py # Complete PyTorch architecture (DualViewEncoder, CVMixer, FlowDecoder)
βββ text_encoder.py # Pinned T5 text tokenizer and projection head
βββ action_utils.py # Action normalization and denormalization helpers
β
βββ c_engine/ # Dependency-free C++ / CUDA Standalone Engine
βββ Makefile # Make-based build script
βββ build.sh # Convenient build script
βββ parvulus_engine.cu / .h # Core C++/CUDA inference engine (136 Hz chunk rate)
βββ vision_encoder.cu / .cuh # Fused DualView vision encoder
βββ parvulus_c_api.cpp / .h # Standard C ABI exports for C/C++/ROS 2/Rust
βββ parvulus_runtime_cli.cpp # Standalone CLI binary entry point
βββ canonical_tasks.json # 40 canonical task instructions
βββ export_from_safetensors.py # Script generating model binary, scale, and embeddings
βββ export_text_encoder_onnx.py # Standalone converter for T5-small text encoder to ONNX
```
---
## 4. Quickstart: Python Inference
### Installation
```bash
pip install -r requirements.txt
```
### 1. Basic Action Chunk Prediction
```python
import numpy as np
from PIL import Image
from infer import ParvulusInference
# Initialize client (auto-detects CUDA / XPU / CPU; enables fast KV-cache by default)
infer = ParvulusInference.from_pretrained(
".",
device="cuda", # Options: 'cuda', 'cpu', 'xpu', 'hybrid', or 'auto'
use_kv_cache=True, # Enables sub-40ms inference
num_ode_steps=10
)
# Prepare inputs: dual 256x256 RGB cameras + text prompt
agentview_img = np.array(Image.open("agentview.png")) # [256, 256, 3] uint8
wristview_img = np.array(Image.open("wristview.png")) # [256, 256, 3] uint8
task_prompt = "pick up the black bowl in the center and place it on the plate"
# Predict 8-step continuous action chunk [8, 7]
# (dx, dy, dz, drot_x, drot_y, drot_z, gripper)
action_chunk = infer.predict_chunk(agentview_img, wristview_img, task_description=task_prompt)
print(f"Predicted action chunk (shape {action_chunk.shape}):\n{action_chunk}")
```
### 2. Environment Stepping Loop (Receding Horizon)
```python
infer.reset(task_description="pick up the milk and place it in the basket")
obs = env.reset()
agentview_img = obs["agentview_image"]
wristview_img = obs["robot0_eye_in_hand_image"]
done = False
while not done:
# In your simulation or real robot loop:
action = infer.step(agentview_img, wristview_img) # [7] float32 action
obs, reward, done, info = env.step(action)
agentview_img = obs["agentview_image"]
wristview_img = obs["robot0_eye_in_hand_image"]
```
### 3. Continuous Overlapped Pipelined Execution (H = 8, C = 4)
To achieve zero-latency continuous robot motion, Parvulus supports overlapping chunk generation with robot motion execution:
1. Predict an 8-step chunk (A_0).
2. Execute the first 4 actions (A_0[0:4]).
3. At step 4, capture a fresh observation and trigger background inference for the next chunk (A_1).
4. Concurrently execute the remaining tail actions (A_0[4:8]) on the robot while the model predicts A_1.
5. Transition seamlessly into A_1 with zero robot idling.
```python
from concurrent.futures import ThreadPoolExecutor
executor = ThreadPoolExecutor(max_workers=1)
infer.reset(task_description="pick up the black bowl in the center and place it on the plate")
# 1. Reset environment and capture initial observation at t = 0
obs = env.reset()
agentview_img = obs["agentview_image"]
wristview_img = obs["robot0_eye_in_hand_image"]
# 2. Generate initial 8-step action chunk (covers t = 0..7)
current_chunk = infer.predict_chunk(agentview_img, wristview_img)
# 3. Execute the initial head (t = 0..3) before entering the pipelined loop
done = False
for a in current_chunk[0:4]:
obs, reward, done, _ = env.step(a)
if done:
break
# Continuous overlapped loop: each cycle executes tail actions (4..7) while predicting the next chunk
while not done:
# 4. Capture observation at step t = 4 (or 4k)
agentview_img = obs["agentview_image"]
wristview_img = obs["robot0_eye_in_hand_image"]
# 5. Trigger background inference for next chunk (covers upcoming horizon)
future_chunk = executor.submit(infer.predict_chunk, agentview_img, wristview_img)
# 6. Overlap: Execute remaining 4 tail actions (4..7) while background inference runs
for a in current_chunk[4:8]:
obs, reward, done, _ = env.step(a)
if done:
break
if done:
break
# 7. Retrieve newly generated chunk with zero robot idle time
current_chunk = future_chunk.result()
```
### 4. World-Model Self-Rollout (Mental Simulation)
Parvulus provides optional auxiliary decoders (`parvulus_auxiliary.safetensors`) for predicting future frames without interacting with the physical environment:
```python
infer = ParvulusInference.from_pretrained(
".",
load_auxiliary=True,
device="cuda"
)
# Predict action chunk and imagined next observation
action_chunk, next_agent_img, next_wrist_img = infer.rollout_step(
agentview_img, wristview_img, task_description="push the button"
)
```
---
## 5. Standalone Native C++ / CUDA Runtime (Vector-First, 7.3 ms / chunk)
For high-frequency physical robot deployments or low-overhead environments, Parvulus includes a standalone C++/CUDA runtime engine in `c_engine/` with **few dependencies**.
### Features
- Complete dual-camera visual encoding + 10-step Euler ODE flow matching loop runs in **<8 ms** (>1,000 Hz action bandwidth).
- Natively conditioned on 256-dim continuous embedding vectors (`task_vector`) to eliminate text tokenization latency in the robot control loop.
- Automatically maps arbitrary natural language instructions or perturbations into 256-dim vectors via ONNX Runtime (<4 ms) before passing to the engine.
### Preparing & Building the Runtime
```bash
# Automated preparation (builds C engine, prepares weights & ONNX text encoder)
python3 prepare_runtime.py
# Or build C++ engine directly:
cd c_engine && make -j$(nproc)
```
This produces:
- `libparvulus.so`: Shared C ABI library for Python ctypes, C, C++, ROS 2, or Rust.
- `parvulus_runtime`: Standalone CLI binary for headless benchmarks and batch action generation.
- `export_text_encoder_onnx.py`: Converter utility to export the 136 MB ONNX text encoder on demand (or download it from Hugging Face).
### Standalone CLI Benchmark
```bash
./c_engine/parvulus_runtime --benchmark 100
```
### Python Quickstart (`parvulus_runtime.py`)
```python
import numpy as np
from parvulus_runtime import ParvulusRuntime
# Initialize standalone C++/CUDA engine
runtime = ParvulusRuntime(device_id=0, num_euler_steps=10)
# 1. Condition with a 256-dim task vector:
runtime.reset(task_vector=my_256_dim_vector)
# Or condition with arbitrary text (automatic dynamic ONNX encoding):
# runtime.reset(task_description="grasp the black bowl and place it on the plate")
# 2. Step in real-time robot loop (streams only camera frames at 136 Hz):
action = runtime.step(agentview_img, wristview_img) # [7] float32
```
For detailed C ABI signatures and non-Python integration instructions, see [`c_engine/README.md`](c_engine/README.md).
---
## 6. Intel Iris Xe / Arc Heterogeneous Hybrid Mode
For deployment on mobile workstations or edge devices with Intel Iris Xe or Arc integrated graphics:
```python
infer = ParvulusInference.from_pretrained(".", device="hybrid")
```
In `hybrid` mode:
1. Perception backbone (`DualViewEncoder`) executes on the iGPU (`torch.xpu`).
2. Only 16 pooled visual tokens (16 KB) transfer over PCIe to the host CPU.
3. The 10-step ODE Transformer loop runs on the high-clock CPU cores.
---
## 7. Model Details & Pinned Dependencies
- **Input Specifications**:
- `agentview`: [256, 256, 3] RGB.
- `wristview`: [256, 256, 3] RGB.
- `instruction`: Natural language string.
- **Output Action Format**:
- 8-step continuous trajectory A_t [8, 7] (dx, dy, dz, dΞΈx, dΞΈy, dΞΈz, gripper).
- **Pinned Text Model**:
- `google-t5/t5-small` pinned to commit revision `df1b051c49625cf57a3d0d8d3863ed4d13564fe4`.
---
## Citation
Parvulus model weights and the associated inference code are released under a permissive license. If you find our work useful, please cite:
```bibtex
@inproceedings{
sivas2026parvulus,
title={Parvulus: A 10M-Scale World Action Model for Robot Manipulation},
author={Abdullah Ali Sivas and Peter Cho-Ho Lam and Zirui Zhou and Yong Zhang},
booktitle={CoRL 2026 Workshop on The Action Gap: Large Generative Models Learn Rich Knowledge},
year={2026},
url={https://openreview.net/forum?id=VcS0dE1Aoz}
}
```
|