File size: 11,982 Bytes
c3c3651
67c4f10
 
c3c3651
67c4f10
 
 
 
 
 
 
 
 
 
c3c3651
67c4f10
 
 
 
 
 
 
 
 
 
f98e714
67c4f10
 
f98e714
 
 
67c4f10
 
 
 
 
 
f98e714
67c4f10
 
 
 
 
 
 
f98e714
67c4f10
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
f98e714
67c4f10
 
f98e714
 
 
 
 
67c4f10
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
f98e714
 
67c4f10
 
f98e714
67c4f10
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
---
language:
- en
license: apache-2.0
tags:
- robotics
- flow-matching
- world-models
- imitation-learning
- libero
- action-chunking
- pytorch
- safetensors
pipeline_tag: robotics
---

# PARVULUS: 10M-Scale World Action Model for Robot Manipulation

**PARVULUS** is an ultra-efficient 10M-parameter Flow Matching World Action Model (WAM) designed for continuous robot manipulation. Trained on the canonical [LIBERO](https://github.com/Lifelong-Robot-Learning/LIBERO) benchmark, Parvulus achieves a **90.88% average success rate across all 40 tasks** while running at real-time control frequencies (>100 Hz in C++/CUDA, sub-40ms with PyTorch KV-caching).

---

## 1. Highlights

- **9.98M parameters** (~20 MB in BF16/FP16, ~38 MB in FP32) for standalone robot control.
- **90.88% Β± 0.37%** average success across all 40 canonical LIBERO tasks (N = 6,000 simulation rollouts).
- Pinned `google-t5/t5-small` text representations with a trained 2-layer projection head.
- **Multiple Execution Modes**:
  - **PyTorch KV-Cached**: β‰ˆ 38.8 ms (206 Hz action bandwidth).
  - **Standalone C++ / CUDA Engine**: **7.28 ms** (>1,000 Hz action bandwidth, pure CUDA/cuBLAS).
  - **Intel Iris Xe Hybrid**: β‰ˆ 106.6 ms on low-power laptop integrated graphics (`xpu` vision + CPU ODE).
- Includes optional auxiliary training decoders (`parvulus_auxiliary.safetensors`, 29.15M params) for auto-regressive visual self-rollout, motion change estimation, and self-reconstruction from noisy images.

---

## 2. Benchmark Performance

Evaluated across the 4 canonical LIBERO suites (N = 6,000 physical rollouts, 10 seeds, receding-horizon chunking H = 8, C = 8):

| Benchmark Suite | Canonical Tasks | Rollout Trials | Success Rate |
| :--- | :---: | :---: | :---: |
| **LIBERO-Spatial** | 10 | 1,500 | **92.67%** |
| **LIBERO-Object** | 10 | 1,500 | **91.13%** |
| **LIBERO-Goal** | 10 | 1,500 | **89.47%** |
| **LIBERO-10 (Long)** | 10 | 1,500 | **90.27%** |
| **Full Benchmark Average** | **40** | **6,000** | **90.88% Β± 0.37%** |

---

## 3. Repository Structure

```
.
β”œβ”€β”€ parvulus.safetensors            # Action-only core policy weights (9.98M params, ~39 MB FP32)
β”œβ”€β”€ parvulus_full.safetensors       # Full model weights: action policy + auxiliary heads (39.12M params, ~156 MB)
β”œβ”€β”€ parvulus_auxiliary.safetensors  # Auxiliary world model heads only (video, change, self-recon; 29.15M params)
β”œβ”€β”€ model.safetensors               # HuggingFace default entrypoint (alias of parvulus.safetensors)
β”‚
β”œβ”€β”€ config.json                     # Model hyperparameters and pinned T5 commit revision
β”œβ”€β”€ action_scale.json               # 7-DoF per-channel min/max normalization boundaries
β”œβ”€β”€ pyproject.toml                  # Standard package configuration
β”œβ”€β”€ requirements.txt                # Minimal Python dependencies
β”œβ”€β”€ prepare_runtime.py              # Setup script: builds C engine, generates binaries, prepares text encoder
β”‚
β”œβ”€β”€ parvulus_runtime.py             # High-performance Python runtime for standalone C engine
β”œβ”€β”€ infer.py                        # High-level ParvulusInference client (supports CUDA, C engine, XPU, Hybrid)
β”œβ”€β”€ parvulus.py                     # Complete PyTorch architecture (DualViewEncoder, CVMixer, FlowDecoder)
β”œβ”€β”€ text_encoder.py                 # Pinned T5 text tokenizer and projection head
β”œβ”€β”€ action_utils.py                 # Action normalization and denormalization helpers
β”‚
└── c_engine/                       # Dependency-free C++ / CUDA Standalone Engine
    β”œβ”€β”€ Makefile                    # Make-based build script
    β”œβ”€β”€ build.sh                    # Convenient build script
    β”œβ”€β”€ parvulus_engine.cu / .h     # Core C++/CUDA inference engine (136 Hz chunk rate)
    β”œβ”€β”€ vision_encoder.cu / .cuh    # Fused DualView vision encoder
    β”œβ”€β”€ parvulus_c_api.cpp / .h     # Standard C ABI exports for C/C++/ROS 2/Rust
    β”œβ”€β”€ parvulus_runtime_cli.cpp    # Standalone CLI binary entry point
    β”œβ”€β”€ canonical_tasks.json        # 40 canonical task instructions
    β”œβ”€β”€ export_from_safetensors.py  # Script generating model binary, scale, and embeddings
    └── export_text_encoder_onnx.py # Standalone converter for T5-small text encoder to ONNX
```

---

## 4. Quickstart: Python Inference

### Installation

```bash
pip install -r requirements.txt
```

### 1. Basic Action Chunk Prediction

```python
import numpy as np
from PIL import Image
from infer import ParvulusInference

# Initialize client (auto-detects CUDA / XPU / CPU; enables fast KV-cache by default)
infer = ParvulusInference.from_pretrained(
    ".",
    device="cuda",      # Options: 'cuda', 'cpu', 'xpu', 'hybrid', or 'auto'
    use_kv_cache=True,  # Enables sub-40ms inference
    num_ode_steps=10
)

# Prepare inputs: dual 256x256 RGB cameras + text prompt
agentview_img = np.array(Image.open("agentview.png"))  # [256, 256, 3] uint8
wristview_img = np.array(Image.open("wristview.png"))  # [256, 256, 3] uint8
task_prompt = "pick up the black bowl in the center and place it on the plate"

# Predict 8-step continuous action chunk [8, 7]
# (dx, dy, dz, drot_x, drot_y, drot_z, gripper)
action_chunk = infer.predict_chunk(agentview_img, wristview_img, task_description=task_prompt)
print(f"Predicted action chunk (shape {action_chunk.shape}):\n{action_chunk}")
```

### 2. Environment Stepping Loop (Receding Horizon)

```python
infer.reset(task_description="pick up the milk and place it in the basket")
obs = env.reset()
agentview_img = obs["agentview_image"]
wristview_img = obs["robot0_eye_in_hand_image"]

done = False
while not done:
    # In your simulation or real robot loop:
    action = infer.step(agentview_img, wristview_img)  # [7] float32 action
    obs, reward, done, info = env.step(action)
    agentview_img = obs["agentview_image"]
    wristview_img = obs["robot0_eye_in_hand_image"]
```

### 3. Continuous Overlapped Pipelined Execution (H = 8, C = 4)

To achieve zero-latency continuous robot motion, Parvulus supports overlapping chunk generation with robot motion execution:
1. Predict an 8-step chunk (A_0).
2. Execute the first 4 actions (A_0[0:4]).
3. At step 4, capture a fresh observation and trigger background inference for the next chunk (A_1).
4. Concurrently execute the remaining tail actions (A_0[4:8]) on the robot while the model predicts A_1.
5. Transition seamlessly into A_1 with zero robot idling.

```python
from concurrent.futures import ThreadPoolExecutor

executor = ThreadPoolExecutor(max_workers=1)
infer.reset(task_description="pick up the black bowl in the center and place it on the plate")

# 1. Reset environment and capture initial observation at t = 0
obs = env.reset()
agentview_img = obs["agentview_image"]
wristview_img = obs["robot0_eye_in_hand_image"]

# 2. Generate initial 8-step action chunk (covers t = 0..7)
current_chunk = infer.predict_chunk(agentview_img, wristview_img)

# 3. Execute the initial head (t = 0..3) before entering the pipelined loop
done = False
for a in current_chunk[0:4]:
    obs, reward, done, _ = env.step(a)
    if done:
        break

# Continuous overlapped loop: each cycle executes tail actions (4..7) while predicting the next chunk
while not done:
    # 4. Capture observation at step t = 4 (or 4k)
    agentview_img = obs["agentview_image"]
    wristview_img = obs["robot0_eye_in_hand_image"]

    # 5. Trigger background inference for next chunk (covers upcoming horizon)
    future_chunk = executor.submit(infer.predict_chunk, agentview_img, wristview_img)

    # 6. Overlap: Execute remaining 4 tail actions (4..7) while background inference runs
    for a in current_chunk[4:8]:
        obs, reward, done, _ = env.step(a)
        if done:
            break
    if done:
        break

    # 7. Retrieve newly generated chunk with zero robot idle time
    current_chunk = future_chunk.result()
```

### 4. World-Model Self-Rollout (Mental Simulation)

Parvulus provides optional auxiliary decoders (`parvulus_auxiliary.safetensors`) for predicting future frames without interacting with the physical environment:

```python
infer = ParvulusInference.from_pretrained(
    ".",
    load_auxiliary=True,
    device="cuda"
)

# Predict action chunk and imagined next observation
action_chunk, next_agent_img, next_wrist_img = infer.rollout_step(
    agentview_img, wristview_img, task_description="push the button"
)
```

---

## 5. Standalone Native C++ / CUDA Runtime (Vector-First, 7.3 ms / chunk)

For high-frequency physical robot deployments or low-overhead environments, Parvulus includes a standalone C++/CUDA runtime engine in `c_engine/` with **few dependencies**.

### Features
- Complete dual-camera visual encoding + 10-step Euler ODE flow matching loop runs in **<8 ms** (>1,000 Hz action bandwidth).
- Natively conditioned on 256-dim continuous embedding vectors (`task_vector`) to eliminate text tokenization latency in the robot control loop.
- Automatically maps arbitrary natural language instructions or perturbations into 256-dim vectors via ONNX Runtime (<4 ms) before passing to the engine.

### Preparing & Building the Runtime
```bash
# Automated preparation (builds C engine, prepares weights & ONNX text encoder)
python3 prepare_runtime.py

# Or build C++ engine directly:
cd c_engine && make -j$(nproc)
```
This produces:
- `libparvulus.so`: Shared C ABI library for Python ctypes, C, C++, ROS 2, or Rust.
- `parvulus_runtime`: Standalone CLI binary for headless benchmarks and batch action generation.
- `export_text_encoder_onnx.py`: Converter utility to export the 136 MB ONNX text encoder on demand (or download it from Hugging Face).

### Standalone CLI Benchmark
```bash
./c_engine/parvulus_runtime --benchmark 100
```

### Python Quickstart (`parvulus_runtime.py`)
```python
import numpy as np
from parvulus_runtime import ParvulusRuntime

# Initialize standalone C++/CUDA engine
runtime = ParvulusRuntime(device_id=0, num_euler_steps=10)

# 1. Condition with a 256-dim task vector:
runtime.reset(task_vector=my_256_dim_vector)

# Or condition with arbitrary text (automatic dynamic ONNX encoding):
# runtime.reset(task_description="grasp the black bowl and place it on the plate")

# 2. Step in real-time robot loop (streams only camera frames at 136 Hz):
action = runtime.step(agentview_img, wristview_img)  # [7] float32
```

For detailed C ABI signatures and non-Python integration instructions, see [`c_engine/README.md`](c_engine/README.md).

---

## 6. Intel Iris Xe / Arc Heterogeneous Hybrid Mode

For deployment on mobile workstations or edge devices with Intel Iris Xe or Arc integrated graphics:

```python
infer = ParvulusInference.from_pretrained(".", device="hybrid")
```

In `hybrid` mode:
1. Perception backbone (`DualViewEncoder`) executes on the iGPU (`torch.xpu`).
2. Only 16 pooled visual tokens (16 KB) transfer over PCIe to the host CPU.
3. The 10-step ODE Transformer loop runs on the high-clock CPU cores.

---

## 7. Model Details & Pinned Dependencies

- **Input Specifications**:
  - `agentview`: [256, 256, 3] RGB.
  - `wristview`: [256, 256, 3] RGB.
  - `instruction`: Natural language string.
- **Output Action Format**:
  - 8-step continuous trajectory A_t [8, 7] (dx, dy, dz, dΞΈx, dΞΈy, dΞΈz, gripper).
- **Pinned Text Model**:
  - `google-t5/t5-small` pinned to commit revision `df1b051c49625cf57a3d0d8d3863ed4d13564fe4`.

---

## Citation
Parvulus model weights and the associated inference code are released under a permissive license. If you find our work useful, please cite:

```bibtex
@inproceedings{
    sivas2026parvulus,
    title={Parvulus: A 10M-Scale World Action Model for Robot Manipulation},
    author={Abdullah Ali Sivas and Peter Cho-Ho Lam and Zirui Zhou and Yong Zhang},
    booktitle={CoRL 2026 Workshop on The Action Gap: Large Generative Models Learn Rich Knowledge},
    year={2026},
    url={https://openreview.net/forum?id=VcS0dE1Aoz}
}
```