Parvulus / README.md
aasivas's picture
Upload folder using huggingface_hub
f98e714 verified
|
Raw History Blame Contribute Delete
12 kB
---
language:
- en
license: apache-2.0
tags:
- robotics
- flow-matching
- world-models
- imitation-learning
- libero
- action-chunking
- pytorch
- safetensors
pipeline_tag: robotics
---
# PARVULUS: 10M-Scale World Action Model for Robot Manipulation
**PARVULUS** is an ultra-efficient 10M-parameter Flow Matching World Action Model (WAM) designed for continuous robot manipulation. Trained on the canonical [LIBERO](https://github.com/Lifelong-Robot-Learning/LIBERO) benchmark, Parvulus achieves a **90.88% average success rate across all 40 tasks** while running at real-time control frequencies (>100 Hz in C++/CUDA, sub-40ms with PyTorch KV-caching).
---
## 1. Highlights
- **9.98M parameters** (~20 MB in BF16/FP16, ~38 MB in FP32) for standalone robot control.
- **90.88% Β± 0.37%** average success across all 40 canonical LIBERO tasks (N = 6,000 simulation rollouts).
- Pinned `google-t5/t5-small` text representations with a trained 2-layer projection head.
- **Multiple Execution Modes**:
- **PyTorch KV-Cached**: β‰ˆ 38.8 ms (206 Hz action bandwidth).
- **Standalone C++ / CUDA Engine**: **7.28 ms** (>1,000 Hz action bandwidth, pure CUDA/cuBLAS).
- **Intel Iris Xe Hybrid**: β‰ˆ 106.6 ms on low-power laptop integrated graphics (`xpu` vision + CPU ODE).
- Includes optional auxiliary training decoders (`parvulus_auxiliary.safetensors`, 29.15M params) for auto-regressive visual self-rollout, motion change estimation, and self-reconstruction from noisy images.
---
## 2. Benchmark Performance
Evaluated across the 4 canonical LIBERO suites (N = 6,000 physical rollouts, 10 seeds, receding-horizon chunking H = 8, C = 8):
| Benchmark Suite | Canonical Tasks | Rollout Trials | Success Rate |
| :--- | :---: | :---: | :---: |
| **LIBERO-Spatial** | 10 | 1,500 | **92.67%** |
| **LIBERO-Object** | 10 | 1,500 | **91.13%** |
| **LIBERO-Goal** | 10 | 1,500 | **89.47%** |
| **LIBERO-10 (Long)** | 10 | 1,500 | **90.27%** |
| **Full Benchmark Average** | **40** | **6,000** | **90.88% Β± 0.37%** |
---
## 3. Repository Structure
```
.
β”œβ”€β”€ parvulus.safetensors # Action-only core policy weights (9.98M params, ~39 MB FP32)
β”œβ”€β”€ parvulus_full.safetensors # Full model weights: action policy + auxiliary heads (39.12M params, ~156 MB)
β”œβ”€β”€ parvulus_auxiliary.safetensors # Auxiliary world model heads only (video, change, self-recon; 29.15M params)
β”œβ”€β”€ model.safetensors # HuggingFace default entrypoint (alias of parvulus.safetensors)
β”‚
β”œβ”€β”€ config.json # Model hyperparameters and pinned T5 commit revision
β”œβ”€β”€ action_scale.json # 7-DoF per-channel min/max normalization boundaries
β”œβ”€β”€ pyproject.toml # Standard package configuration
β”œβ”€β”€ requirements.txt # Minimal Python dependencies
β”œβ”€β”€ prepare_runtime.py # Setup script: builds C engine, generates binaries, prepares text encoder
β”‚
β”œβ”€β”€ parvulus_runtime.py # High-performance Python runtime for standalone C engine
β”œβ”€β”€ infer.py # High-level ParvulusInference client (supports CUDA, C engine, XPU, Hybrid)
β”œβ”€β”€ parvulus.py # Complete PyTorch architecture (DualViewEncoder, CVMixer, FlowDecoder)
β”œβ”€β”€ text_encoder.py # Pinned T5 text tokenizer and projection head
β”œβ”€β”€ action_utils.py # Action normalization and denormalization helpers
β”‚
└── c_engine/ # Dependency-free C++ / CUDA Standalone Engine
β”œβ”€β”€ Makefile # Make-based build script
β”œβ”€β”€ build.sh # Convenient build script
β”œβ”€β”€ parvulus_engine.cu / .h # Core C++/CUDA inference engine (136 Hz chunk rate)
β”œβ”€β”€ vision_encoder.cu / .cuh # Fused DualView vision encoder
β”œβ”€β”€ parvulus_c_api.cpp / .h # Standard C ABI exports for C/C++/ROS 2/Rust
β”œβ”€β”€ parvulus_runtime_cli.cpp # Standalone CLI binary entry point
β”œβ”€β”€ canonical_tasks.json # 40 canonical task instructions
β”œβ”€β”€ export_from_safetensors.py # Script generating model binary, scale, and embeddings
└── export_text_encoder_onnx.py # Standalone converter for T5-small text encoder to ONNX
```
---
## 4. Quickstart: Python Inference
### Installation
```bash
pip install -r requirements.txt
```
### 1. Basic Action Chunk Prediction
```python
import numpy as np
from PIL import Image
from infer import ParvulusInference
# Initialize client (auto-detects CUDA / XPU / CPU; enables fast KV-cache by default)
infer = ParvulusInference.from_pretrained(
".",
device="cuda", # Options: 'cuda', 'cpu', 'xpu', 'hybrid', or 'auto'
use_kv_cache=True, # Enables sub-40ms inference
num_ode_steps=10
)
# Prepare inputs: dual 256x256 RGB cameras + text prompt
agentview_img = np.array(Image.open("agentview.png")) # [256, 256, 3] uint8
wristview_img = np.array(Image.open("wristview.png")) # [256, 256, 3] uint8
task_prompt = "pick up the black bowl in the center and place it on the plate"
# Predict 8-step continuous action chunk [8, 7]
# (dx, dy, dz, drot_x, drot_y, drot_z, gripper)
action_chunk = infer.predict_chunk(agentview_img, wristview_img, task_description=task_prompt)
print(f"Predicted action chunk (shape {action_chunk.shape}):\n{action_chunk}")
```
### 2. Environment Stepping Loop (Receding Horizon)
```python
infer.reset(task_description="pick up the milk and place it in the basket")
obs = env.reset()
agentview_img = obs["agentview_image"]
wristview_img = obs["robot0_eye_in_hand_image"]
done = False
while not done:
# In your simulation or real robot loop:
action = infer.step(agentview_img, wristview_img) # [7] float32 action
obs, reward, done, info = env.step(action)
agentview_img = obs["agentview_image"]
wristview_img = obs["robot0_eye_in_hand_image"]
```
### 3. Continuous Overlapped Pipelined Execution (H = 8, C = 4)
To achieve zero-latency continuous robot motion, Parvulus supports overlapping chunk generation with robot motion execution:
1. Predict an 8-step chunk (A_0).
2. Execute the first 4 actions (A_0[0:4]).
3. At step 4, capture a fresh observation and trigger background inference for the next chunk (A_1).
4. Concurrently execute the remaining tail actions (A_0[4:8]) on the robot while the model predicts A_1.
5. Transition seamlessly into A_1 with zero robot idling.
```python
from concurrent.futures import ThreadPoolExecutor
executor = ThreadPoolExecutor(max_workers=1)
infer.reset(task_description="pick up the black bowl in the center and place it on the plate")
# 1. Reset environment and capture initial observation at t = 0
obs = env.reset()
agentview_img = obs["agentview_image"]
wristview_img = obs["robot0_eye_in_hand_image"]
# 2. Generate initial 8-step action chunk (covers t = 0..7)
current_chunk = infer.predict_chunk(agentview_img, wristview_img)
# 3. Execute the initial head (t = 0..3) before entering the pipelined loop
done = False
for a in current_chunk[0:4]:
obs, reward, done, _ = env.step(a)
if done:
break
# Continuous overlapped loop: each cycle executes tail actions (4..7) while predicting the next chunk
while not done:
# 4. Capture observation at step t = 4 (or 4k)
agentview_img = obs["agentview_image"]
wristview_img = obs["robot0_eye_in_hand_image"]
# 5. Trigger background inference for next chunk (covers upcoming horizon)
future_chunk = executor.submit(infer.predict_chunk, agentview_img, wristview_img)
# 6. Overlap: Execute remaining 4 tail actions (4..7) while background inference runs
for a in current_chunk[4:8]:
obs, reward, done, _ = env.step(a)
if done:
break
if done:
break
# 7. Retrieve newly generated chunk with zero robot idle time
current_chunk = future_chunk.result()
```
### 4. World-Model Self-Rollout (Mental Simulation)
Parvulus provides optional auxiliary decoders (`parvulus_auxiliary.safetensors`) for predicting future frames without interacting with the physical environment:
```python
infer = ParvulusInference.from_pretrained(
".",
load_auxiliary=True,
device="cuda"
)
# Predict action chunk and imagined next observation
action_chunk, next_agent_img, next_wrist_img = infer.rollout_step(
agentview_img, wristview_img, task_description="push the button"
)
```
---
## 5. Standalone Native C++ / CUDA Runtime (Vector-First, 7.3 ms / chunk)
For high-frequency physical robot deployments or low-overhead environments, Parvulus includes a standalone C++/CUDA runtime engine in `c_engine/` with **few dependencies**.
### Features
- Complete dual-camera visual encoding + 10-step Euler ODE flow matching loop runs in **<8 ms** (>1,000 Hz action bandwidth).
- Natively conditioned on 256-dim continuous embedding vectors (`task_vector`) to eliminate text tokenization latency in the robot control loop.
- Automatically maps arbitrary natural language instructions or perturbations into 256-dim vectors via ONNX Runtime (<4 ms) before passing to the engine.
### Preparing & Building the Runtime
```bash
# Automated preparation (builds C engine, prepares weights & ONNX text encoder)
python3 prepare_runtime.py
# Or build C++ engine directly:
cd c_engine && make -j$(nproc)
```
This produces:
- `libparvulus.so`: Shared C ABI library for Python ctypes, C, C++, ROS 2, or Rust.
- `parvulus_runtime`: Standalone CLI binary for headless benchmarks and batch action generation.
- `export_text_encoder_onnx.py`: Converter utility to export the 136 MB ONNX text encoder on demand (or download it from Hugging Face).
### Standalone CLI Benchmark
```bash
./c_engine/parvulus_runtime --benchmark 100
```
### Python Quickstart (`parvulus_runtime.py`)
```python
import numpy as np
from parvulus_runtime import ParvulusRuntime
# Initialize standalone C++/CUDA engine
runtime = ParvulusRuntime(device_id=0, num_euler_steps=10)
# 1. Condition with a 256-dim task vector:
runtime.reset(task_vector=my_256_dim_vector)
# Or condition with arbitrary text (automatic dynamic ONNX encoding):
# runtime.reset(task_description="grasp the black bowl and place it on the plate")
# 2. Step in real-time robot loop (streams only camera frames at 136 Hz):
action = runtime.step(agentview_img, wristview_img) # [7] float32
```
For detailed C ABI signatures and non-Python integration instructions, see [`c_engine/README.md`](c_engine/README.md).
---
## 6. Intel Iris Xe / Arc Heterogeneous Hybrid Mode
For deployment on mobile workstations or edge devices with Intel Iris Xe or Arc integrated graphics:
```python
infer = ParvulusInference.from_pretrained(".", device="hybrid")
```
In `hybrid` mode:
1. Perception backbone (`DualViewEncoder`) executes on the iGPU (`torch.xpu`).
2. Only 16 pooled visual tokens (16 KB) transfer over PCIe to the host CPU.
3. The 10-step ODE Transformer loop runs on the high-clock CPU cores.
---
## 7. Model Details & Pinned Dependencies
- **Input Specifications**:
- `agentview`: [256, 256, 3] RGB.
- `wristview`: [256, 256, 3] RGB.
- `instruction`: Natural language string.
- **Output Action Format**:
- 8-step continuous trajectory A_t [8, 7] (dx, dy, dz, dΞΈx, dΞΈy, dΞΈz, gripper).
- **Pinned Text Model**:
- `google-t5/t5-small` pinned to commit revision `df1b051c49625cf57a3d0d8d3863ed4d13564fe4`.
---
## Citation
Parvulus model weights and the associated inference code are released under a permissive license. If you find our work useful, please cite:
```bibtex
@inproceedings{
sivas2026parvulus,
title={Parvulus: A 10M-Scale World Action Model for Robot Manipulation},
author={Abdullah Ali Sivas and Peter Cho-Ho Lam and Zirui Zhou and Yong Zhang},
booktitle={CoRL 2026 Workshop on The Action Gap: Large Generative Models Learn Rich Knowledge},
year={2026},
url={https://openreview.net/forum?id=VcS0dE1Aoz}
}
```