Download README.md from aasivas/Parvulus: direct link, hf CLI and curl.
- Browser
- Download file 12 kB
-
https://huggingface.co/aasivas/Parvulus/resolve/main/README.md
- Command line
-
hf download hf://aasivas/Parvulus/README.md
-
curl -L -o README.md https://huggingface.co/aasivas/Parvulus/resolve/main/README.md
language:
- en
license: apache-2.0
tags:
- robotics
- flow-matching
- world-models
- imitation-learning
- libero
- action-chunking
- pytorch
- safetensors
pipeline_tag: robotics
PARVULUS: 10M-Scale World Action Model for Robot Manipulation
PARVULUS is an ultra-efficient 10M-parameter Flow Matching World Action Model (WAM) designed for continuous robot manipulation. Trained on the canonical LIBERO benchmark, Parvulus achieves a 90.88% average success rate across all 40 tasks while running at real-time control frequencies (>100 Hz in C++/CUDA, sub-40ms with PyTorch KV-caching).
1. Highlights
- 9.98M parameters (~20 MB in BF16/FP16, ~38 MB in FP32) for standalone robot control.
- 90.88% Β± 0.37% average success across all 40 canonical LIBERO tasks (N = 6,000 simulation rollouts).
- Pinned
google-t5/t5-smalltext representations with a trained 2-layer projection head. - Multiple Execution Modes:
- PyTorch KV-Cached: β 38.8 ms (206 Hz action bandwidth).
- Standalone C++ / CUDA Engine: 7.28 ms (>1,000 Hz action bandwidth, pure CUDA/cuBLAS).
- Intel Iris Xe Hybrid: β 106.6 ms on low-power laptop integrated graphics (
xpuvision + CPU ODE).
- Includes optional auxiliary training decoders (
parvulus_auxiliary.safetensors, 29.15M params) for auto-regressive visual self-rollout, motion change estimation, and self-reconstruction from noisy images.
2. Benchmark Performance
Evaluated across the 4 canonical LIBERO suites (N = 6,000 physical rollouts, 10 seeds, receding-horizon chunking H = 8, C = 8):
| Benchmark Suite | Canonical Tasks | Rollout Trials | Success Rate |
|---|---|---|---|
| LIBERO-Spatial | 10 | 1,500 | 92.67% |
| LIBERO-Object | 10 | 1,500 | 91.13% |
| LIBERO-Goal | 10 | 1,500 | 89.47% |
| LIBERO-10 (Long) | 10 | 1,500 | 90.27% |
| Full Benchmark Average | 40 | 6,000 | 90.88% Β± 0.37% |
3. Repository Structure
.
βββ parvulus.safetensors # Action-only core policy weights (9.98M params, ~39 MB FP32)
βββ parvulus_full.safetensors # Full model weights: action policy + auxiliary heads (39.12M params, ~156 MB)
βββ parvulus_auxiliary.safetensors # Auxiliary world model heads only (video, change, self-recon; 29.15M params)
βββ model.safetensors # HuggingFace default entrypoint (alias of parvulus.safetensors)
β
βββ config.json # Model hyperparameters and pinned T5 commit revision
βββ action_scale.json # 7-DoF per-channel min/max normalization boundaries
βββ pyproject.toml # Standard package configuration
βββ requirements.txt # Minimal Python dependencies
βββ prepare_runtime.py # Setup script: builds C engine, generates binaries, prepares text encoder
β
βββ parvulus_runtime.py # High-performance Python runtime for standalone C engine
βββ infer.py # High-level ParvulusInference client (supports CUDA, C engine, XPU, Hybrid)
βββ parvulus.py # Complete PyTorch architecture (DualViewEncoder, CVMixer, FlowDecoder)
βββ text_encoder.py # Pinned T5 text tokenizer and projection head
βββ action_utils.py # Action normalization and denormalization helpers
β
βββ c_engine/ # Dependency-free C++ / CUDA Standalone Engine
βββ Makefile # Make-based build script
βββ build.sh # Convenient build script
βββ parvulus_engine.cu / .h # Core C++/CUDA inference engine (136 Hz chunk rate)
βββ vision_encoder.cu / .cuh # Fused DualView vision encoder
βββ parvulus_c_api.cpp / .h # Standard C ABI exports for C/C++/ROS 2/Rust
βββ parvulus_runtime_cli.cpp # Standalone CLI binary entry point
βββ canonical_tasks.json # 40 canonical task instructions
βββ export_from_safetensors.py # Script generating model binary, scale, and embeddings
βββ export_text_encoder_onnx.py # Standalone converter for T5-small text encoder to ONNX
4. Quickstart: Python Inference
Installation
pip install -r requirements.txt
1. Basic Action Chunk Prediction
import numpy as np
from PIL import Image
from infer import ParvulusInference
# Initialize client (auto-detects CUDA / XPU / CPU; enables fast KV-cache by default)
infer = ParvulusInference.from_pretrained(
".",
device="cuda", # Options: 'cuda', 'cpu', 'xpu', 'hybrid', or 'auto'
use_kv_cache=True, # Enables sub-40ms inference
num_ode_steps=10
)
# Prepare inputs: dual 256x256 RGB cameras + text prompt
agentview_img = np.array(Image.open("agentview.png")) # [256, 256, 3] uint8
wristview_img = np.array(Image.open("wristview.png")) # [256, 256, 3] uint8
task_prompt = "pick up the black bowl in the center and place it on the plate"
# Predict 8-step continuous action chunk [8, 7]
# (dx, dy, dz, drot_x, drot_y, drot_z, gripper)
action_chunk = infer.predict_chunk(agentview_img, wristview_img, task_description=task_prompt)
print(f"Predicted action chunk (shape {action_chunk.shape}):\n{action_chunk}")
2. Environment Stepping Loop (Receding Horizon)
infer.reset(task_description="pick up the milk and place it in the basket")
obs = env.reset()
agentview_img = obs["agentview_image"]
wristview_img = obs["robot0_eye_in_hand_image"]
done = False
while not done:
# In your simulation or real robot loop:
action = infer.step(agentview_img, wristview_img) # [7] float32 action
obs, reward, done, info = env.step(action)
agentview_img = obs["agentview_image"]
wristview_img = obs["robot0_eye_in_hand_image"]
3. Continuous Overlapped Pipelined Execution (H = 8, C = 4)
To achieve zero-latency continuous robot motion, Parvulus supports overlapping chunk generation with robot motion execution:
- Predict an 8-step chunk (A_0).
- Execute the first 4 actions (A_0[0:4]).
- At step 4, capture a fresh observation and trigger background inference for the next chunk (A_1).
- Concurrently execute the remaining tail actions (A_0[4:8]) on the robot while the model predicts A_1.
- Transition seamlessly into A_1 with zero robot idling.
from concurrent.futures import ThreadPoolExecutor
executor = ThreadPoolExecutor(max_workers=1)
infer.reset(task_description="pick up the black bowl in the center and place it on the plate")
# 1. Reset environment and capture initial observation at t = 0
obs = env.reset()
agentview_img = obs["agentview_image"]
wristview_img = obs["robot0_eye_in_hand_image"]
# 2. Generate initial 8-step action chunk (covers t = 0..7)
current_chunk = infer.predict_chunk(agentview_img, wristview_img)
# 3. Execute the initial head (t = 0..3) before entering the pipelined loop
done = False
for a in current_chunk[0:4]:
obs, reward, done, _ = env.step(a)
if done:
break
# Continuous overlapped loop: each cycle executes tail actions (4..7) while predicting the next chunk
while not done:
# 4. Capture observation at step t = 4 (or 4k)
agentview_img = obs["agentview_image"]
wristview_img = obs["robot0_eye_in_hand_image"]
# 5. Trigger background inference for next chunk (covers upcoming horizon)
future_chunk = executor.submit(infer.predict_chunk, agentview_img, wristview_img)
# 6. Overlap: Execute remaining 4 tail actions (4..7) while background inference runs
for a in current_chunk[4:8]:
obs, reward, done, _ = env.step(a)
if done:
break
if done:
break
# 7. Retrieve newly generated chunk with zero robot idle time
current_chunk = future_chunk.result()
4. World-Model Self-Rollout (Mental Simulation)
Parvulus provides optional auxiliary decoders (parvulus_auxiliary.safetensors) for predicting future frames without interacting with the physical environment:
infer = ParvulusInference.from_pretrained(
".",
load_auxiliary=True,
device="cuda"
)
# Predict action chunk and imagined next observation
action_chunk, next_agent_img, next_wrist_img = infer.rollout_step(
agentview_img, wristview_img, task_description="push the button"
)
5. Standalone Native C++ / CUDA Runtime (Vector-First, 7.3 ms / chunk)
For high-frequency physical robot deployments or low-overhead environments, Parvulus includes a standalone C++/CUDA runtime engine in c_engine/ with few dependencies.
Features
- Complete dual-camera visual encoding + 10-step Euler ODE flow matching loop runs in **<8 ms** (>1,000 Hz action bandwidth).
- Natively conditioned on 256-dim continuous embedding vectors (
task_vector) to eliminate text tokenization latency in the robot control loop. - Automatically maps arbitrary natural language instructions or perturbations into 256-dim vectors via ONNX Runtime (<4 ms) before passing to the engine.
Preparing & Building the Runtime
# Automated preparation (builds C engine, prepares weights & ONNX text encoder)
python3 prepare_runtime.py
# Or build C++ engine directly:
cd c_engine && make -j$(nproc)
This produces:
libparvulus.so: Shared C ABI library for Python ctypes, C, C++, ROS 2, or Rust.parvulus_runtime: Standalone CLI binary for headless benchmarks and batch action generation.export_text_encoder_onnx.py: Converter utility to export the 136 MB ONNX text encoder on demand (or download it from Hugging Face).
Standalone CLI Benchmark
./c_engine/parvulus_runtime --benchmark 100
Python Quickstart (parvulus_runtime.py)
import numpy as np
from parvulus_runtime import ParvulusRuntime
# Initialize standalone C++/CUDA engine
runtime = ParvulusRuntime(device_id=0, num_euler_steps=10)
# 1. Condition with a 256-dim task vector:
runtime.reset(task_vector=my_256_dim_vector)
# Or condition with arbitrary text (automatic dynamic ONNX encoding):
# runtime.reset(task_description="grasp the black bowl and place it on the plate")
# 2. Step in real-time robot loop (streams only camera frames at 136 Hz):
action = runtime.step(agentview_img, wristview_img) # [7] float32
For detailed C ABI signatures and non-Python integration instructions, see c_engine/README.md.
6. Intel Iris Xe / Arc Heterogeneous Hybrid Mode
For deployment on mobile workstations or edge devices with Intel Iris Xe or Arc integrated graphics:
infer = ParvulusInference.from_pretrained(".", device="hybrid")
In hybrid mode:
- Perception backbone (
DualViewEncoder) executes on the iGPU (torch.xpu). - Only 16 pooled visual tokens (16 KB) transfer over PCIe to the host CPU.
- The 10-step ODE Transformer loop runs on the high-clock CPU cores.
7. Model Details & Pinned Dependencies
- Input Specifications:
agentview: [256, 256, 3] RGB.wristview: [256, 256, 3] RGB.instruction: Natural language string.
- Output Action Format:
- 8-step continuous trajectory A_t [8, 7] (dx, dy, dz, dΞΈx, dΞΈy, dΞΈz, gripper).
- Pinned Text Model:
google-t5/t5-smallpinned to commit revisiondf1b051c49625cf57a3d0d8d3863ed4d13564fe4.
Citation
Parvulus model weights and the associated inference code are released under a permissive license. If you find our work useful, please cite:
@inproceedings{
sivas2026parvulus,
title={Parvulus: A 10M-Scale World Action Model for Robot Manipulation},
author={Abdullah Ali Sivas and Peter Cho-Ho Lam and Zirui Zhou and Yong Zhang},
booktitle={CoRL 2026 Workshop on The Action Gap: Large Generative Models Learn Rich Knowledge},
year={2026},
url={https://openreview.net/forum?id=VcS0dE1Aoz}
}