--- language: - en license: apache-2.0 tags: - robotics - flow-matching - world-models - imitation-learning - libero - action-chunking - pytorch - safetensors pipeline_tag: robotics --- # PARVULUS: 10M-Scale World Action Model for Robot Manipulation **PARVULUS** is an ultra-efficient 10M-parameter Flow Matching World Action Model (WAM) designed for continuous robot manipulation. Trained on the canonical [LIBERO](https://github.com/Lifelong-Robot-Learning/LIBERO) benchmark, Parvulus achieves a **90.88% average success rate across all 40 tasks** while running at real-time control frequencies (>100 Hz in C++/CUDA, sub-40ms with PyTorch KV-caching). --- ## 1. Highlights - **9.98M parameters** (~20 MB in BF16/FP16, ~38 MB in FP32) for standalone robot control. - **90.88% ± 0.37%** average success across all 40 canonical LIBERO tasks (N = 6,000 simulation rollouts). - Pinned `google-t5/t5-small` text representations with a trained 2-layer projection head. - **Multiple Execution Modes**: - **PyTorch KV-Cached**: ≈ 38.8 ms (206 Hz action bandwidth). - **Standalone C++ / CUDA Engine**: **7.28 ms** (>1,000 Hz action bandwidth, pure CUDA/cuBLAS). - **Intel Iris Xe Hybrid**: ≈ 106.6 ms on low-power laptop integrated graphics (`xpu` vision + CPU ODE). - Includes optional auxiliary training decoders (`parvulus_auxiliary.safetensors`, 29.15M params) for auto-regressive visual self-rollout, motion change estimation, and self-reconstruction from noisy images. --- ## 2. Benchmark Performance Evaluated across the 4 canonical LIBERO suites (N = 6,000 physical rollouts, 10 seeds, receding-horizon chunking H = 8, C = 8): | Benchmark Suite | Canonical Tasks | Rollout Trials | Success Rate | | :--- | :---: | :---: | :---: | | **LIBERO-Spatial** | 10 | 1,500 | **92.67%** | | **LIBERO-Object** | 10 | 1,500 | **91.13%** | | **LIBERO-Goal** | 10 | 1,500 | **89.47%** | | **LIBERO-10 (Long)** | 10 | 1,500 | **90.27%** | | **Full Benchmark Average** | **40** | **6,000** | **90.88% ± 0.37%** | --- ## 3. Repository Structure ``` . ├── parvulus.safetensors # Action-only core policy weights (9.98M params, ~39 MB FP32) ├── parvulus_full.safetensors # Full model weights: action policy + auxiliary heads (39.12M params, ~156 MB) ├── parvulus_auxiliary.safetensors # Auxiliary world model heads only (video, change, self-recon; 29.15M params) ├── model.safetensors # HuggingFace default entrypoint (alias of parvulus.safetensors) │ ├── config.json # Model hyperparameters and pinned T5 commit revision ├── action_scale.json # 7-DoF per-channel min/max normalization boundaries ├── pyproject.toml # Standard package configuration ├── requirements.txt # Minimal Python dependencies ├── prepare_runtime.py # Setup script: builds C engine, generates binaries, prepares text encoder │ ├── parvulus_runtime.py # High-performance Python runtime for standalone C engine ├── infer.py # High-level ParvulusInference client (supports CUDA, C engine, XPU, Hybrid) ├── parvulus.py # Complete PyTorch architecture (DualViewEncoder, CVMixer, FlowDecoder) ├── text_encoder.py # Pinned T5 text tokenizer and projection head ├── action_utils.py # Action normalization and denormalization helpers │ └── c_engine/ # Dependency-free C++ / CUDA Standalone Engine ├── Makefile # Make-based build script ├── build.sh # Convenient build script ├── parvulus_engine.cu / .h # Core C++/CUDA inference engine (136 Hz chunk rate) ├── vision_encoder.cu / .cuh # Fused DualView vision encoder ├── parvulus_c_api.cpp / .h # Standard C ABI exports for C/C++/ROS 2/Rust ├── parvulus_runtime_cli.cpp # Standalone CLI binary entry point ├── canonical_tasks.json # 40 canonical task instructions ├── export_from_safetensors.py # Script generating model binary, scale, and embeddings └── export_text_encoder_onnx.py # Standalone converter for T5-small text encoder to ONNX ``` --- ## 4. Quickstart: Python Inference ### Installation ```bash pip install -r requirements.txt ``` ### 1. Basic Action Chunk Prediction ```python import numpy as np from PIL import Image from infer import ParvulusInference # Initialize client (auto-detects CUDA / XPU / CPU; enables fast KV-cache by default) infer = ParvulusInference.from_pretrained( ".", device="cuda", # Options: 'cuda', 'cpu', 'xpu', 'hybrid', or 'auto' use_kv_cache=True, # Enables sub-40ms inference num_ode_steps=10 ) # Prepare inputs: dual 256x256 RGB cameras + text prompt agentview_img = np.array(Image.open("agentview.png")) # [256, 256, 3] uint8 wristview_img = np.array(Image.open("wristview.png")) # [256, 256, 3] uint8 task_prompt = "pick up the black bowl in the center and place it on the plate" # Predict 8-step continuous action chunk [8, 7] # (dx, dy, dz, drot_x, drot_y, drot_z, gripper) action_chunk = infer.predict_chunk(agentview_img, wristview_img, task_description=task_prompt) print(f"Predicted action chunk (shape {action_chunk.shape}):\n{action_chunk}") ``` ### 2. Environment Stepping Loop (Receding Horizon) ```python infer.reset(task_description="pick up the milk and place it in the basket") obs = env.reset() agentview_img = obs["agentview_image"] wristview_img = obs["robot0_eye_in_hand_image"] done = False while not done: # In your simulation or real robot loop: action = infer.step(agentview_img, wristview_img) # [7] float32 action obs, reward, done, info = env.step(action) agentview_img = obs["agentview_image"] wristview_img = obs["robot0_eye_in_hand_image"] ``` ### 3. Continuous Overlapped Pipelined Execution (H = 8, C = 4) To achieve zero-latency continuous robot motion, Parvulus supports overlapping chunk generation with robot motion execution: 1. Predict an 8-step chunk (A_0). 2. Execute the first 4 actions (A_0[0:4]). 3. At step 4, capture a fresh observation and trigger background inference for the next chunk (A_1). 4. Concurrently execute the remaining tail actions (A_0[4:8]) on the robot while the model predicts A_1. 5. Transition seamlessly into A_1 with zero robot idling. ```python from concurrent.futures import ThreadPoolExecutor executor = ThreadPoolExecutor(max_workers=1) infer.reset(task_description="pick up the black bowl in the center and place it on the plate") # 1. Reset environment and capture initial observation at t = 0 obs = env.reset() agentview_img = obs["agentview_image"] wristview_img = obs["robot0_eye_in_hand_image"] # 2. Generate initial 8-step action chunk (covers t = 0..7) current_chunk = infer.predict_chunk(agentview_img, wristview_img) # 3. Execute the initial head (t = 0..3) before entering the pipelined loop done = False for a in current_chunk[0:4]: obs, reward, done, _ = env.step(a) if done: break # Continuous overlapped loop: each cycle executes tail actions (4..7) while predicting the next chunk while not done: # 4. Capture observation at step t = 4 (or 4k) agentview_img = obs["agentview_image"] wristview_img = obs["robot0_eye_in_hand_image"] # 5. Trigger background inference for next chunk (covers upcoming horizon) future_chunk = executor.submit(infer.predict_chunk, agentview_img, wristview_img) # 6. Overlap: Execute remaining 4 tail actions (4..7) while background inference runs for a in current_chunk[4:8]: obs, reward, done, _ = env.step(a) if done: break if done: break # 7. Retrieve newly generated chunk with zero robot idle time current_chunk = future_chunk.result() ``` ### 4. World-Model Self-Rollout (Mental Simulation) Parvulus provides optional auxiliary decoders (`parvulus_auxiliary.safetensors`) for predicting future frames without interacting with the physical environment: ```python infer = ParvulusInference.from_pretrained( ".", load_auxiliary=True, device="cuda" ) # Predict action chunk and imagined next observation action_chunk, next_agent_img, next_wrist_img = infer.rollout_step( agentview_img, wristview_img, task_description="push the button" ) ``` --- ## 5. Standalone Native C++ / CUDA Runtime (Vector-First, 7.3 ms / chunk) For high-frequency physical robot deployments or low-overhead environments, Parvulus includes a standalone C++/CUDA runtime engine in `c_engine/` with **few dependencies**. ### Features - Complete dual-camera visual encoding + 10-step Euler ODE flow matching loop runs in **<8 ms** (>1,000 Hz action bandwidth). - Natively conditioned on 256-dim continuous embedding vectors (`task_vector`) to eliminate text tokenization latency in the robot control loop. - Automatically maps arbitrary natural language instructions or perturbations into 256-dim vectors via ONNX Runtime (<4 ms) before passing to the engine. ### Preparing & Building the Runtime ```bash # Automated preparation (builds C engine, prepares weights & ONNX text encoder) python3 prepare_runtime.py # Or build C++ engine directly: cd c_engine && make -j$(nproc) ``` This produces: - `libparvulus.so`: Shared C ABI library for Python ctypes, C, C++, ROS 2, or Rust. - `parvulus_runtime`: Standalone CLI binary for headless benchmarks and batch action generation. - `export_text_encoder_onnx.py`: Converter utility to export the 136 MB ONNX text encoder on demand (or download it from Hugging Face). ### Standalone CLI Benchmark ```bash ./c_engine/parvulus_runtime --benchmark 100 ``` ### Python Quickstart (`parvulus_runtime.py`) ```python import numpy as np from parvulus_runtime import ParvulusRuntime # Initialize standalone C++/CUDA engine runtime = ParvulusRuntime(device_id=0, num_euler_steps=10) # 1. Condition with a 256-dim task vector: runtime.reset(task_vector=my_256_dim_vector) # Or condition with arbitrary text (automatic dynamic ONNX encoding): # runtime.reset(task_description="grasp the black bowl and place it on the plate") # 2. Step in real-time robot loop (streams only camera frames at 136 Hz): action = runtime.step(agentview_img, wristview_img) # [7] float32 ``` For detailed C ABI signatures and non-Python integration instructions, see [`c_engine/README.md`](c_engine/README.md). --- ## 6. Intel Iris Xe / Arc Heterogeneous Hybrid Mode For deployment on mobile workstations or edge devices with Intel Iris Xe or Arc integrated graphics: ```python infer = ParvulusInference.from_pretrained(".", device="hybrid") ``` In `hybrid` mode: 1. Perception backbone (`DualViewEncoder`) executes on the iGPU (`torch.xpu`). 2. Only 16 pooled visual tokens (16 KB) transfer over PCIe to the host CPU. 3. The 10-step ODE Transformer loop runs on the high-clock CPU cores. --- ## 7. Model Details & Pinned Dependencies - **Input Specifications**: - `agentview`: [256, 256, 3] RGB. - `wristview`: [256, 256, 3] RGB. - `instruction`: Natural language string. - **Output Action Format**: - 8-step continuous trajectory A_t [8, 7] (dx, dy, dz, dθx, dθy, dθz, gripper). - **Pinned Text Model**: - `google-t5/t5-small` pinned to commit revision `df1b051c49625cf57a3d0d8d3863ed4d13564fe4`. --- ## Citation Parvulus model weights and the associated inference code are released under a permissive license. If you find our work useful, please cite: ```bibtex @inproceedings{ sivas2026parvulus, title={Parvulus: A 10M-Scale World Action Model for Robot Manipulation}, author={Abdullah Ali Sivas and Peter Cho-Ho Lam and Zirui Zhou and Yong Zhang}, booktitle={CoRL 2026 Workshop on The Action Gap: Large Generative Models Learn Rich Knowledge}, year={2026}, url={https://openreview.net/forum?id=VcS0dE1Aoz} } ```