|
Download README.md from aasivas/Parvulus: direct link, hf CLI and curl.
- Browser
- Download file 12 kB
-
https://huggingface.co/aasivas/Parvulus/resolve/main/README.md
- Command line
-
hf download hf://aasivas/Parvulus/README.md
-
curl -L -o README.md https://huggingface.co/aasivas/Parvulus/resolve/main/README.md
12 kB
| language: | |
| - en | |
| license: apache-2.0 | |
| tags: | |
| - robotics | |
| - flow-matching | |
| - world-models | |
| - imitation-learning | |
| - libero | |
| - action-chunking | |
| - pytorch | |
| - safetensors | |
| pipeline_tag: robotics | |
| # PARVULUS: 10M-Scale World Action Model for Robot Manipulation | |
| **PARVULUS** is an ultra-efficient 10M-parameter Flow Matching World Action Model (WAM) designed for continuous robot manipulation. Trained on the canonical [LIBERO](https://github.com/Lifelong-Robot-Learning/LIBERO) benchmark, Parvulus achieves a **90.88% average success rate across all 40 tasks** while running at real-time control frequencies (>100 Hz in C++/CUDA, sub-40ms with PyTorch KV-caching). | |
| --- | |
| ## 1. Highlights | |
| - **9.98M parameters** (~20 MB in BF16/FP16, ~38 MB in FP32) for standalone robot control. | |
| - **90.88% Β± 0.37%** average success across all 40 canonical LIBERO tasks (N = 6,000 simulation rollouts). | |
| - Pinned `google-t5/t5-small` text representations with a trained 2-layer projection head. | |
| - **Multiple Execution Modes**: | |
| - **PyTorch KV-Cached**: β 38.8 ms (206 Hz action bandwidth). | |
| - **Standalone C++ / CUDA Engine**: **7.28 ms** (>1,000 Hz action bandwidth, pure CUDA/cuBLAS). | |
| - **Intel Iris Xe Hybrid**: β 106.6 ms on low-power laptop integrated graphics (`xpu` vision + CPU ODE). | |
| - Includes optional auxiliary training decoders (`parvulus_auxiliary.safetensors`, 29.15M params) for auto-regressive visual self-rollout, motion change estimation, and self-reconstruction from noisy images. | |
| --- | |
| ## 2. Benchmark Performance | |
| Evaluated across the 4 canonical LIBERO suites (N = 6,000 physical rollouts, 10 seeds, receding-horizon chunking H = 8, C = 8): | |
| | Benchmark Suite | Canonical Tasks | Rollout Trials | Success Rate | | |
| | :--- | :---: | :---: | :---: | | |
| | **LIBERO-Spatial** | 10 | 1,500 | **92.67%** | | |
| | **LIBERO-Object** | 10 | 1,500 | **91.13%** | | |
| | **LIBERO-Goal** | 10 | 1,500 | **89.47%** | | |
| | **LIBERO-10 (Long)** | 10 | 1,500 | **90.27%** | | |
| | **Full Benchmark Average** | **40** | **6,000** | **90.88% Β± 0.37%** | | |
| --- | |
| ## 3. Repository Structure | |
| ``` | |
| . | |
| βββ parvulus.safetensors # Action-only core policy weights (9.98M params, ~39 MB FP32) | |
| βββ parvulus_full.safetensors # Full model weights: action policy + auxiliary heads (39.12M params, ~156 MB) | |
| βββ parvulus_auxiliary.safetensors # Auxiliary world model heads only (video, change, self-recon; 29.15M params) | |
| βββ model.safetensors # HuggingFace default entrypoint (alias of parvulus.safetensors) | |
| β | |
| βββ config.json # Model hyperparameters and pinned T5 commit revision | |
| βββ action_scale.json # 7-DoF per-channel min/max normalization boundaries | |
| βββ pyproject.toml # Standard package configuration | |
| βββ requirements.txt # Minimal Python dependencies | |
| βββ prepare_runtime.py # Setup script: builds C engine, generates binaries, prepares text encoder | |
| β | |
| βββ parvulus_runtime.py # High-performance Python runtime for standalone C engine | |
| βββ infer.py # High-level ParvulusInference client (supports CUDA, C engine, XPU, Hybrid) | |
| βββ parvulus.py # Complete PyTorch architecture (DualViewEncoder, CVMixer, FlowDecoder) | |
| βββ text_encoder.py # Pinned T5 text tokenizer and projection head | |
| βββ action_utils.py # Action normalization and denormalization helpers | |
| β | |
| βββ c_engine/ # Dependency-free C++ / CUDA Standalone Engine | |
| βββ Makefile # Make-based build script | |
| βββ build.sh # Convenient build script | |
| βββ parvulus_engine.cu / .h # Core C++/CUDA inference engine (136 Hz chunk rate) | |
| βββ vision_encoder.cu / .cuh # Fused DualView vision encoder | |
| βββ parvulus_c_api.cpp / .h # Standard C ABI exports for C/C++/ROS 2/Rust | |
| βββ parvulus_runtime_cli.cpp # Standalone CLI binary entry point | |
| βββ canonical_tasks.json # 40 canonical task instructions | |
| βββ export_from_safetensors.py # Script generating model binary, scale, and embeddings | |
| βββ export_text_encoder_onnx.py # Standalone converter for T5-small text encoder to ONNX | |
| ``` | |
| --- | |
| ## 4. Quickstart: Python Inference | |
| ### Installation | |
| ```bash | |
| pip install -r requirements.txt | |
| ``` | |
| ### 1. Basic Action Chunk Prediction | |
| ```python | |
| import numpy as np | |
| from PIL import Image | |
| from infer import ParvulusInference | |
| # Initialize client (auto-detects CUDA / XPU / CPU; enables fast KV-cache by default) | |
| infer = ParvulusInference.from_pretrained( | |
| ".", | |
| device="cuda", # Options: 'cuda', 'cpu', 'xpu', 'hybrid', or 'auto' | |
| use_kv_cache=True, # Enables sub-40ms inference | |
| num_ode_steps=10 | |
| ) | |
| # Prepare inputs: dual 256x256 RGB cameras + text prompt | |
| agentview_img = np.array(Image.open("agentview.png")) # [256, 256, 3] uint8 | |
| wristview_img = np.array(Image.open("wristview.png")) # [256, 256, 3] uint8 | |
| task_prompt = "pick up the black bowl in the center and place it on the plate" | |
| # Predict 8-step continuous action chunk [8, 7] | |
| # (dx, dy, dz, drot_x, drot_y, drot_z, gripper) | |
| action_chunk = infer.predict_chunk(agentview_img, wristview_img, task_description=task_prompt) | |
| print(f"Predicted action chunk (shape {action_chunk.shape}):\n{action_chunk}") | |
| ``` | |
| ### 2. Environment Stepping Loop (Receding Horizon) | |
| ```python | |
| infer.reset(task_description="pick up the milk and place it in the basket") | |
| obs = env.reset() | |
| agentview_img = obs["agentview_image"] | |
| wristview_img = obs["robot0_eye_in_hand_image"] | |
| done = False | |
| while not done: | |
| # In your simulation or real robot loop: | |
| action = infer.step(agentview_img, wristview_img) # [7] float32 action | |
| obs, reward, done, info = env.step(action) | |
| agentview_img = obs["agentview_image"] | |
| wristview_img = obs["robot0_eye_in_hand_image"] | |
| ``` | |
| ### 3. Continuous Overlapped Pipelined Execution (H = 8, C = 4) | |
| To achieve zero-latency continuous robot motion, Parvulus supports overlapping chunk generation with robot motion execution: | |
| 1. Predict an 8-step chunk (A_0). | |
| 2. Execute the first 4 actions (A_0[0:4]). | |
| 3. At step 4, capture a fresh observation and trigger background inference for the next chunk (A_1). | |
| 4. Concurrently execute the remaining tail actions (A_0[4:8]) on the robot while the model predicts A_1. | |
| 5. Transition seamlessly into A_1 with zero robot idling. | |
| ```python | |
| from concurrent.futures import ThreadPoolExecutor | |
| executor = ThreadPoolExecutor(max_workers=1) | |
| infer.reset(task_description="pick up the black bowl in the center and place it on the plate") | |
| # 1. Reset environment and capture initial observation at t = 0 | |
| obs = env.reset() | |
| agentview_img = obs["agentview_image"] | |
| wristview_img = obs["robot0_eye_in_hand_image"] | |
| # 2. Generate initial 8-step action chunk (covers t = 0..7) | |
| current_chunk = infer.predict_chunk(agentview_img, wristview_img) | |
| # 3. Execute the initial head (t = 0..3) before entering the pipelined loop | |
| done = False | |
| for a in current_chunk[0:4]: | |
| obs, reward, done, _ = env.step(a) | |
| if done: | |
| break | |
| # Continuous overlapped loop: each cycle executes tail actions (4..7) while predicting the next chunk | |
| while not done: | |
| # 4. Capture observation at step t = 4 (or 4k) | |
| agentview_img = obs["agentview_image"] | |
| wristview_img = obs["robot0_eye_in_hand_image"] | |
| # 5. Trigger background inference for next chunk (covers upcoming horizon) | |
| future_chunk = executor.submit(infer.predict_chunk, agentview_img, wristview_img) | |
| # 6. Overlap: Execute remaining 4 tail actions (4..7) while background inference runs | |
| for a in current_chunk[4:8]: | |
| obs, reward, done, _ = env.step(a) | |
| if done: | |
| break | |
| if done: | |
| break | |
| # 7. Retrieve newly generated chunk with zero robot idle time | |
| current_chunk = future_chunk.result() | |
| ``` | |
| ### 4. World-Model Self-Rollout (Mental Simulation) | |
| Parvulus provides optional auxiliary decoders (`parvulus_auxiliary.safetensors`) for predicting future frames without interacting with the physical environment: | |
| ```python | |
| infer = ParvulusInference.from_pretrained( | |
| ".", | |
| load_auxiliary=True, | |
| device="cuda" | |
| ) | |
| # Predict action chunk and imagined next observation | |
| action_chunk, next_agent_img, next_wrist_img = infer.rollout_step( | |
| agentview_img, wristview_img, task_description="push the button" | |
| ) | |
| ``` | |
| --- | |
| ## 5. Standalone Native C++ / CUDA Runtime (Vector-First, 7.3 ms / chunk) | |
| For high-frequency physical robot deployments or low-overhead environments, Parvulus includes a standalone C++/CUDA runtime engine in `c_engine/` with **few dependencies**. | |
| ### Features | |
| - Complete dual-camera visual encoding + 10-step Euler ODE flow matching loop runs in **<8 ms** (>1,000 Hz action bandwidth). | |
| - Natively conditioned on 256-dim continuous embedding vectors (`task_vector`) to eliminate text tokenization latency in the robot control loop. | |
| - Automatically maps arbitrary natural language instructions or perturbations into 256-dim vectors via ONNX Runtime (<4 ms) before passing to the engine. | |
| ### Preparing & Building the Runtime | |
| ```bash | |
| # Automated preparation (builds C engine, prepares weights & ONNX text encoder) | |
| python3 prepare_runtime.py | |
| # Or build C++ engine directly: | |
| cd c_engine && make -j$(nproc) | |
| ``` | |
| This produces: | |
| - `libparvulus.so`: Shared C ABI library for Python ctypes, C, C++, ROS 2, or Rust. | |
| - `parvulus_runtime`: Standalone CLI binary for headless benchmarks and batch action generation. | |
| - `export_text_encoder_onnx.py`: Converter utility to export the 136 MB ONNX text encoder on demand (or download it from Hugging Face). | |
| ### Standalone CLI Benchmark | |
| ```bash | |
| ./c_engine/parvulus_runtime --benchmark 100 | |
| ``` | |
| ### Python Quickstart (`parvulus_runtime.py`) | |
| ```python | |
| import numpy as np | |
| from parvulus_runtime import ParvulusRuntime | |
| # Initialize standalone C++/CUDA engine | |
| runtime = ParvulusRuntime(device_id=0, num_euler_steps=10) | |
| # 1. Condition with a 256-dim task vector: | |
| runtime.reset(task_vector=my_256_dim_vector) | |
| # Or condition with arbitrary text (automatic dynamic ONNX encoding): | |
| # runtime.reset(task_description="grasp the black bowl and place it on the plate") | |
| # 2. Step in real-time robot loop (streams only camera frames at 136 Hz): | |
| action = runtime.step(agentview_img, wristview_img) # [7] float32 | |
| ``` | |
| For detailed C ABI signatures and non-Python integration instructions, see [`c_engine/README.md`](c_engine/README.md). | |
| --- | |
| ## 6. Intel Iris Xe / Arc Heterogeneous Hybrid Mode | |
| For deployment on mobile workstations or edge devices with Intel Iris Xe or Arc integrated graphics: | |
| ```python | |
| infer = ParvulusInference.from_pretrained(".", device="hybrid") | |
| ``` | |
| In `hybrid` mode: | |
| 1. Perception backbone (`DualViewEncoder`) executes on the iGPU (`torch.xpu`). | |
| 2. Only 16 pooled visual tokens (16 KB) transfer over PCIe to the host CPU. | |
| 3. The 10-step ODE Transformer loop runs on the high-clock CPU cores. | |
| --- | |
| ## 7. Model Details & Pinned Dependencies | |
| - **Input Specifications**: | |
| - `agentview`: [256, 256, 3] RGB. | |
| - `wristview`: [256, 256, 3] RGB. | |
| - `instruction`: Natural language string. | |
| - **Output Action Format**: | |
| - 8-step continuous trajectory A_t [8, 7] (dx, dy, dz, dΞΈx, dΞΈy, dΞΈz, gripper). | |
| - **Pinned Text Model**: | |
| - `google-t5/t5-small` pinned to commit revision `df1b051c49625cf57a3d0d8d3863ed4d13564fe4`. | |
| --- | |
| ## Citation | |
| Parvulus model weights and the associated inference code are released under a permissive license. If you find our work useful, please cite: | |
| ```bibtex | |
| @inproceedings{ | |
| sivas2026parvulus, | |
| title={Parvulus: A 10M-Scale World Action Model for Robot Manipulation}, | |
| author={Abdullah Ali Sivas and Peter Cho-Ho Lam and Zirui Zhou and Yong Zhang}, | |
| booktitle={CoRL 2026 Workshop on The Action Gap: Large Generative Models Learn Rich Knowledge}, | |
| year={2026}, | |
| url={https://openreview.net/forum?id=VcS0dE1Aoz} | |
| } | |
| ``` | |