Parvulus / README.md
aasivas's picture
Upload folder using huggingface_hub
f98e714 verified
|
Raw History Blame Contribute Delete
12 kB
metadata
language:
  - en
license: apache-2.0
tags:
  - robotics
  - flow-matching
  - world-models
  - imitation-learning
  - libero
  - action-chunking
  - pytorch
  - safetensors
pipeline_tag: robotics

PARVULUS: 10M-Scale World Action Model for Robot Manipulation

PARVULUS is an ultra-efficient 10M-parameter Flow Matching World Action Model (WAM) designed for continuous robot manipulation. Trained on the canonical LIBERO benchmark, Parvulus achieves a 90.88% average success rate across all 40 tasks while running at real-time control frequencies (>100 Hz in C++/CUDA, sub-40ms with PyTorch KV-caching).


1. Highlights

  • 9.98M parameters (~20 MB in BF16/FP16, ~38 MB in FP32) for standalone robot control.
  • 90.88% Β± 0.37% average success across all 40 canonical LIBERO tasks (N = 6,000 simulation rollouts).
  • Pinned google-t5/t5-small text representations with a trained 2-layer projection head.
  • Multiple Execution Modes:
    • PyTorch KV-Cached: β‰ˆ 38.8 ms (206 Hz action bandwidth).
    • Standalone C++ / CUDA Engine: 7.28 ms (>1,000 Hz action bandwidth, pure CUDA/cuBLAS).
    • Intel Iris Xe Hybrid: β‰ˆ 106.6 ms on low-power laptop integrated graphics (xpu vision + CPU ODE).
  • Includes optional auxiliary training decoders (parvulus_auxiliary.safetensors, 29.15M params) for auto-regressive visual self-rollout, motion change estimation, and self-reconstruction from noisy images.

2. Benchmark Performance

Evaluated across the 4 canonical LIBERO suites (N = 6,000 physical rollouts, 10 seeds, receding-horizon chunking H = 8, C = 8):

Benchmark Suite Canonical Tasks Rollout Trials Success Rate
LIBERO-Spatial 10 1,500 92.67%
LIBERO-Object 10 1,500 91.13%
LIBERO-Goal 10 1,500 89.47%
LIBERO-10 (Long) 10 1,500 90.27%
Full Benchmark Average 40 6,000 90.88% Β± 0.37%

3. Repository Structure

.
β”œβ”€β”€ parvulus.safetensors            # Action-only core policy weights (9.98M params, ~39 MB FP32)
β”œβ”€β”€ parvulus_full.safetensors       # Full model weights: action policy + auxiliary heads (39.12M params, ~156 MB)
β”œβ”€β”€ parvulus_auxiliary.safetensors  # Auxiliary world model heads only (video, change, self-recon; 29.15M params)
β”œβ”€β”€ model.safetensors               # HuggingFace default entrypoint (alias of parvulus.safetensors)
β”‚
β”œβ”€β”€ config.json                     # Model hyperparameters and pinned T5 commit revision
β”œβ”€β”€ action_scale.json               # 7-DoF per-channel min/max normalization boundaries
β”œβ”€β”€ pyproject.toml                  # Standard package configuration
β”œβ”€β”€ requirements.txt                # Minimal Python dependencies
β”œβ”€β”€ prepare_runtime.py              # Setup script: builds C engine, generates binaries, prepares text encoder
β”‚
β”œβ”€β”€ parvulus_runtime.py             # High-performance Python runtime for standalone C engine
β”œβ”€β”€ infer.py                        # High-level ParvulusInference client (supports CUDA, C engine, XPU, Hybrid)
β”œβ”€β”€ parvulus.py                     # Complete PyTorch architecture (DualViewEncoder, CVMixer, FlowDecoder)
β”œβ”€β”€ text_encoder.py                 # Pinned T5 text tokenizer and projection head
β”œβ”€β”€ action_utils.py                 # Action normalization and denormalization helpers
β”‚
└── c_engine/                       # Dependency-free C++ / CUDA Standalone Engine
    β”œβ”€β”€ Makefile                    # Make-based build script
    β”œβ”€β”€ build.sh                    # Convenient build script
    β”œβ”€β”€ parvulus_engine.cu / .h     # Core C++/CUDA inference engine (136 Hz chunk rate)
    β”œβ”€β”€ vision_encoder.cu / .cuh    # Fused DualView vision encoder
    β”œβ”€β”€ parvulus_c_api.cpp / .h     # Standard C ABI exports for C/C++/ROS 2/Rust
    β”œβ”€β”€ parvulus_runtime_cli.cpp    # Standalone CLI binary entry point
    β”œβ”€β”€ canonical_tasks.json        # 40 canonical task instructions
    β”œβ”€β”€ export_from_safetensors.py  # Script generating model binary, scale, and embeddings
    └── export_text_encoder_onnx.py # Standalone converter for T5-small text encoder to ONNX

4. Quickstart: Python Inference

Installation

pip install -r requirements.txt

1. Basic Action Chunk Prediction

import numpy as np
from PIL import Image
from infer import ParvulusInference

# Initialize client (auto-detects CUDA / XPU / CPU; enables fast KV-cache by default)
infer = ParvulusInference.from_pretrained(
    ".",
    device="cuda",      # Options: 'cuda', 'cpu', 'xpu', 'hybrid', or 'auto'
    use_kv_cache=True,  # Enables sub-40ms inference
    num_ode_steps=10
)

# Prepare inputs: dual 256x256 RGB cameras + text prompt
agentview_img = np.array(Image.open("agentview.png"))  # [256, 256, 3] uint8
wristview_img = np.array(Image.open("wristview.png"))  # [256, 256, 3] uint8
task_prompt = "pick up the black bowl in the center and place it on the plate"

# Predict 8-step continuous action chunk [8, 7]
# (dx, dy, dz, drot_x, drot_y, drot_z, gripper)
action_chunk = infer.predict_chunk(agentview_img, wristview_img, task_description=task_prompt)
print(f"Predicted action chunk (shape {action_chunk.shape}):\n{action_chunk}")

2. Environment Stepping Loop (Receding Horizon)

infer.reset(task_description="pick up the milk and place it in the basket")
obs = env.reset()
agentview_img = obs["agentview_image"]
wristview_img = obs["robot0_eye_in_hand_image"]

done = False
while not done:
    # In your simulation or real robot loop:
    action = infer.step(agentview_img, wristview_img)  # [7] float32 action
    obs, reward, done, info = env.step(action)
    agentview_img = obs["agentview_image"]
    wristview_img = obs["robot0_eye_in_hand_image"]

3. Continuous Overlapped Pipelined Execution (H = 8, C = 4)

To achieve zero-latency continuous robot motion, Parvulus supports overlapping chunk generation with robot motion execution:

  1. Predict an 8-step chunk (A_0).
  2. Execute the first 4 actions (A_0[0:4]).
  3. At step 4, capture a fresh observation and trigger background inference for the next chunk (A_1).
  4. Concurrently execute the remaining tail actions (A_0[4:8]) on the robot while the model predicts A_1.
  5. Transition seamlessly into A_1 with zero robot idling.
from concurrent.futures import ThreadPoolExecutor

executor = ThreadPoolExecutor(max_workers=1)
infer.reset(task_description="pick up the black bowl in the center and place it on the plate")

# 1. Reset environment and capture initial observation at t = 0
obs = env.reset()
agentview_img = obs["agentview_image"]
wristview_img = obs["robot0_eye_in_hand_image"]

# 2. Generate initial 8-step action chunk (covers t = 0..7)
current_chunk = infer.predict_chunk(agentview_img, wristview_img)

# 3. Execute the initial head (t = 0..3) before entering the pipelined loop
done = False
for a in current_chunk[0:4]:
    obs, reward, done, _ = env.step(a)
    if done:
        break

# Continuous overlapped loop: each cycle executes tail actions (4..7) while predicting the next chunk
while not done:
    # 4. Capture observation at step t = 4 (or 4k)
    agentview_img = obs["agentview_image"]
    wristview_img = obs["robot0_eye_in_hand_image"]

    # 5. Trigger background inference for next chunk (covers upcoming horizon)
    future_chunk = executor.submit(infer.predict_chunk, agentview_img, wristview_img)

    # 6. Overlap: Execute remaining 4 tail actions (4..7) while background inference runs
    for a in current_chunk[4:8]:
        obs, reward, done, _ = env.step(a)
        if done:
            break
    if done:
        break

    # 7. Retrieve newly generated chunk with zero robot idle time
    current_chunk = future_chunk.result()

4. World-Model Self-Rollout (Mental Simulation)

Parvulus provides optional auxiliary decoders (parvulus_auxiliary.safetensors) for predicting future frames without interacting with the physical environment:

infer = ParvulusInference.from_pretrained(
    ".",
    load_auxiliary=True,
    device="cuda"
)

# Predict action chunk and imagined next observation
action_chunk, next_agent_img, next_wrist_img = infer.rollout_step(
    agentview_img, wristview_img, task_description="push the button"
)

5. Standalone Native C++ / CUDA Runtime (Vector-First, 7.3 ms / chunk)

For high-frequency physical robot deployments or low-overhead environments, Parvulus includes a standalone C++/CUDA runtime engine in c_engine/ with few dependencies.

Features

  • Complete dual-camera visual encoding + 10-step Euler ODE flow matching loop runs in **<8 ms** (>1,000 Hz action bandwidth).
  • Natively conditioned on 256-dim continuous embedding vectors (task_vector) to eliminate text tokenization latency in the robot control loop.
  • Automatically maps arbitrary natural language instructions or perturbations into 256-dim vectors via ONNX Runtime (<4 ms) before passing to the engine.

Preparing & Building the Runtime

# Automated preparation (builds C engine, prepares weights & ONNX text encoder)
python3 prepare_runtime.py

# Or build C++ engine directly:
cd c_engine && make -j$(nproc)

This produces:

  • libparvulus.so: Shared C ABI library for Python ctypes, C, C++, ROS 2, or Rust.
  • parvulus_runtime: Standalone CLI binary for headless benchmarks and batch action generation.
  • export_text_encoder_onnx.py: Converter utility to export the 136 MB ONNX text encoder on demand (or download it from Hugging Face).

Standalone CLI Benchmark

./c_engine/parvulus_runtime --benchmark 100

Python Quickstart (parvulus_runtime.py)

import numpy as np
from parvulus_runtime import ParvulusRuntime

# Initialize standalone C++/CUDA engine
runtime = ParvulusRuntime(device_id=0, num_euler_steps=10)

# 1. Condition with a 256-dim task vector:
runtime.reset(task_vector=my_256_dim_vector)

# Or condition with arbitrary text (automatic dynamic ONNX encoding):
# runtime.reset(task_description="grasp the black bowl and place it on the plate")

# 2. Step in real-time robot loop (streams only camera frames at 136 Hz):
action = runtime.step(agentview_img, wristview_img)  # [7] float32

For detailed C ABI signatures and non-Python integration instructions, see c_engine/README.md.


6. Intel Iris Xe / Arc Heterogeneous Hybrid Mode

For deployment on mobile workstations or edge devices with Intel Iris Xe or Arc integrated graphics:

infer = ParvulusInference.from_pretrained(".", device="hybrid")

In hybrid mode:

  1. Perception backbone (DualViewEncoder) executes on the iGPU (torch.xpu).
  2. Only 16 pooled visual tokens (16 KB) transfer over PCIe to the host CPU.
  3. The 10-step ODE Transformer loop runs on the high-clock CPU cores.

7. Model Details & Pinned Dependencies

  • Input Specifications:
    • agentview: [256, 256, 3] RGB.
    • wristview: [256, 256, 3] RGB.
    • instruction: Natural language string.
  • Output Action Format:
    • 8-step continuous trajectory A_t [8, 7] (dx, dy, dz, dΞΈx, dΞΈy, dΞΈz, gripper).
  • Pinned Text Model:
    • google-t5/t5-small pinned to commit revision df1b051c49625cf57a3d0d8d3863ed4d13564fe4.

Citation

Parvulus model weights and the associated inference code are released under a permissive license. If you find our work useful, please cite:

@inproceedings{
    sivas2026parvulus,
    title={Parvulus: A 10M-Scale World Action Model for Robot Manipulation},
    author={Abdullah Ali Sivas and Peter Cho-Ho Lam and Zirui Zhou and Yong Zhang},
    booktitle={CoRL 2026 Workshop on The Action Gap: Large Generative Models Learn Rich Knowledge},
    year={2026},
    url={https://openreview.net/forum?id=VcS0dE1Aoz}
}