Parvulus / c_engine /README.md
aasivas's picture
Upload folder using huggingface_hub
f98e714 verified
|
Raw History Blame Contribute Delete
9.03 kB

PARVULUS Standalone Native Runtime Engine (C++ / CUDA)

High-performance, pure CUDA/C++ inference runtime for the PARVULUS 10M-parameter Flow Matching World Action Model.

  • Built solely with nvcc, g++, libcublas, and libcudart.
  • Policy natively accepts 256-dimensional continuous conditioning vectors ([256] float32). Needs to be evaluated once per episode or can be modified asynchronously.
  • **< 8 ms per 8-step action chunk** (~>125 Hz chunk rate, >1,000 Hz action bandwidth).
  • Direct C ABI shared library (libparvulus.so), standalone CLI binary (parvulus_runtime), and zero-overhead Python bindings.

1. Directory Structure

c_engine/
β”œβ”€β”€ Makefile                     # Make-based build system for library, CLI, and test binaries
β”œβ”€β”€ build.sh                     # Build script (runs make clean && make -j)
β”‚
β”œβ”€β”€ parvulus_engine.h            # Core C++ ParvulusEngine class header
β”œβ”€β”€ parvulus_engine.cu           # Pure CUDA/cuBLAS implementation (Euler ODE loop, cross-attention)
β”œβ”€β”€ vision_encoder.h             # DualView visual backbone header
β”œβ”€β”€ vision_encoder.cuh           # DualView CUDA kernels (patch embed, depthwise-separable convs)
β”œβ”€β”€ vision_encoder.cu            # DualView CUDA implementation
β”œβ”€β”€ kernels.cuh                  # Fused CUDA kernels (LayerNorm, SiLU, Softmax, Add, Unscale)
β”‚
β”œβ”€β”€ parvulus_c_api.h             # Standard C ABI header for external language bindings / middleware
β”œβ”€β”€ parvulus_c_api.cpp           # C ABI export implementation
β”‚
β”œβ”€β”€ parvulus_runtime_cli.cpp     # Standalone CLI binary entry point
β”œβ”€β”€ test_vision_encoder.cpp      # Standalone numerical parity test for vision backbone
β”œβ”€β”€ infer_c.py                   # Python ctypes wrapper
β”‚
β”œβ”€β”€ export_from_safetensors.py   # Serializes .safetensors into binary weight files
β”œβ”€β”€ export_text_encoder_onnx.py  # Standalone converter for T5 + projection head to ONNX
β”œβ”€β”€ parvulus_model_weights.bin   # Exported FP32 weights (37.28 MB, 273 tensors)
β”œβ”€β”€ task_embeddings.bin          # Pre-cached 256-dim embeddings for canonical tasks
└── action_scale.bin             # 7-DoF min/max action normalization bounds

2. Requirements & Building

Prerequisites

  • CUDA Toolkit (11.8+ or 12.0+) with nvcc, libcudart, libcublas
  • C++17 Compiler (g++ 9.0+)
  • Python or PyTorch runtime are not needed for C/C++ usage.

Build Targets

To build the shared library libparvulus.so, standalone CLI binary parvulus_runtime, and test binary:

cd c_engine
make -j$(nproc)

Or run the build script:

./build.sh

Run Sanity Verification Tests & Benchmark:

make test

This runs:

  1. test_vision_encoder: Verifies DualView vision backbone parity and reports perception latency.
  2. parvulus_runtime --benchmark 20: Runs a 20-chunk end-to-end benchmark and verifies end-to-end latency.

3. Standalone CLI Binary (parvulus_runtime)

The binary parvulus_runtime can be executed directly from bash or integrated into non-Python robotics controllers.

./c_engine/parvulus_runtime [options]

Command-Line Options

Option Type Default Description
--device int 0 Target CUDA GPU device ID
--weights path parvulus_model_weights.bin Path to binary model weights
--embeddings path task_embeddings.bin Optional lookup table for canonical tasks
--scale path action_scale.bin Action scale boundaries
--nfe int 10 Euler ODE flow steps (10 recommended)
--task_vector_file path "" Load 256-dim task vector from .bin, .npy, or .txt
--task_vector string "" Comma-separated 256 floats: "0.12,-0.05,..."
--task string "" Task string (resolved via pre-cached table)
--agent_img path "" Path to raw 256 x 256 x 3 uint8 RGB agent camera frame
--wrist_img path "" Path to raw 256 x 256 x 3 uint8 RGB wrist camera frame
--output path "" File path to write predicted 8 x 7 action trajectory
--benchmark int 0 Run N benchmark iterations and print latency breakdown

CLI Usage Examples

1. High-frequency Latency Benchmark:

./c_engine/parvulus_runtime --benchmark 100

Output:

[Benchmark] Completed 100 chunks in 727.5 ms
[Benchmark] Mean Latency: 7.275 ms / chunk
[Benchmark] Action Rate:  1099.7 Hz (8 actions / 7.275 ms)

2. Predict Actions from a Custom Task Vector File:

./c_engine/parvulus_runtime \
  --task_vector_file custom_task.bin \
  --agent_img agentview.raw \
  --wrist_img wristview.raw \
  --output predicted_actions.txt

4. C ABI Interface (parvulus_c_api.h)

For embedding into C, C++, ROS 2 nodes, or foreign languages (Rust, Go, C#), include parvulus_c_api.h:

#include "parvulus_c_api.h"

// 1. Initialize engine on GPU 0
ParvulusHandle engine = parvulus_create(0);

// 2. Load model weights and action normalizer
parvulus_load(engine, "parvulus_model_weights.bin", "task_embeddings.bin", "action_scale.bin");

// 3. Set episode task conditioning vector (256 floats)
float task_vector[256];
load_my_task_vector(task_vector);
parvulus_set_task_vector(engine, task_vector);

// 4. In robot real-time loop:
uint8_t agentview_rgb[256 * 256 * 3];
uint8_t wristview_rgb[256 * 256 * 3];
float action_chunk[8 * 7]; // 8 timesteps x 7-DoF actions

// Pass NULL for task_description to use the active task vector in GPU memory
parvulus_predict_chunk(engine, agentview_rgb, wristview_rgb, NULL, action_chunk, 10, NULL);

// 5. Cleanup
parvulus_destroy(engine);

Exported Functions

// Create / Destroy
ParvulusHandle parvulus_create(int device_id);
void parvulus_destroy(ParvulusHandle handle);

// Model Weights Loading
bool parvulus_load(ParvulusHandle handle, const char* weights_path, const char* embeddings_path, const char* scale_path);

// Task Conditioning (Vector-First)
bool parvulus_set_task_vector(ParvulusHandle handle, const float* task_vector_256);
bool parvulus_set_task_string(ParvulusHandle handle, const char* task_description);
bool parvulus_register_task_embedding(ParvulusHandle handle, const char* task_description, const float* embedding_256);

// Chunk Prediction
bool parvulus_predict_chunk(ParvulusHandle handle, const uint8_t* agentview_rgb, const uint8_t* wristview_rgb,
                            const char* task_description, float* out_actions_56, int num_euler_steps,
                            const float* custom_z0);

bool parvulus_predict_chunk_with_vector(ParvulusHandle handle, const uint8_t* agentview_rgb, const uint8_t* wristview_rgb,
                                       const float* task_vector_256, float* out_actions_56, int num_euler_steps,
                                       const float* custom_z0);

5. Python Integration

Quickstart with ParvulusRuntime

import numpy as np
from parvulus_runtime import ParvulusRuntime

# Initialize runtime
runtime = ParvulusRuntime(device_id=0, num_euler_steps=10)

# Mode A: Vector-First conditioning (e.g., custom, perturbed, or retrieved embedding)
custom_vector = np.random.randn(256).astype(np.float32)
runtime.reset(task_vector=custom_vector)

# Mode B: Dynamic text instruction (automatically handles perturbations via ONNX)
runtime.reset(task_description="grasp the black bowl and set it onto the plate")

# Real-time robot step loop (Receding Horizon, 136 Hz chunking)
agent_rgb = np.zeros((256, 256, 3), dtype=np.uint8)
wrist_rgb = np.zeros((256, 256, 3), dtype=np.uint8)

# Single 7-DoF step
action = runtime.step(agent_rgb, wrist_rgb)

# Or full 8-step chunk prediction [8, 7]
chunk = runtime.predict_chunk(agent_rgb, wrist_rgb)

6. Language Handling

When using string instructions:

  1. If the instruction matches a canonical benchmark task, its embedding is resolved instantly (< 1 Β΅s) from task_embeddings.bin.
  2. If an instruction differs (e.g., "grasp the black bowl between the plate and the ramekin" vs. "take the black bowl and set it onto the plate"):
    • If parvulus_text_encoder.onnx is present (or downloaded from Hugging Face / generated via converter), ParvulusRuntime encodes the prompt.
    • If ONNX is not generated, ParvulusRuntime automatically uses PyTorch (text_encoder.py) without failure.
  3. Task conditioning is set once per episode (t = 0). The control loop streams only camera frames, completely bypassing any text-processing overhead.

Converting or Downloading the ONNX Text Encoder

The ONNX text encoder binary is not tracked. To prepare it:

# Option A: Convert locally from base weights + T5-small
python3 c_engine/export_text_encoder_onnx.py

# Option B: Run automatic runtime preparation (downloads from Hugging Face or converts locally)
python3 prepare_runtime.py [--hf_repo <user/repo>]