# PARVULUS Standalone Native Runtime Engine (C++ / CUDA) High-performance, pure CUDA/C++ inference runtime for the **PARVULUS** 10M-parameter Flow Matching World Action Model. - Built solely with `nvcc`, `g++`, `libcublas`, and `libcudart`. - Policy natively accepts 256-dimensional continuous conditioning vectors ([256] float32). Needs to be evaluated once per episode or can be modified asynchronously. - **< 8 ms per 8-step action chunk** (~>125 Hz chunk rate, >1,000 Hz action bandwidth). - Direct C ABI shared library (`libparvulus.so`), standalone CLI binary (`parvulus_runtime`), and zero-overhead Python bindings. --- ## 1. Directory Structure ``` c_engine/ ├── Makefile # Make-based build system for library, CLI, and test binaries ├── build.sh # Build script (runs make clean && make -j) │ ├── parvulus_engine.h # Core C++ ParvulusEngine class header ├── parvulus_engine.cu # Pure CUDA/cuBLAS implementation (Euler ODE loop, cross-attention) ├── vision_encoder.h # DualView visual backbone header ├── vision_encoder.cuh # DualView CUDA kernels (patch embed, depthwise-separable convs) ├── vision_encoder.cu # DualView CUDA implementation ├── kernels.cuh # Fused CUDA kernels (LayerNorm, SiLU, Softmax, Add, Unscale) │ ├── parvulus_c_api.h # Standard C ABI header for external language bindings / middleware ├── parvulus_c_api.cpp # C ABI export implementation │ ├── parvulus_runtime_cli.cpp # Standalone CLI binary entry point ├── test_vision_encoder.cpp # Standalone numerical parity test for vision backbone ├── infer_c.py # Python ctypes wrapper │ ├── export_from_safetensors.py # Serializes .safetensors into binary weight files ├── export_text_encoder_onnx.py # Standalone converter for T5 + projection head to ONNX ├── parvulus_model_weights.bin # Exported FP32 weights (37.28 MB, 273 tensors) ├── task_embeddings.bin # Pre-cached 256-dim embeddings for canonical tasks └── action_scale.bin # 7-DoF min/max action normalization bounds ``` --- ## 2. Requirements & Building ### Prerequisites - **CUDA Toolkit** (11.8+ or 12.0+) with `nvcc`, `libcudart`, `libcublas` - **C++17 Compiler** (`g++` 9.0+) - *Python or PyTorch runtime are not needed for C/C++ usage.* ### Build Targets To build the shared library `libparvulus.so`, standalone CLI binary `parvulus_runtime`, and test binary: ```bash cd c_engine make -j$(nproc) ``` Or run the build script: ```bash ./build.sh ``` ### Run Sanity Verification Tests & Benchmark: ```bash make test ``` This runs: 1. `test_vision_encoder`: Verifies DualView vision backbone parity and reports perception latency. 2. `parvulus_runtime --benchmark 20`: Runs a 20-chunk end-to-end benchmark and verifies end-to-end latency. --- ## 3. Standalone CLI Binary (`parvulus_runtime`) The binary `parvulus_runtime` can be executed directly from bash or integrated into non-Python robotics controllers. ```bash ./c_engine/parvulus_runtime [options] ``` ### Command-Line Options | Option | Type | Default | Description | | :--- | :--- | :--- | :--- | | `--device` | `int` | `0` | Target CUDA GPU device ID | | `--weights` | `path` | `parvulus_model_weights.bin` | Path to binary model weights | | `--embeddings` | `path` | `task_embeddings.bin` | Optional lookup table for canonical tasks | | `--scale` | `path` | `action_scale.bin` | Action scale boundaries | | `--nfe` | `int` | `10` | Euler ODE flow steps (10 recommended) | | `--task_vector_file` | `path` | `""` | Load 256-dim task vector from `.bin`, `.npy`, or `.txt` | | `--task_vector` | `string` | `""` | Comma-separated 256 floats: `"0.12,-0.05,..."` | | `--task` | `string` | `""` | Task string (resolved via pre-cached table) | | `--agent_img` | `path` | `""` | Path to raw 256 x 256 x 3 uint8 RGB agent camera frame | | `--wrist_img` | `path` | `""` | Path to raw 256 x 256 x 3 uint8 RGB wrist camera frame | | `--output` | `path` | `""` | File path to write predicted 8 x 7 action trajectory | | `--benchmark` | `int` | `0` | Run N benchmark iterations and print latency breakdown | ### CLI Usage Examples **1. High-frequency Latency Benchmark:** ```bash ./c_engine/parvulus_runtime --benchmark 100 ``` *Output:* ``` [Benchmark] Completed 100 chunks in 727.5 ms [Benchmark] Mean Latency: 7.275 ms / chunk [Benchmark] Action Rate: 1099.7 Hz (8 actions / 7.275 ms) ``` **2. Predict Actions from a Custom Task Vector File:** ```bash ./c_engine/parvulus_runtime \ --task_vector_file custom_task.bin \ --agent_img agentview.raw \ --wrist_img wristview.raw \ --output predicted_actions.txt ``` --- ## 4. C ABI Interface (`parvulus_c_api.h`) For embedding into C, C++, ROS 2 nodes, or foreign languages (Rust, Go, C#), include `parvulus_c_api.h`: ```c #include "parvulus_c_api.h" // 1. Initialize engine on GPU 0 ParvulusHandle engine = parvulus_create(0); // 2. Load model weights and action normalizer parvulus_load(engine, "parvulus_model_weights.bin", "task_embeddings.bin", "action_scale.bin"); // 3. Set episode task conditioning vector (256 floats) float task_vector[256]; load_my_task_vector(task_vector); parvulus_set_task_vector(engine, task_vector); // 4. In robot real-time loop: uint8_t agentview_rgb[256 * 256 * 3]; uint8_t wristview_rgb[256 * 256 * 3]; float action_chunk[8 * 7]; // 8 timesteps x 7-DoF actions // Pass NULL for task_description to use the active task vector in GPU memory parvulus_predict_chunk(engine, agentview_rgb, wristview_rgb, NULL, action_chunk, 10, NULL); // 5. Cleanup parvulus_destroy(engine); ``` ### Exported Functions ```c // Create / Destroy ParvulusHandle parvulus_create(int device_id); void parvulus_destroy(ParvulusHandle handle); // Model Weights Loading bool parvulus_load(ParvulusHandle handle, const char* weights_path, const char* embeddings_path, const char* scale_path); // Task Conditioning (Vector-First) bool parvulus_set_task_vector(ParvulusHandle handle, const float* task_vector_256); bool parvulus_set_task_string(ParvulusHandle handle, const char* task_description); bool parvulus_register_task_embedding(ParvulusHandle handle, const char* task_description, const float* embedding_256); // Chunk Prediction bool parvulus_predict_chunk(ParvulusHandle handle, const uint8_t* agentview_rgb, const uint8_t* wristview_rgb, const char* task_description, float* out_actions_56, int num_euler_steps, const float* custom_z0); bool parvulus_predict_chunk_with_vector(ParvulusHandle handle, const uint8_t* agentview_rgb, const uint8_t* wristview_rgb, const float* task_vector_256, float* out_actions_56, int num_euler_steps, const float* custom_z0); ``` --- ## 5. Python Integration ### Quickstart with `ParvulusRuntime` ```python import numpy as np from parvulus_runtime import ParvulusRuntime # Initialize runtime runtime = ParvulusRuntime(device_id=0, num_euler_steps=10) # Mode A: Vector-First conditioning (e.g., custom, perturbed, or retrieved embedding) custom_vector = np.random.randn(256).astype(np.float32) runtime.reset(task_vector=custom_vector) # Mode B: Dynamic text instruction (automatically handles perturbations via ONNX) runtime.reset(task_description="grasp the black bowl and set it onto the plate") # Real-time robot step loop (Receding Horizon, 136 Hz chunking) agent_rgb = np.zeros((256, 256, 3), dtype=np.uint8) wrist_rgb = np.zeros((256, 256, 3), dtype=np.uint8) # Single 7-DoF step action = runtime.step(agent_rgb, wrist_rgb) # Or full 8-step chunk prediction [8, 7] chunk = runtime.predict_chunk(agent_rgb, wrist_rgb) ``` --- ## 6. Language Handling When using string instructions: 1. If the instruction matches a canonical benchmark task, its embedding is resolved instantly (< 1 µs) from `task_embeddings.bin`. 2. If an instruction differs (e.g., *"grasp the black bowl between the plate and the ramekin"* vs. *"take the black bowl and set it onto the plate"*): - If `parvulus_text_encoder.onnx` is present (or downloaded from Hugging Face / generated via converter), `ParvulusRuntime` encodes the prompt. - If ONNX is not generated, `ParvulusRuntime` automatically uses PyTorch (`text_encoder.py`) without failure. 3. Task conditioning is set once per episode (t = 0). The control loop streams only camera frames, completely bypassing any text-processing overhead. ### Converting or Downloading the ONNX Text Encoder The ONNX text encoder binary is not tracked. To prepare it: ```bash # Option A: Convert locally from base weights + T5-small python3 c_engine/export_text_encoder_onnx.py # Option B: Run automatic runtime preparation (downloads from Hugging Face or converts locally) python3 prepare_runtime.py [--hf_repo ] ```