|
Download c_engine/README.md from aasivas/Parvulus: direct link, hf CLI and curl.
- Browser
- Download file 9.03 kB
-
https://huggingface.co/aasivas/Parvulus/resolve/main/c_engine/README.md
- Command line
-
hf download hf://aasivas/Parvulus/c_engine/README.md
-
curl -L -o README.md https://huggingface.co/aasivas/Parvulus/resolve/main/c_engine/README.md
9.03 kB
PARVULUS Standalone Native Runtime Engine (C++ / CUDA)
High-performance, pure CUDA/C++ inference runtime for the PARVULUS 10M-parameter Flow Matching World Action Model.
- Built solely with
nvcc,g++,libcublas, andlibcudart. - Policy natively accepts 256-dimensional continuous conditioning vectors ([256] float32). Needs to be evaluated once per episode or can be modified asynchronously.
- **< 8 ms per 8-step action chunk** (~>125 Hz chunk rate, >1,000 Hz action bandwidth).
- Direct C ABI shared library (
libparvulus.so), standalone CLI binary (parvulus_runtime), and zero-overhead Python bindings.
1. Directory Structure
c_engine/
βββ Makefile # Make-based build system for library, CLI, and test binaries
βββ build.sh # Build script (runs make clean && make -j)
β
βββ parvulus_engine.h # Core C++ ParvulusEngine class header
βββ parvulus_engine.cu # Pure CUDA/cuBLAS implementation (Euler ODE loop, cross-attention)
βββ vision_encoder.h # DualView visual backbone header
βββ vision_encoder.cuh # DualView CUDA kernels (patch embed, depthwise-separable convs)
βββ vision_encoder.cu # DualView CUDA implementation
βββ kernels.cuh # Fused CUDA kernels (LayerNorm, SiLU, Softmax, Add, Unscale)
β
βββ parvulus_c_api.h # Standard C ABI header for external language bindings / middleware
βββ parvulus_c_api.cpp # C ABI export implementation
β
βββ parvulus_runtime_cli.cpp # Standalone CLI binary entry point
βββ test_vision_encoder.cpp # Standalone numerical parity test for vision backbone
βββ infer_c.py # Python ctypes wrapper
β
βββ export_from_safetensors.py # Serializes .safetensors into binary weight files
βββ export_text_encoder_onnx.py # Standalone converter for T5 + projection head to ONNX
βββ parvulus_model_weights.bin # Exported FP32 weights (37.28 MB, 273 tensors)
βββ task_embeddings.bin # Pre-cached 256-dim embeddings for canonical tasks
βββ action_scale.bin # 7-DoF min/max action normalization bounds
2. Requirements & Building
Prerequisites
- CUDA Toolkit (11.8+ or 12.0+) with
nvcc,libcudart,libcublas - C++17 Compiler (
g++9.0+) - Python or PyTorch runtime are not needed for C/C++ usage.
Build Targets
To build the shared library libparvulus.so, standalone CLI binary parvulus_runtime, and test binary:
cd c_engine
make -j$(nproc)
Or run the build script:
./build.sh
Run Sanity Verification Tests & Benchmark:
make test
This runs:
test_vision_encoder: Verifies DualView vision backbone parity and reports perception latency.parvulus_runtime --benchmark 20: Runs a 20-chunk end-to-end benchmark and verifies end-to-end latency.
3. Standalone CLI Binary (parvulus_runtime)
The binary parvulus_runtime can be executed directly from bash or integrated into non-Python robotics controllers.
./c_engine/parvulus_runtime [options]
Command-Line Options
| Option | Type | Default | Description |
|---|---|---|---|
--device |
int |
0 |
Target CUDA GPU device ID |
--weights |
path |
parvulus_model_weights.bin |
Path to binary model weights |
--embeddings |
path |
task_embeddings.bin |
Optional lookup table for canonical tasks |
--scale |
path |
action_scale.bin |
Action scale boundaries |
--nfe |
int |
10 |
Euler ODE flow steps (10 recommended) |
--task_vector_file |
path |
"" |
Load 256-dim task vector from .bin, .npy, or .txt |
--task_vector |
string |
"" |
Comma-separated 256 floats: "0.12,-0.05,..." |
--task |
string |
"" |
Task string (resolved via pre-cached table) |
--agent_img |
path |
"" |
Path to raw 256 x 256 x 3 uint8 RGB agent camera frame |
--wrist_img |
path |
"" |
Path to raw 256 x 256 x 3 uint8 RGB wrist camera frame |
--output |
path |
"" |
File path to write predicted 8 x 7 action trajectory |
--benchmark |
int |
0 |
Run N benchmark iterations and print latency breakdown |
CLI Usage Examples
1. High-frequency Latency Benchmark:
./c_engine/parvulus_runtime --benchmark 100
Output:
[Benchmark] Completed 100 chunks in 727.5 ms
[Benchmark] Mean Latency: 7.275 ms / chunk
[Benchmark] Action Rate: 1099.7 Hz (8 actions / 7.275 ms)
2. Predict Actions from a Custom Task Vector File:
./c_engine/parvulus_runtime \
--task_vector_file custom_task.bin \
--agent_img agentview.raw \
--wrist_img wristview.raw \
--output predicted_actions.txt
4. C ABI Interface (parvulus_c_api.h)
For embedding into C, C++, ROS 2 nodes, or foreign languages (Rust, Go, C#), include parvulus_c_api.h:
#include "parvulus_c_api.h"
// 1. Initialize engine on GPU 0
ParvulusHandle engine = parvulus_create(0);
// 2. Load model weights and action normalizer
parvulus_load(engine, "parvulus_model_weights.bin", "task_embeddings.bin", "action_scale.bin");
// 3. Set episode task conditioning vector (256 floats)
float task_vector[256];
load_my_task_vector(task_vector);
parvulus_set_task_vector(engine, task_vector);
// 4. In robot real-time loop:
uint8_t agentview_rgb[256 * 256 * 3];
uint8_t wristview_rgb[256 * 256 * 3];
float action_chunk[8 * 7]; // 8 timesteps x 7-DoF actions
// Pass NULL for task_description to use the active task vector in GPU memory
parvulus_predict_chunk(engine, agentview_rgb, wristview_rgb, NULL, action_chunk, 10, NULL);
// 5. Cleanup
parvulus_destroy(engine);
Exported Functions
// Create / Destroy
ParvulusHandle parvulus_create(int device_id);
void parvulus_destroy(ParvulusHandle handle);
// Model Weights Loading
bool parvulus_load(ParvulusHandle handle, const char* weights_path, const char* embeddings_path, const char* scale_path);
// Task Conditioning (Vector-First)
bool parvulus_set_task_vector(ParvulusHandle handle, const float* task_vector_256);
bool parvulus_set_task_string(ParvulusHandle handle, const char* task_description);
bool parvulus_register_task_embedding(ParvulusHandle handle, const char* task_description, const float* embedding_256);
// Chunk Prediction
bool parvulus_predict_chunk(ParvulusHandle handle, const uint8_t* agentview_rgb, const uint8_t* wristview_rgb,
const char* task_description, float* out_actions_56, int num_euler_steps,
const float* custom_z0);
bool parvulus_predict_chunk_with_vector(ParvulusHandle handle, const uint8_t* agentview_rgb, const uint8_t* wristview_rgb,
const float* task_vector_256, float* out_actions_56, int num_euler_steps,
const float* custom_z0);
5. Python Integration
Quickstart with ParvulusRuntime
import numpy as np
from parvulus_runtime import ParvulusRuntime
# Initialize runtime
runtime = ParvulusRuntime(device_id=0, num_euler_steps=10)
# Mode A: Vector-First conditioning (e.g., custom, perturbed, or retrieved embedding)
custom_vector = np.random.randn(256).astype(np.float32)
runtime.reset(task_vector=custom_vector)
# Mode B: Dynamic text instruction (automatically handles perturbations via ONNX)
runtime.reset(task_description="grasp the black bowl and set it onto the plate")
# Real-time robot step loop (Receding Horizon, 136 Hz chunking)
agent_rgb = np.zeros((256, 256, 3), dtype=np.uint8)
wrist_rgb = np.zeros((256, 256, 3), dtype=np.uint8)
# Single 7-DoF step
action = runtime.step(agent_rgb, wrist_rgb)
# Or full 8-step chunk prediction [8, 7]
chunk = runtime.predict_chunk(agent_rgb, wrist_rgb)
6. Language Handling
When using string instructions:
- If the instruction matches a canonical benchmark task, its embedding is resolved instantly (< 1 Β΅s) from
task_embeddings.bin. - If an instruction differs (e.g., "grasp the black bowl between the plate and the ramekin" vs. "take the black bowl and set it onto the plate"):
- If
parvulus_text_encoder.onnxis present (or downloaded from Hugging Face / generated via converter),ParvulusRuntimeencodes the prompt. - If ONNX is not generated,
ParvulusRuntimeautomatically uses PyTorch (text_encoder.py) without failure.
- If
- Task conditioning is set once per episode (t = 0). The control loop streams only camera frames, completely bypassing any text-processing overhead.
Converting or Downloading the ONNX Text Encoder
The ONNX text encoder binary is not tracked. To prepare it:
# Option A: Convert locally from base weights + T5-small
python3 c_engine/export_text_encoder_onnx.py
# Option B: Run automatic runtime preparation (downloads from Hugging Face or converts locally)
python3 prepare_runtime.py [--hf_repo <user/repo>]