|
Download c_engine/README.md from aasivas/Parvulus: direct link, hf CLI and curl.
- Browser
- Download file 9.03 kB
-
https://huggingface.co/aasivas/Parvulus/resolve/main/c_engine/README.md
- Command line
-
hf download hf://aasivas/Parvulus/c_engine/README.md
-
curl -L -o README.md https://huggingface.co/aasivas/Parvulus/resolve/main/c_engine/README.md
9.03 kB
| # PARVULUS Standalone Native Runtime Engine (C++ / CUDA) | |
| High-performance, pure CUDA/C++ inference runtime for the **PARVULUS** 10M-parameter Flow Matching World Action Model. | |
| - Built solely with `nvcc`, `g++`, `libcublas`, and `libcudart`. | |
| - Policy natively accepts 256-dimensional continuous conditioning vectors ([256] float32). Needs to be evaluated once per episode or can be modified asynchronously. | |
| - **< 8 ms per 8-step action chunk** (~>125 Hz chunk rate, >1,000 Hz action bandwidth). | |
| - Direct C ABI shared library (`libparvulus.so`), standalone CLI binary (`parvulus_runtime`), and zero-overhead Python bindings. | |
| --- | |
| ## 1. Directory Structure | |
| ``` | |
| c_engine/ | |
| βββ Makefile # Make-based build system for library, CLI, and test binaries | |
| βββ build.sh # Build script (runs make clean && make -j) | |
| β | |
| βββ parvulus_engine.h # Core C++ ParvulusEngine class header | |
| βββ parvulus_engine.cu # Pure CUDA/cuBLAS implementation (Euler ODE loop, cross-attention) | |
| βββ vision_encoder.h # DualView visual backbone header | |
| βββ vision_encoder.cuh # DualView CUDA kernels (patch embed, depthwise-separable convs) | |
| βββ vision_encoder.cu # DualView CUDA implementation | |
| βββ kernels.cuh # Fused CUDA kernels (LayerNorm, SiLU, Softmax, Add, Unscale) | |
| β | |
| βββ parvulus_c_api.h # Standard C ABI header for external language bindings / middleware | |
| βββ parvulus_c_api.cpp # C ABI export implementation | |
| β | |
| βββ parvulus_runtime_cli.cpp # Standalone CLI binary entry point | |
| βββ test_vision_encoder.cpp # Standalone numerical parity test for vision backbone | |
| βββ infer_c.py # Python ctypes wrapper | |
| β | |
| βββ export_from_safetensors.py # Serializes .safetensors into binary weight files | |
| βββ export_text_encoder_onnx.py # Standalone converter for T5 + projection head to ONNX | |
| βββ parvulus_model_weights.bin # Exported FP32 weights (37.28 MB, 273 tensors) | |
| βββ task_embeddings.bin # Pre-cached 256-dim embeddings for canonical tasks | |
| βββ action_scale.bin # 7-DoF min/max action normalization bounds | |
| ``` | |
| --- | |
| ## 2. Requirements & Building | |
| ### Prerequisites | |
| - **CUDA Toolkit** (11.8+ or 12.0+) with `nvcc`, `libcudart`, `libcublas` | |
| - **C++17 Compiler** (`g++` 9.0+) | |
| - *Python or PyTorch runtime are not needed for C/C++ usage.* | |
| ### Build Targets | |
| To build the shared library `libparvulus.so`, standalone CLI binary `parvulus_runtime`, and test binary: | |
| ```bash | |
| cd c_engine | |
| make -j$(nproc) | |
| ``` | |
| Or run the build script: | |
| ```bash | |
| ./build.sh | |
| ``` | |
| ### Run Sanity Verification Tests & Benchmark: | |
| ```bash | |
| make test | |
| ``` | |
| This runs: | |
| 1. `test_vision_encoder`: Verifies DualView vision backbone parity and reports perception latency. | |
| 2. `parvulus_runtime --benchmark 20`: Runs a 20-chunk end-to-end benchmark and verifies end-to-end latency. | |
| --- | |
| ## 3. Standalone CLI Binary (`parvulus_runtime`) | |
| The binary `parvulus_runtime` can be executed directly from bash or integrated into non-Python robotics controllers. | |
| ```bash | |
| ./c_engine/parvulus_runtime [options] | |
| ``` | |
| ### Command-Line Options | |
| | Option | Type | Default | Description | | |
| | :--- | :--- | :--- | :--- | | |
| | `--device` | `int` | `0` | Target CUDA GPU device ID | | |
| | `--weights` | `path` | `parvulus_model_weights.bin` | Path to binary model weights | | |
| | `--embeddings` | `path` | `task_embeddings.bin` | Optional lookup table for canonical tasks | | |
| | `--scale` | `path` | `action_scale.bin` | Action scale boundaries | | |
| | `--nfe` | `int` | `10` | Euler ODE flow steps (10 recommended) | | |
| | `--task_vector_file` | `path` | `""` | Load 256-dim task vector from `.bin`, `.npy`, or `.txt` | | |
| | `--task_vector` | `string` | `""` | Comma-separated 256 floats: `"0.12,-0.05,..."` | | |
| | `--task` | `string` | `""` | Task string (resolved via pre-cached table) | | |
| | `--agent_img` | `path` | `""` | Path to raw 256 x 256 x 3 uint8 RGB agent camera frame | | |
| | `--wrist_img` | `path` | `""` | Path to raw 256 x 256 x 3 uint8 RGB wrist camera frame | | |
| | `--output` | `path` | `""` | File path to write predicted 8 x 7 action trajectory | | |
| | `--benchmark` | `int` | `0` | Run N benchmark iterations and print latency breakdown | | |
| ### CLI Usage Examples | |
| **1. High-frequency Latency Benchmark:** | |
| ```bash | |
| ./c_engine/parvulus_runtime --benchmark 100 | |
| ``` | |
| *Output:* | |
| ``` | |
| [Benchmark] Completed 100 chunks in 727.5 ms | |
| [Benchmark] Mean Latency: 7.275 ms / chunk | |
| [Benchmark] Action Rate: 1099.7 Hz (8 actions / 7.275 ms) | |
| ``` | |
| **2. Predict Actions from a Custom Task Vector File:** | |
| ```bash | |
| ./c_engine/parvulus_runtime \ | |
| --task_vector_file custom_task.bin \ | |
| --agent_img agentview.raw \ | |
| --wrist_img wristview.raw \ | |
| --output predicted_actions.txt | |
| ``` | |
| --- | |
| ## 4. C ABI Interface (`parvulus_c_api.h`) | |
| For embedding into C, C++, ROS 2 nodes, or foreign languages (Rust, Go, C#), include `parvulus_c_api.h`: | |
| ```c | |
| #include "parvulus_c_api.h" | |
| // 1. Initialize engine on GPU 0 | |
| ParvulusHandle engine = parvulus_create(0); | |
| // 2. Load model weights and action normalizer | |
| parvulus_load(engine, "parvulus_model_weights.bin", "task_embeddings.bin", "action_scale.bin"); | |
| // 3. Set episode task conditioning vector (256 floats) | |
| float task_vector[256]; | |
| load_my_task_vector(task_vector); | |
| parvulus_set_task_vector(engine, task_vector); | |
| // 4. In robot real-time loop: | |
| uint8_t agentview_rgb[256 * 256 * 3]; | |
| uint8_t wristview_rgb[256 * 256 * 3]; | |
| float action_chunk[8 * 7]; // 8 timesteps x 7-DoF actions | |
| // Pass NULL for task_description to use the active task vector in GPU memory | |
| parvulus_predict_chunk(engine, agentview_rgb, wristview_rgb, NULL, action_chunk, 10, NULL); | |
| // 5. Cleanup | |
| parvulus_destroy(engine); | |
| ``` | |
| ### Exported Functions | |
| ```c | |
| // Create / Destroy | |
| ParvulusHandle parvulus_create(int device_id); | |
| void parvulus_destroy(ParvulusHandle handle); | |
| // Model Weights Loading | |
| bool parvulus_load(ParvulusHandle handle, const char* weights_path, const char* embeddings_path, const char* scale_path); | |
| // Task Conditioning (Vector-First) | |
| bool parvulus_set_task_vector(ParvulusHandle handle, const float* task_vector_256); | |
| bool parvulus_set_task_string(ParvulusHandle handle, const char* task_description); | |
| bool parvulus_register_task_embedding(ParvulusHandle handle, const char* task_description, const float* embedding_256); | |
| // Chunk Prediction | |
| bool parvulus_predict_chunk(ParvulusHandle handle, const uint8_t* agentview_rgb, const uint8_t* wristview_rgb, | |
| const char* task_description, float* out_actions_56, int num_euler_steps, | |
| const float* custom_z0); | |
| bool parvulus_predict_chunk_with_vector(ParvulusHandle handle, const uint8_t* agentview_rgb, const uint8_t* wristview_rgb, | |
| const float* task_vector_256, float* out_actions_56, int num_euler_steps, | |
| const float* custom_z0); | |
| ``` | |
| --- | |
| ## 5. Python Integration | |
| ### Quickstart with `ParvulusRuntime` | |
| ```python | |
| import numpy as np | |
| from parvulus_runtime import ParvulusRuntime | |
| # Initialize runtime | |
| runtime = ParvulusRuntime(device_id=0, num_euler_steps=10) | |
| # Mode A: Vector-First conditioning (e.g., custom, perturbed, or retrieved embedding) | |
| custom_vector = np.random.randn(256).astype(np.float32) | |
| runtime.reset(task_vector=custom_vector) | |
| # Mode B: Dynamic text instruction (automatically handles perturbations via ONNX) | |
| runtime.reset(task_description="grasp the black bowl and set it onto the plate") | |
| # Real-time robot step loop (Receding Horizon, 136 Hz chunking) | |
| agent_rgb = np.zeros((256, 256, 3), dtype=np.uint8) | |
| wrist_rgb = np.zeros((256, 256, 3), dtype=np.uint8) | |
| # Single 7-DoF step | |
| action = runtime.step(agent_rgb, wrist_rgb) | |
| # Or full 8-step chunk prediction [8, 7] | |
| chunk = runtime.predict_chunk(agent_rgb, wrist_rgb) | |
| ``` | |
| --- | |
| ## 6. Language Handling | |
| When using string instructions: | |
| 1. If the instruction matches a canonical benchmark task, its embedding is resolved instantly (< 1 Β΅s) from `task_embeddings.bin`. | |
| 2. If an instruction differs (e.g., *"grasp the black bowl between the plate and the ramekin"* vs. *"take the black bowl and set it onto the plate"*): | |
| - If `parvulus_text_encoder.onnx` is present (or downloaded from Hugging Face / generated via converter), `ParvulusRuntime` encodes the prompt. | |
| - If ONNX is not generated, `ParvulusRuntime` automatically uses PyTorch (`text_encoder.py`) without failure. | |
| 3. Task conditioning is set once per episode (t = 0). The control loop streams only camera frames, completely bypassing any text-processing overhead. | |
| ### Converting or Downloading the ONNX Text Encoder | |
| The ONNX text encoder binary is not tracked. To prepare it: | |
| ```bash | |
| # Option A: Convert locally from base weights + T5-small | |
| python3 c_engine/export_text_encoder_onnx.py | |
| # Option B: Run automatic runtime preparation (downloads from Hugging Face or converts locally) | |
| python3 prepare_runtime.py [--hf_repo <user/repo>] | |
| ``` | |