File size: 9,028 Bytes
67c4f10 f98e714 67c4f10 f98e714 67c4f10 f98e714 67c4f10 f98e714 67c4f10 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 | # PARVULUS Standalone Native Runtime Engine (C++ / CUDA)
High-performance, pure CUDA/C++ inference runtime for the **PARVULUS** 10M-parameter Flow Matching World Action Model.
- Built solely with `nvcc`, `g++`, `libcublas`, and `libcudart`.
- Policy natively accepts 256-dimensional continuous conditioning vectors ([256] float32). Needs to be evaluated once per episode or can be modified asynchronously.
- **< 8 ms per 8-step action chunk** (~>125 Hz chunk rate, >1,000 Hz action bandwidth).
- Direct C ABI shared library (`libparvulus.so`), standalone CLI binary (`parvulus_runtime`), and zero-overhead Python bindings.
---
## 1. Directory Structure
```
c_engine/
βββ Makefile # Make-based build system for library, CLI, and test binaries
βββ build.sh # Build script (runs make clean && make -j)
β
βββ parvulus_engine.h # Core C++ ParvulusEngine class header
βββ parvulus_engine.cu # Pure CUDA/cuBLAS implementation (Euler ODE loop, cross-attention)
βββ vision_encoder.h # DualView visual backbone header
βββ vision_encoder.cuh # DualView CUDA kernels (patch embed, depthwise-separable convs)
βββ vision_encoder.cu # DualView CUDA implementation
βββ kernels.cuh # Fused CUDA kernels (LayerNorm, SiLU, Softmax, Add, Unscale)
β
βββ parvulus_c_api.h # Standard C ABI header for external language bindings / middleware
βββ parvulus_c_api.cpp # C ABI export implementation
β
βββ parvulus_runtime_cli.cpp # Standalone CLI binary entry point
βββ test_vision_encoder.cpp # Standalone numerical parity test for vision backbone
βββ infer_c.py # Python ctypes wrapper
β
βββ export_from_safetensors.py # Serializes .safetensors into binary weight files
βββ export_text_encoder_onnx.py # Standalone converter for T5 + projection head to ONNX
βββ parvulus_model_weights.bin # Exported FP32 weights (37.28 MB, 273 tensors)
βββ task_embeddings.bin # Pre-cached 256-dim embeddings for canonical tasks
βββ action_scale.bin # 7-DoF min/max action normalization bounds
```
---
## 2. Requirements & Building
### Prerequisites
- **CUDA Toolkit** (11.8+ or 12.0+) with `nvcc`, `libcudart`, `libcublas`
- **C++17 Compiler** (`g++` 9.0+)
- *Python or PyTorch runtime are not needed for C/C++ usage.*
### Build Targets
To build the shared library `libparvulus.so`, standalone CLI binary `parvulus_runtime`, and test binary:
```bash
cd c_engine
make -j$(nproc)
```
Or run the build script:
```bash
./build.sh
```
### Run Sanity Verification Tests & Benchmark:
```bash
make test
```
This runs:
1. `test_vision_encoder`: Verifies DualView vision backbone parity and reports perception latency.
2. `parvulus_runtime --benchmark 20`: Runs a 20-chunk end-to-end benchmark and verifies end-to-end latency.
---
## 3. Standalone CLI Binary (`parvulus_runtime`)
The binary `parvulus_runtime` can be executed directly from bash or integrated into non-Python robotics controllers.
```bash
./c_engine/parvulus_runtime [options]
```
### Command-Line Options
| Option | Type | Default | Description |
| :--- | :--- | :--- | :--- |
| `--device` | `int` | `0` | Target CUDA GPU device ID |
| `--weights` | `path` | `parvulus_model_weights.bin` | Path to binary model weights |
| `--embeddings` | `path` | `task_embeddings.bin` | Optional lookup table for canonical tasks |
| `--scale` | `path` | `action_scale.bin` | Action scale boundaries |
| `--nfe` | `int` | `10` | Euler ODE flow steps (10 recommended) |
| `--task_vector_file` | `path` | `""` | Load 256-dim task vector from `.bin`, `.npy`, or `.txt` |
| `--task_vector` | `string` | `""` | Comma-separated 256 floats: `"0.12,-0.05,..."` |
| `--task` | `string` | `""` | Task string (resolved via pre-cached table) |
| `--agent_img` | `path` | `""` | Path to raw 256 x 256 x 3 uint8 RGB agent camera frame |
| `--wrist_img` | `path` | `""` | Path to raw 256 x 256 x 3 uint8 RGB wrist camera frame |
| `--output` | `path` | `""` | File path to write predicted 8 x 7 action trajectory |
| `--benchmark` | `int` | `0` | Run N benchmark iterations and print latency breakdown |
### CLI Usage Examples
**1. High-frequency Latency Benchmark:**
```bash
./c_engine/parvulus_runtime --benchmark 100
```
*Output:*
```
[Benchmark] Completed 100 chunks in 727.5 ms
[Benchmark] Mean Latency: 7.275 ms / chunk
[Benchmark] Action Rate: 1099.7 Hz (8 actions / 7.275 ms)
```
**2. Predict Actions from a Custom Task Vector File:**
```bash
./c_engine/parvulus_runtime \
--task_vector_file custom_task.bin \
--agent_img agentview.raw \
--wrist_img wristview.raw \
--output predicted_actions.txt
```
---
## 4. C ABI Interface (`parvulus_c_api.h`)
For embedding into C, C++, ROS 2 nodes, or foreign languages (Rust, Go, C#), include `parvulus_c_api.h`:
```c
#include "parvulus_c_api.h"
// 1. Initialize engine on GPU 0
ParvulusHandle engine = parvulus_create(0);
// 2. Load model weights and action normalizer
parvulus_load(engine, "parvulus_model_weights.bin", "task_embeddings.bin", "action_scale.bin");
// 3. Set episode task conditioning vector (256 floats)
float task_vector[256];
load_my_task_vector(task_vector);
parvulus_set_task_vector(engine, task_vector);
// 4. In robot real-time loop:
uint8_t agentview_rgb[256 * 256 * 3];
uint8_t wristview_rgb[256 * 256 * 3];
float action_chunk[8 * 7]; // 8 timesteps x 7-DoF actions
// Pass NULL for task_description to use the active task vector in GPU memory
parvulus_predict_chunk(engine, agentview_rgb, wristview_rgb, NULL, action_chunk, 10, NULL);
// 5. Cleanup
parvulus_destroy(engine);
```
### Exported Functions
```c
// Create / Destroy
ParvulusHandle parvulus_create(int device_id);
void parvulus_destroy(ParvulusHandle handle);
// Model Weights Loading
bool parvulus_load(ParvulusHandle handle, const char* weights_path, const char* embeddings_path, const char* scale_path);
// Task Conditioning (Vector-First)
bool parvulus_set_task_vector(ParvulusHandle handle, const float* task_vector_256);
bool parvulus_set_task_string(ParvulusHandle handle, const char* task_description);
bool parvulus_register_task_embedding(ParvulusHandle handle, const char* task_description, const float* embedding_256);
// Chunk Prediction
bool parvulus_predict_chunk(ParvulusHandle handle, const uint8_t* agentview_rgb, const uint8_t* wristview_rgb,
const char* task_description, float* out_actions_56, int num_euler_steps,
const float* custom_z0);
bool parvulus_predict_chunk_with_vector(ParvulusHandle handle, const uint8_t* agentview_rgb, const uint8_t* wristview_rgb,
const float* task_vector_256, float* out_actions_56, int num_euler_steps,
const float* custom_z0);
```
---
## 5. Python Integration
### Quickstart with `ParvulusRuntime`
```python
import numpy as np
from parvulus_runtime import ParvulusRuntime
# Initialize runtime
runtime = ParvulusRuntime(device_id=0, num_euler_steps=10)
# Mode A: Vector-First conditioning (e.g., custom, perturbed, or retrieved embedding)
custom_vector = np.random.randn(256).astype(np.float32)
runtime.reset(task_vector=custom_vector)
# Mode B: Dynamic text instruction (automatically handles perturbations via ONNX)
runtime.reset(task_description="grasp the black bowl and set it onto the plate")
# Real-time robot step loop (Receding Horizon, 136 Hz chunking)
agent_rgb = np.zeros((256, 256, 3), dtype=np.uint8)
wrist_rgb = np.zeros((256, 256, 3), dtype=np.uint8)
# Single 7-DoF step
action = runtime.step(agent_rgb, wrist_rgb)
# Or full 8-step chunk prediction [8, 7]
chunk = runtime.predict_chunk(agent_rgb, wrist_rgb)
```
---
## 6. Language Handling
When using string instructions:
1. If the instruction matches a canonical benchmark task, its embedding is resolved instantly (< 1 Β΅s) from `task_embeddings.bin`.
2. If an instruction differs (e.g., *"grasp the black bowl between the plate and the ramekin"* vs. *"take the black bowl and set it onto the plate"*):
- If `parvulus_text_encoder.onnx` is present (or downloaded from Hugging Face / generated via converter), `ParvulusRuntime` encodes the prompt.
- If ONNX is not generated, `ParvulusRuntime` automatically uses PyTorch (`text_encoder.py`) without failure.
3. Task conditioning is set once per episode (t = 0). The control loop streams only camera frames, completely bypassing any text-processing overhead.
### Converting or Downloading the ONNX Text Encoder
The ONNX text encoder binary is not tracked. To prepare it:
```bash
# Option A: Convert locally from base weights + T5-small
python3 c_engine/export_text_encoder_onnx.py
# Option B: Run automatic runtime preparation (downloads from Hugging Face or converts locally)
python3 prepare_runtime.py [--hf_repo <user/repo>]
```
|