Parvulus / c_engine /README.md
aasivas's picture
Upload folder using huggingface_hub
f98e714 verified
|
Raw History Blame Contribute Delete
9.03 kB
# PARVULUS Standalone Native Runtime Engine (C++ / CUDA)
High-performance, pure CUDA/C++ inference runtime for the **PARVULUS** 10M-parameter Flow Matching World Action Model.
- Built solely with `nvcc`, `g++`, `libcublas`, and `libcudart`.
- Policy natively accepts 256-dimensional continuous conditioning vectors ([256] float32). Needs to be evaluated once per episode or can be modified asynchronously.
- **< 8 ms per 8-step action chunk** (~>125 Hz chunk rate, >1,000 Hz action bandwidth).
- Direct C ABI shared library (`libparvulus.so`), standalone CLI binary (`parvulus_runtime`), and zero-overhead Python bindings.
---
## 1. Directory Structure
```
c_engine/
β”œβ”€β”€ Makefile # Make-based build system for library, CLI, and test binaries
β”œβ”€β”€ build.sh # Build script (runs make clean && make -j)
β”‚
β”œβ”€β”€ parvulus_engine.h # Core C++ ParvulusEngine class header
β”œβ”€β”€ parvulus_engine.cu # Pure CUDA/cuBLAS implementation (Euler ODE loop, cross-attention)
β”œβ”€β”€ vision_encoder.h # DualView visual backbone header
β”œβ”€β”€ vision_encoder.cuh # DualView CUDA kernels (patch embed, depthwise-separable convs)
β”œβ”€β”€ vision_encoder.cu # DualView CUDA implementation
β”œβ”€β”€ kernels.cuh # Fused CUDA kernels (LayerNorm, SiLU, Softmax, Add, Unscale)
β”‚
β”œβ”€β”€ parvulus_c_api.h # Standard C ABI header for external language bindings / middleware
β”œβ”€β”€ parvulus_c_api.cpp # C ABI export implementation
β”‚
β”œβ”€β”€ parvulus_runtime_cli.cpp # Standalone CLI binary entry point
β”œβ”€β”€ test_vision_encoder.cpp # Standalone numerical parity test for vision backbone
β”œβ”€β”€ infer_c.py # Python ctypes wrapper
β”‚
β”œβ”€β”€ export_from_safetensors.py # Serializes .safetensors into binary weight files
β”œβ”€β”€ export_text_encoder_onnx.py # Standalone converter for T5 + projection head to ONNX
β”œβ”€β”€ parvulus_model_weights.bin # Exported FP32 weights (37.28 MB, 273 tensors)
β”œβ”€β”€ task_embeddings.bin # Pre-cached 256-dim embeddings for canonical tasks
└── action_scale.bin # 7-DoF min/max action normalization bounds
```
---
## 2. Requirements & Building
### Prerequisites
- **CUDA Toolkit** (11.8+ or 12.0+) with `nvcc`, `libcudart`, `libcublas`
- **C++17 Compiler** (`g++` 9.0+)
- *Python or PyTorch runtime are not needed for C/C++ usage.*
### Build Targets
To build the shared library `libparvulus.so`, standalone CLI binary `parvulus_runtime`, and test binary:
```bash
cd c_engine
make -j$(nproc)
```
Or run the build script:
```bash
./build.sh
```
### Run Sanity Verification Tests & Benchmark:
```bash
make test
```
This runs:
1. `test_vision_encoder`: Verifies DualView vision backbone parity and reports perception latency.
2. `parvulus_runtime --benchmark 20`: Runs a 20-chunk end-to-end benchmark and verifies end-to-end latency.
---
## 3. Standalone CLI Binary (`parvulus_runtime`)
The binary `parvulus_runtime` can be executed directly from bash or integrated into non-Python robotics controllers.
```bash
./c_engine/parvulus_runtime [options]
```
### Command-Line Options
| Option | Type | Default | Description |
| :--- | :--- | :--- | :--- |
| `--device` | `int` | `0` | Target CUDA GPU device ID |
| `--weights` | `path` | `parvulus_model_weights.bin` | Path to binary model weights |
| `--embeddings` | `path` | `task_embeddings.bin` | Optional lookup table for canonical tasks |
| `--scale` | `path` | `action_scale.bin` | Action scale boundaries |
| `--nfe` | `int` | `10` | Euler ODE flow steps (10 recommended) |
| `--task_vector_file` | `path` | `""` | Load 256-dim task vector from `.bin`, `.npy`, or `.txt` |
| `--task_vector` | `string` | `""` | Comma-separated 256 floats: `"0.12,-0.05,..."` |
| `--task` | `string` | `""` | Task string (resolved via pre-cached table) |
| `--agent_img` | `path` | `""` | Path to raw 256 x 256 x 3 uint8 RGB agent camera frame |
| `--wrist_img` | `path` | `""` | Path to raw 256 x 256 x 3 uint8 RGB wrist camera frame |
| `--output` | `path` | `""` | File path to write predicted 8 x 7 action trajectory |
| `--benchmark` | `int` | `0` | Run N benchmark iterations and print latency breakdown |
### CLI Usage Examples
**1. High-frequency Latency Benchmark:**
```bash
./c_engine/parvulus_runtime --benchmark 100
```
*Output:*
```
[Benchmark] Completed 100 chunks in 727.5 ms
[Benchmark] Mean Latency: 7.275 ms / chunk
[Benchmark] Action Rate: 1099.7 Hz (8 actions / 7.275 ms)
```
**2. Predict Actions from a Custom Task Vector File:**
```bash
./c_engine/parvulus_runtime \
--task_vector_file custom_task.bin \
--agent_img agentview.raw \
--wrist_img wristview.raw \
--output predicted_actions.txt
```
---
## 4. C ABI Interface (`parvulus_c_api.h`)
For embedding into C, C++, ROS 2 nodes, or foreign languages (Rust, Go, C#), include `parvulus_c_api.h`:
```c
#include "parvulus_c_api.h"
// 1. Initialize engine on GPU 0
ParvulusHandle engine = parvulus_create(0);
// 2. Load model weights and action normalizer
parvulus_load(engine, "parvulus_model_weights.bin", "task_embeddings.bin", "action_scale.bin");
// 3. Set episode task conditioning vector (256 floats)
float task_vector[256];
load_my_task_vector(task_vector);
parvulus_set_task_vector(engine, task_vector);
// 4. In robot real-time loop:
uint8_t agentview_rgb[256 * 256 * 3];
uint8_t wristview_rgb[256 * 256 * 3];
float action_chunk[8 * 7]; // 8 timesteps x 7-DoF actions
// Pass NULL for task_description to use the active task vector in GPU memory
parvulus_predict_chunk(engine, agentview_rgb, wristview_rgb, NULL, action_chunk, 10, NULL);
// 5. Cleanup
parvulus_destroy(engine);
```
### Exported Functions
```c
// Create / Destroy
ParvulusHandle parvulus_create(int device_id);
void parvulus_destroy(ParvulusHandle handle);
// Model Weights Loading
bool parvulus_load(ParvulusHandle handle, const char* weights_path, const char* embeddings_path, const char* scale_path);
// Task Conditioning (Vector-First)
bool parvulus_set_task_vector(ParvulusHandle handle, const float* task_vector_256);
bool parvulus_set_task_string(ParvulusHandle handle, const char* task_description);
bool parvulus_register_task_embedding(ParvulusHandle handle, const char* task_description, const float* embedding_256);
// Chunk Prediction
bool parvulus_predict_chunk(ParvulusHandle handle, const uint8_t* agentview_rgb, const uint8_t* wristview_rgb,
const char* task_description, float* out_actions_56, int num_euler_steps,
const float* custom_z0);
bool parvulus_predict_chunk_with_vector(ParvulusHandle handle, const uint8_t* agentview_rgb, const uint8_t* wristview_rgb,
const float* task_vector_256, float* out_actions_56, int num_euler_steps,
const float* custom_z0);
```
---
## 5. Python Integration
### Quickstart with `ParvulusRuntime`
```python
import numpy as np
from parvulus_runtime import ParvulusRuntime
# Initialize runtime
runtime = ParvulusRuntime(device_id=0, num_euler_steps=10)
# Mode A: Vector-First conditioning (e.g., custom, perturbed, or retrieved embedding)
custom_vector = np.random.randn(256).astype(np.float32)
runtime.reset(task_vector=custom_vector)
# Mode B: Dynamic text instruction (automatically handles perturbations via ONNX)
runtime.reset(task_description="grasp the black bowl and set it onto the plate")
# Real-time robot step loop (Receding Horizon, 136 Hz chunking)
agent_rgb = np.zeros((256, 256, 3), dtype=np.uint8)
wrist_rgb = np.zeros((256, 256, 3), dtype=np.uint8)
# Single 7-DoF step
action = runtime.step(agent_rgb, wrist_rgb)
# Or full 8-step chunk prediction [8, 7]
chunk = runtime.predict_chunk(agent_rgb, wrist_rgb)
```
---
## 6. Language Handling
When using string instructions:
1. If the instruction matches a canonical benchmark task, its embedding is resolved instantly (< 1 Β΅s) from `task_embeddings.bin`.
2. If an instruction differs (e.g., *"grasp the black bowl between the plate and the ramekin"* vs. *"take the black bowl and set it onto the plate"*):
- If `parvulus_text_encoder.onnx` is present (or downloaded from Hugging Face / generated via converter), `ParvulusRuntime` encodes the prompt.
- If ONNX is not generated, `ParvulusRuntime` automatically uses PyTorch (`text_encoder.py`) without failure.
3. Task conditioning is set once per episode (t = 0). The control loop streams only camera frames, completely bypassing any text-processing overhead.
### Converting or Downloading the ONNX Text Encoder
The ONNX text encoder binary is not tracked. To prepare it:
```bash
# Option A: Convert locally from base weights + T5-small
python3 c_engine/export_text_encoder_onnx.py
# Option B: Run automatic runtime preparation (downloads from Hugging Face or converts locally)
python3 prepare_runtime.py [--hf_repo <user/repo>]
```