File size: 9,028 Bytes
67c4f10
 
 
 
 
f98e714
 
67c4f10
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
f98e714
 
 
 
67c4f10
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
f98e714
67c4f10
 
 
f98e714
67c4f10
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
# PARVULUS Standalone Native Runtime Engine (C++ / CUDA)

High-performance, pure CUDA/C++ inference runtime for the **PARVULUS** 10M-parameter Flow Matching World Action Model.

- Built solely with `nvcc`, `g++`, `libcublas`, and `libcudart`.
- Policy natively accepts 256-dimensional continuous conditioning vectors ([256] float32). Needs to be evaluated once per episode or can be modified asynchronously.
- **< 8 ms per 8-step action chunk** (~>125 Hz chunk rate, >1,000 Hz action bandwidth).
- Direct C ABI shared library (`libparvulus.so`), standalone CLI binary (`parvulus_runtime`), and zero-overhead Python bindings.

---

## 1. Directory Structure

```
c_engine/
β”œβ”€β”€ Makefile                     # Make-based build system for library, CLI, and test binaries
β”œβ”€β”€ build.sh                     # Build script (runs make clean && make -j)
β”‚
β”œβ”€β”€ parvulus_engine.h            # Core C++ ParvulusEngine class header
β”œβ”€β”€ parvulus_engine.cu           # Pure CUDA/cuBLAS implementation (Euler ODE loop, cross-attention)
β”œβ”€β”€ vision_encoder.h             # DualView visual backbone header
β”œβ”€β”€ vision_encoder.cuh           # DualView CUDA kernels (patch embed, depthwise-separable convs)
β”œβ”€β”€ vision_encoder.cu            # DualView CUDA implementation
β”œβ”€β”€ kernels.cuh                  # Fused CUDA kernels (LayerNorm, SiLU, Softmax, Add, Unscale)
β”‚
β”œβ”€β”€ parvulus_c_api.h             # Standard C ABI header for external language bindings / middleware
β”œβ”€β”€ parvulus_c_api.cpp           # C ABI export implementation
β”‚
β”œβ”€β”€ parvulus_runtime_cli.cpp     # Standalone CLI binary entry point
β”œβ”€β”€ test_vision_encoder.cpp      # Standalone numerical parity test for vision backbone
β”œβ”€β”€ infer_c.py                   # Python ctypes wrapper
β”‚
β”œβ”€β”€ export_from_safetensors.py   # Serializes .safetensors into binary weight files
β”œβ”€β”€ export_text_encoder_onnx.py  # Standalone converter for T5 + projection head to ONNX
β”œβ”€β”€ parvulus_model_weights.bin   # Exported FP32 weights (37.28 MB, 273 tensors)
β”œβ”€β”€ task_embeddings.bin          # Pre-cached 256-dim embeddings for canonical tasks
└── action_scale.bin             # 7-DoF min/max action normalization bounds
```

---

## 2. Requirements & Building

### Prerequisites
- **CUDA Toolkit** (11.8+ or 12.0+) with `nvcc`, `libcudart`, `libcublas`
- **C++17 Compiler** (`g++` 9.0+)
- *Python or PyTorch runtime are not needed for C/C++ usage.*

### Build Targets

To build the shared library `libparvulus.so`, standalone CLI binary `parvulus_runtime`, and test binary:

```bash
cd c_engine
make -j$(nproc)
```

Or run the build script:
```bash
./build.sh
```

### Run Sanity Verification Tests & Benchmark:
```bash
make test
```
This runs:
1. `test_vision_encoder`: Verifies DualView vision backbone parity and reports perception latency.
2. `parvulus_runtime --benchmark 20`: Runs a 20-chunk end-to-end benchmark and verifies end-to-end latency.

---

## 3. Standalone CLI Binary (`parvulus_runtime`)

The binary `parvulus_runtime` can be executed directly from bash or integrated into non-Python robotics controllers.

```bash
./c_engine/parvulus_runtime [options]
```

### Command-Line Options

| Option | Type | Default | Description |
| :--- | :--- | :--- | :--- |
| `--device` | `int` | `0` | Target CUDA GPU device ID |
| `--weights` | `path` | `parvulus_model_weights.bin` | Path to binary model weights |
| `--embeddings` | `path` | `task_embeddings.bin` | Optional lookup table for canonical tasks |
| `--scale` | `path` | `action_scale.bin` | Action scale boundaries |
| `--nfe` | `int` | `10` | Euler ODE flow steps (10 recommended) |
| `--task_vector_file` | `path` | `""` | Load 256-dim task vector from `.bin`, `.npy`, or `.txt` |
| `--task_vector` | `string` | `""` | Comma-separated 256 floats: `"0.12,-0.05,..."` |
| `--task` | `string` | `""` | Task string (resolved via pre-cached table) |
| `--agent_img` | `path` | `""` | Path to raw 256 x 256 x 3 uint8 RGB agent camera frame |
| `--wrist_img` | `path` | `""` | Path to raw 256 x 256 x 3 uint8 RGB wrist camera frame |
| `--output` | `path` | `""` | File path to write predicted 8 x 7 action trajectory |
| `--benchmark` | `int` | `0` | Run N benchmark iterations and print latency breakdown |

### CLI Usage Examples

**1. High-frequency Latency Benchmark:**
```bash
./c_engine/parvulus_runtime --benchmark 100
```
*Output:*
```
[Benchmark] Completed 100 chunks in 727.5 ms
[Benchmark] Mean Latency: 7.275 ms / chunk
[Benchmark] Action Rate:  1099.7 Hz (8 actions / 7.275 ms)
```

**2. Predict Actions from a Custom Task Vector File:**
```bash
./c_engine/parvulus_runtime \
  --task_vector_file custom_task.bin \
  --agent_img agentview.raw \
  --wrist_img wristview.raw \
  --output predicted_actions.txt
```

---

## 4. C ABI Interface (`parvulus_c_api.h`)

For embedding into C, C++, ROS 2 nodes, or foreign languages (Rust, Go, C#), include `parvulus_c_api.h`:

```c
#include "parvulus_c_api.h"

// 1. Initialize engine on GPU 0
ParvulusHandle engine = parvulus_create(0);

// 2. Load model weights and action normalizer
parvulus_load(engine, "parvulus_model_weights.bin", "task_embeddings.bin", "action_scale.bin");

// 3. Set episode task conditioning vector (256 floats)
float task_vector[256];
load_my_task_vector(task_vector);
parvulus_set_task_vector(engine, task_vector);

// 4. In robot real-time loop:
uint8_t agentview_rgb[256 * 256 * 3];
uint8_t wristview_rgb[256 * 256 * 3];
float action_chunk[8 * 7]; // 8 timesteps x 7-DoF actions

// Pass NULL for task_description to use the active task vector in GPU memory
parvulus_predict_chunk(engine, agentview_rgb, wristview_rgb, NULL, action_chunk, 10, NULL);

// 5. Cleanup
parvulus_destroy(engine);
```

### Exported Functions

```c
// Create / Destroy
ParvulusHandle parvulus_create(int device_id);
void parvulus_destroy(ParvulusHandle handle);

// Model Weights Loading
bool parvulus_load(ParvulusHandle handle, const char* weights_path, const char* embeddings_path, const char* scale_path);

// Task Conditioning (Vector-First)
bool parvulus_set_task_vector(ParvulusHandle handle, const float* task_vector_256);
bool parvulus_set_task_string(ParvulusHandle handle, const char* task_description);
bool parvulus_register_task_embedding(ParvulusHandle handle, const char* task_description, const float* embedding_256);

// Chunk Prediction
bool parvulus_predict_chunk(ParvulusHandle handle, const uint8_t* agentview_rgb, const uint8_t* wristview_rgb,
                            const char* task_description, float* out_actions_56, int num_euler_steps,
                            const float* custom_z0);

bool parvulus_predict_chunk_with_vector(ParvulusHandle handle, const uint8_t* agentview_rgb, const uint8_t* wristview_rgb,
                                       const float* task_vector_256, float* out_actions_56, int num_euler_steps,
                                       const float* custom_z0);
```

---

## 5. Python Integration

### Quickstart with `ParvulusRuntime`

```python
import numpy as np
from parvulus_runtime import ParvulusRuntime

# Initialize runtime
runtime = ParvulusRuntime(device_id=0, num_euler_steps=10)

# Mode A: Vector-First conditioning (e.g., custom, perturbed, or retrieved embedding)
custom_vector = np.random.randn(256).astype(np.float32)
runtime.reset(task_vector=custom_vector)

# Mode B: Dynamic text instruction (automatically handles perturbations via ONNX)
runtime.reset(task_description="grasp the black bowl and set it onto the plate")

# Real-time robot step loop (Receding Horizon, 136 Hz chunking)
agent_rgb = np.zeros((256, 256, 3), dtype=np.uint8)
wrist_rgb = np.zeros((256, 256, 3), dtype=np.uint8)

# Single 7-DoF step
action = runtime.step(agent_rgb, wrist_rgb)

# Or full 8-step chunk prediction [8, 7]
chunk = runtime.predict_chunk(agent_rgb, wrist_rgb)
```

---

## 6. Language Handling

When using string instructions:
1. If the instruction matches a canonical benchmark task, its embedding is resolved instantly (< 1 Β΅s) from `task_embeddings.bin`.
2. If an instruction differs (e.g., *"grasp the black bowl between the plate and the ramekin"* vs. *"take the black bowl and set it onto the plate"*):
   - If `parvulus_text_encoder.onnx` is present (or downloaded from Hugging Face / generated via converter), `ParvulusRuntime` encodes the prompt.
   - If ONNX is not generated, `ParvulusRuntime` automatically uses PyTorch (`text_encoder.py`) without failure.
3. Task conditioning is set once per episode (t = 0). The control loop streams only camera frames, completely bypassing any text-processing overhead.

### Converting or Downloading the ONNX Text Encoder
The ONNX text encoder binary is not tracked. To prepare it:
```bash
# Option A: Convert locally from base weights + T5-small
python3 c_engine/export_text_encoder_onnx.py

# Option B: Run automatic runtime preparation (downloads from Hugging Face or converts locally)
python3 prepare_runtime.py [--hf_repo <user/repo>]
```