Upload folder using huggingface_hub
Browse files- README.md +15 -15
- c_engine/README.md +8 -8
README.md
CHANGED
|
@@ -23,19 +23,19 @@ pipeline_tag: robotics
|
|
| 23 |
## 1. Highlights
|
| 24 |
|
| 25 |
- **9.98M parameters** (~20 MB in BF16/FP16, ~38 MB in FP32) for standalone robot control.
|
| 26 |
-
- **90.88%
|
| 27 |
- Pinned `google-t5/t5-small` text representations with a trained 2-layer projection head.
|
| 28 |
- **Multiple Execution Modes**:
|
| 29 |
-
- **PyTorch KV-Cached**:
|
| 30 |
-
- **Standalone C++ / CUDA Engine**:
|
| 31 |
-
- **Intel Iris Xe Hybrid**:
|
| 32 |
- Includes optional auxiliary training decoders (`parvulus_auxiliary.safetensors`, 29.15M params) for auto-regressive visual self-rollout, motion change estimation, and self-reconstruction from noisy images.
|
| 33 |
|
| 34 |
---
|
| 35 |
|
| 36 |
## 2. Benchmark Performance
|
| 37 |
|
| 38 |
-
Evaluated across the 4 canonical LIBERO suites (
|
| 39 |
|
| 40 |
| Benchmark Suite | Canonical Tasks | Rollout Trials | Success Rate |
|
| 41 |
| :--- | :---: | :---: | :---: |
|
|
@@ -43,7 +43,7 @@ Evaluated across the 4 canonical LIBERO suites ($N=6,000$ physical rollouts, 10
|
|
| 43 |
| **LIBERO-Object** | 10 | 1,500 | **91.13%** |
|
| 44 |
| **LIBERO-Goal** | 10 | 1,500 | **89.47%** |
|
| 45 |
| **LIBERO-10 (Long)** | 10 | 1,500 | **90.27%** |
|
| 46 |
-
| **Full Benchmark Average** | **40** | **6,000** | **90.88%
|
| 47 |
|
| 48 |
---
|
| 49 |
|
|
@@ -133,14 +133,14 @@ while not done:
|
|
| 133 |
wristview_img = obs["robot0_eye_in_hand_image"]
|
| 134 |
```
|
| 135 |
|
| 136 |
-
### 3. Continuous Overlapped Pipelined Execution (
|
| 137 |
|
| 138 |
To achieve zero-latency continuous robot motion, Parvulus supports overlapping chunk generation with robot motion execution:
|
| 139 |
-
1. Predict an 8-step chunk (
|
| 140 |
-
2. Execute the first 4 actions (
|
| 141 |
-
3. At step 4, capture a fresh observation and trigger background inference for the next chunk (
|
| 142 |
-
4. Concurrently execute the remaining tail actions (
|
| 143 |
-
5. Transition seamlessly into
|
| 144 |
|
| 145 |
```python
|
| 146 |
from concurrent.futures import ThreadPoolExecutor
|
|
@@ -270,11 +270,11 @@ In `hybrid` mode:
|
|
| 270 |
## 7. Model Details & Pinned Dependencies
|
| 271 |
|
| 272 |
- **Input Specifications**:
|
| 273 |
-
- `agentview`:
|
| 274 |
-
- `wristview`:
|
| 275 |
- `instruction`: Natural language string.
|
| 276 |
- **Output Action Format**:
|
| 277 |
-
- 8-step continuous trajectory
|
| 278 |
- **Pinned Text Model**:
|
| 279 |
- `google-t5/t5-small` pinned to commit revision `df1b051c49625cf57a3d0d8d3863ed4d13564fe4`.
|
| 280 |
|
|
|
|
| 23 |
## 1. Highlights
|
| 24 |
|
| 25 |
- **9.98M parameters** (~20 MB in BF16/FP16, ~38 MB in FP32) for standalone robot control.
|
| 26 |
+
- **90.88% \\(\pm\\) 0.37%** average success across all 40 canonical LIBERO tasks (\\(N=6,000\\) simulation rollouts).
|
| 27 |
- Pinned `google-t5/t5-small` text representations with a trained 2-layer projection head.
|
| 28 |
- **Multiple Execution Modes**:
|
| 29 |
+
- **PyTorch KV-Cached**: \\(\approx 38.8\\,\\text{ms}\\) (\\(206\\,\\text{Hz}\\) action bandwidth).
|
| 30 |
+
- **Standalone C++ / CUDA Engine**: \\(\\mathbf{7.28\\,\\text{ms}}\\) (\\(\\gt 1,000\\,\\text{Hz}\\) action bandwidth, pure CUDA/cuBLAS).
|
| 31 |
+
- **Intel Iris Xe Hybrid**: \\(\approx 106.6\\,\\text{ms}\\) on low-power laptop integrated graphics (`xpu` vision + CPU ODE).
|
| 32 |
- Includes optional auxiliary training decoders (`parvulus_auxiliary.safetensors`, 29.15M params) for auto-regressive visual self-rollout, motion change estimation, and self-reconstruction from noisy images.
|
| 33 |
|
| 34 |
---
|
| 35 |
|
| 36 |
## 2. Benchmark Performance
|
| 37 |
|
| 38 |
+
Evaluated across the 4 canonical LIBERO suites (\\(N=6,000\\) physical rollouts, 10 seeds, receding-horizon chunking \\(H=8, C=8\\)):
|
| 39 |
|
| 40 |
| Benchmark Suite | Canonical Tasks | Rollout Trials | Success Rate |
|
| 41 |
| :--- | :---: | :---: | :---: |
|
|
|
|
| 43 |
| **LIBERO-Object** | 10 | 1,500 | **91.13%** |
|
| 44 |
| **LIBERO-Goal** | 10 | 1,500 | **89.47%** |
|
| 45 |
| **LIBERO-10 (Long)** | 10 | 1,500 | **90.27%** |
|
| 46 |
+
| **Full Benchmark Average** | **40** | **6,000** | **90.88% \\(\pm\\) 0.37%** |
|
| 47 |
|
| 48 |
---
|
| 49 |
|
|
|
|
| 133 |
wristview_img = obs["robot0_eye_in_hand_image"]
|
| 134 |
```
|
| 135 |
|
| 136 |
+
### 3. Continuous Overlapped Pipelined Execution (\\(H=8, C=4\\))
|
| 137 |
|
| 138 |
To achieve zero-latency continuous robot motion, Parvulus supports overlapping chunk generation with robot motion execution:
|
| 139 |
+
1. Predict an 8-step chunk (\\(A_0\\)).
|
| 140 |
+
2. Execute the first 4 actions (\\(A_0[0:4]\\)).
|
| 141 |
+
3. At step 4, capture a fresh observation and trigger background inference for the next chunk (\\(A_1\\)).
|
| 142 |
+
4. Concurrently execute the remaining tail actions (\\(A_0[4:8]\\)) on the robot while the model predicts \\(A_1\\).
|
| 143 |
+
5. Transition seamlessly into \\(A_1\\) with zero robot idling.
|
| 144 |
|
| 145 |
```python
|
| 146 |
from concurrent.futures import ThreadPoolExecutor
|
|
|
|
| 270 |
## 7. Model Details & Pinned Dependencies
|
| 271 |
|
| 272 |
- **Input Specifications**:
|
| 273 |
+
- `agentview`: \\([256, 256, 3]\\) RGB.
|
| 274 |
+
- `wristview`: \\([256, 256, 3]\\) RGB.
|
| 275 |
- `instruction`: Natural language string.
|
| 276 |
- **Output Action Format**:
|
| 277 |
+
- 8-step continuous trajectory \\(\\mathbf{A}_t \\in \\mathbb{R}^{8 \\times 7}\\) (\\(dx, dy, dz, d\\theta_x, d\\theta_y, d\\theta_z, \\text{gripper}\\)).
|
| 278 |
- **Pinned Text Model**:
|
| 279 |
- `google-t5/t5-small` pinned to commit revision `df1b051c49625cf57a3d0d8d3863ed4d13564fe4`.
|
| 280 |
|
c_engine/README.md
CHANGED
|
@@ -3,8 +3,8 @@
|
|
| 3 |
High-performance, pure CUDA/C++ inference runtime for the **PARVULUS** 10M-parameter Flow Matching World Action Model.
|
| 4 |
|
| 5 |
- Built solely with `nvcc`, `g++`, `libcublas`, and `libcudart`.
|
| 6 |
-
- Policy natively accepts 256-dimensional continuous conditioning vectors (
|
| 7 |
-
- **
|
| 8 |
- Direct C ABI shared library (`libparvulus.so`), standalone CLI binary (`parvulus_runtime`), and zero-overhead Python bindings.
|
| 9 |
|
| 10 |
---
|
|
@@ -90,10 +90,10 @@ The binary `parvulus_runtime` can be executed directly from bash or integrated i
|
|
| 90 |
| `--task_vector_file` | `path` | `""` | Load 256-dim task vector from `.bin`, `.npy`, or `.txt` |
|
| 91 |
| `--task_vector` | `string` | `""` | Comma-separated 256 floats: `"0.12,-0.05,..."` |
|
| 92 |
| `--task` | `string` | `""` | Task string (resolved via pre-cached table) |
|
| 93 |
-
| `--agent_img` | `path` | `""` | Path to raw
|
| 94 |
-
| `--wrist_img` | `path` | `""` | Path to raw
|
| 95 |
-
| `--output` | `path` | `""` | File path to write predicted
|
| 96 |
-
| `--benchmark` | `int` | `0` | Run
|
| 97 |
|
| 98 |
### CLI Usage Examples
|
| 99 |
|
|
@@ -210,11 +210,11 @@ chunk = runtime.predict_chunk(agent_rgb, wrist_rgb)
|
|
| 210 |
## 6. Language Handling
|
| 211 |
|
| 212 |
When using string instructions:
|
| 213 |
-
1. If the instruction matches a canonical benchmark task, its embedding is resolved instantly (
|
| 214 |
2. If an instruction differs (e.g., *"grasp the black bowl between the plate and the ramekin"* vs. *"take the black bowl and set it onto the plate"*):
|
| 215 |
- If `parvulus_text_encoder.onnx` is present (or downloaded from Hugging Face / generated via converter), `ParvulusRuntime` encodes the prompt.
|
| 216 |
- If ONNX is not generated, `ParvulusRuntime` automatically uses PyTorch (`text_encoder.py`) without failure.
|
| 217 |
-
3. Task conditioning is set once per episode (
|
| 218 |
|
| 219 |
### Converting or Downloading the ONNX Text Encoder
|
| 220 |
The ONNX text encoder binary is not tracked. To prepare it:
|
|
|
|
| 3 |
High-performance, pure CUDA/C++ inference runtime for the **PARVULUS** 10M-parameter Flow Matching World Action Model.
|
| 4 |
|
| 5 |
- Built solely with `nvcc`, `g++`, `libcublas`, and `libcudart`.
|
| 6 |
+
- Policy natively accepts 256-dimensional continuous conditioning vectors (\\([256]\\text{ float32}\\)). Needs to be evaluated once per episode or can be modified asynchronously.
|
| 7 |
+
- **\\(\\mathbf{\\lt 8\\,\\text{ms}}\\) per 8-step action chunk** (~\\(\\gt 125\\,\\text{Hz}\\) chunk rate, \\(\\gt 1,000\\,\\text{Hz}\\) action bandwidth).
|
| 8 |
- Direct C ABI shared library (`libparvulus.so`), standalone CLI binary (`parvulus_runtime`), and zero-overhead Python bindings.
|
| 9 |
|
| 10 |
---
|
|
|
|
| 90 |
| `--task_vector_file` | `path` | `""` | Load 256-dim task vector from `.bin`, `.npy`, or `.txt` |
|
| 91 |
| `--task_vector` | `string` | `""` | Comma-separated 256 floats: `"0.12,-0.05,..."` |
|
| 92 |
| `--task` | `string` | `""` | Task string (resolved via pre-cached table) |
|
| 93 |
+
| `--agent_img` | `path` | `""` | Path to raw \\(256 \\times 256 \\times 3\\) uint8 RGB agent camera frame |
|
| 94 |
+
| `--wrist_img` | `path` | `""` | Path to raw \\(256 \\times 256 \\times 3\\) uint8 RGB wrist camera frame |
|
| 95 |
+
| `--output` | `path` | `""` | File path to write predicted \\(8 \\times 7\\) action trajectory |
|
| 96 |
+
| `--benchmark` | `int` | `0` | Run \\(N\\) benchmark iterations and print latency breakdown |
|
| 97 |
|
| 98 |
### CLI Usage Examples
|
| 99 |
|
|
|
|
| 210 |
## 6. Language Handling
|
| 211 |
|
| 212 |
When using string instructions:
|
| 213 |
+
1. If the instruction matches a canonical benchmark task, its embedding is resolved instantly (\\(\\lt 1\\,\\mu\\text{s}\\)) from `task_embeddings.bin`.
|
| 214 |
2. If an instruction differs (e.g., *"grasp the black bowl between the plate and the ramekin"* vs. *"take the black bowl and set it onto the plate"*):
|
| 215 |
- If `parvulus_text_encoder.onnx` is present (or downloaded from Hugging Face / generated via converter), `ParvulusRuntime` encodes the prompt.
|
| 216 |
- If ONNX is not generated, `ParvulusRuntime` automatically uses PyTorch (`text_encoder.py`) without failure.
|
| 217 |
+
3. Task conditioning is set once per episode (\\(t=0\\)). The control loop streams only camera frames, completely bypassing any text-processing overhead.
|
| 218 |
|
| 219 |
### Converting or Downloading the ONNX Text Encoder
|
| 220 |
The ONNX text encoder binary is not tracked. To prepare it:
|