Upload folder using huggingface_hub
Browse files- README.md +15 -15
- c_engine/README.md +8 -8
README.md
CHANGED
|
@@ -23,19 +23,19 @@ pipeline_tag: robotics
|
|
| 23 |
## 1. Highlights
|
| 24 |
|
| 25 |
- **9.98M parameters** (~20 MB in BF16/FP16, ~38 MB in FP32) for standalone robot control.
|
| 26 |
-
- **90.88%
|
| 27 |
- Pinned `google-t5/t5-small` text representations with a trained 2-layer projection head.
|
| 28 |
- **Multiple Execution Modes**:
|
| 29 |
-
- **PyTorch KV-Cached**:
|
| 30 |
-
- **Standalone C++ / CUDA Engine**:
|
| 31 |
-
- **Intel Iris Xe Hybrid**:
|
| 32 |
- Includes optional auxiliary training decoders (`parvulus_auxiliary.safetensors`, 29.15M params) for auto-regressive visual self-rollout, motion change estimation, and self-reconstruction from noisy images.
|
| 33 |
|
| 34 |
---
|
| 35 |
|
| 36 |
## 2. Benchmark Performance
|
| 37 |
|
| 38 |
-
Evaluated across the 4 canonical LIBERO suites (
|
| 39 |
|
| 40 |
| Benchmark Suite | Canonical Tasks | Rollout Trials | Success Rate |
|
| 41 |
| :--- | :---: | :---: | :---: |
|
|
@@ -43,7 +43,7 @@ Evaluated across the 4 canonical LIBERO suites (\\(N=6,000\\) physical rollouts,
|
|
| 43 |
| **LIBERO-Object** | 10 | 1,500 | **91.13%** |
|
| 44 |
| **LIBERO-Goal** | 10 | 1,500 | **89.47%** |
|
| 45 |
| **LIBERO-10 (Long)** | 10 | 1,500 | **90.27%** |
|
| 46 |
-
| **Full Benchmark Average** | **40** | **6,000** | **90.88%
|
| 47 |
|
| 48 |
---
|
| 49 |
|
|
@@ -133,14 +133,14 @@ while not done:
|
|
| 133 |
wristview_img = obs["robot0_eye_in_hand_image"]
|
| 134 |
```
|
| 135 |
|
| 136 |
-
### 3. Continuous Overlapped Pipelined Execution (
|
| 137 |
|
| 138 |
To achieve zero-latency continuous robot motion, Parvulus supports overlapping chunk generation with robot motion execution:
|
| 139 |
-
1. Predict an 8-step chunk (
|
| 140 |
-
2. Execute the first 4 actions (
|
| 141 |
-
3. At step 4, capture a fresh observation and trigger background inference for the next chunk (
|
| 142 |
-
4. Concurrently execute the remaining tail actions (
|
| 143 |
-
5. Transition seamlessly into
|
| 144 |
|
| 145 |
```python
|
| 146 |
from concurrent.futures import ThreadPoolExecutor
|
|
@@ -270,11 +270,11 @@ In `hybrid` mode:
|
|
| 270 |
## 7. Model Details & Pinned Dependencies
|
| 271 |
|
| 272 |
- **Input Specifications**:
|
| 273 |
-
- `agentview`:
|
| 274 |
-
- `wristview`:
|
| 275 |
- `instruction`: Natural language string.
|
| 276 |
- **Output Action Format**:
|
| 277 |
-
- 8-step continuous trajectory
|
| 278 |
- **Pinned Text Model**:
|
| 279 |
- `google-t5/t5-small` pinned to commit revision `df1b051c49625cf57a3d0d8d3863ed4d13564fe4`.
|
| 280 |
|
|
|
|
| 23 |
## 1. Highlights
|
| 24 |
|
| 25 |
- **9.98M parameters** (~20 MB in BF16/FP16, ~38 MB in FP32) for standalone robot control.
|
| 26 |
+
- **90.88% ± 0.37%** average success across all 40 canonical LIBERO tasks (N = 6,000 simulation rollouts).
|
| 27 |
- Pinned `google-t5/t5-small` text representations with a trained 2-layer projection head.
|
| 28 |
- **Multiple Execution Modes**:
|
| 29 |
+
- **PyTorch KV-Cached**: ≈ 38.8 ms (206 Hz action bandwidth).
|
| 30 |
+
- **Standalone C++ / CUDA Engine**: **7.28 ms** (>1,000 Hz action bandwidth, pure CUDA/cuBLAS).
|
| 31 |
+
- **Intel Iris Xe Hybrid**: ≈ 106.6 ms on low-power laptop integrated graphics (`xpu` vision + CPU ODE).
|
| 32 |
- Includes optional auxiliary training decoders (`parvulus_auxiliary.safetensors`, 29.15M params) for auto-regressive visual self-rollout, motion change estimation, and self-reconstruction from noisy images.
|
| 33 |
|
| 34 |
---
|
| 35 |
|
| 36 |
## 2. Benchmark Performance
|
| 37 |
|
| 38 |
+
Evaluated across the 4 canonical LIBERO suites (N = 6,000 physical rollouts, 10 seeds, receding-horizon chunking H = 8, C = 8):
|
| 39 |
|
| 40 |
| Benchmark Suite | Canonical Tasks | Rollout Trials | Success Rate |
|
| 41 |
| :--- | :---: | :---: | :---: |
|
|
|
|
| 43 |
| **LIBERO-Object** | 10 | 1,500 | **91.13%** |
|
| 44 |
| **LIBERO-Goal** | 10 | 1,500 | **89.47%** |
|
| 45 |
| **LIBERO-10 (Long)** | 10 | 1,500 | **90.27%** |
|
| 46 |
+
| **Full Benchmark Average** | **40** | **6,000** | **90.88% ± 0.37%** |
|
| 47 |
|
| 48 |
---
|
| 49 |
|
|
|
|
| 133 |
wristview_img = obs["robot0_eye_in_hand_image"]
|
| 134 |
```
|
| 135 |
|
| 136 |
+
### 3. Continuous Overlapped Pipelined Execution (H = 8, C = 4)
|
| 137 |
|
| 138 |
To achieve zero-latency continuous robot motion, Parvulus supports overlapping chunk generation with robot motion execution:
|
| 139 |
+
1. Predict an 8-step chunk (A_0).
|
| 140 |
+
2. Execute the first 4 actions (A_0[0:4]).
|
| 141 |
+
3. At step 4, capture a fresh observation and trigger background inference for the next chunk (A_1).
|
| 142 |
+
4. Concurrently execute the remaining tail actions (A_0[4:8]) on the robot while the model predicts A_1.
|
| 143 |
+
5. Transition seamlessly into A_1 with zero robot idling.
|
| 144 |
|
| 145 |
```python
|
| 146 |
from concurrent.futures import ThreadPoolExecutor
|
|
|
|
| 270 |
## 7. Model Details & Pinned Dependencies
|
| 271 |
|
| 272 |
- **Input Specifications**:
|
| 273 |
+
- `agentview`: [256, 256, 3] RGB.
|
| 274 |
+
- `wristview`: [256, 256, 3] RGB.
|
| 275 |
- `instruction`: Natural language string.
|
| 276 |
- **Output Action Format**:
|
| 277 |
+
- 8-step continuous trajectory A_t [8, 7] (dx, dy, dz, dθx, dθy, dθz, gripper).
|
| 278 |
- **Pinned Text Model**:
|
| 279 |
- `google-t5/t5-small` pinned to commit revision `df1b051c49625cf57a3d0d8d3863ed4d13564fe4`.
|
| 280 |
|
c_engine/README.md
CHANGED
|
@@ -3,8 +3,8 @@
|
|
| 3 |
High-performance, pure CUDA/C++ inference runtime for the **PARVULUS** 10M-parameter Flow Matching World Action Model.
|
| 4 |
|
| 5 |
- Built solely with `nvcc`, `g++`, `libcublas`, and `libcudart`.
|
| 6 |
-
- Policy natively accepts 256-dimensional continuous conditioning vectors (
|
| 7 |
-
- **
|
| 8 |
- Direct C ABI shared library (`libparvulus.so`), standalone CLI binary (`parvulus_runtime`), and zero-overhead Python bindings.
|
| 9 |
|
| 10 |
---
|
|
@@ -90,10 +90,10 @@ The binary `parvulus_runtime` can be executed directly from bash or integrated i
|
|
| 90 |
| `--task_vector_file` | `path` | `""` | Load 256-dim task vector from `.bin`, `.npy`, or `.txt` |
|
| 91 |
| `--task_vector` | `string` | `""` | Comma-separated 256 floats: `"0.12,-0.05,..."` |
|
| 92 |
| `--task` | `string` | `""` | Task string (resolved via pre-cached table) |
|
| 93 |
-
| `--agent_img` | `path` | `""` | Path to raw
|
| 94 |
-
| `--wrist_img` | `path` | `""` | Path to raw
|
| 95 |
-
| `--output` | `path` | `""` | File path to write predicted
|
| 96 |
-
| `--benchmark` | `int` | `0` | Run
|
| 97 |
|
| 98 |
### CLI Usage Examples
|
| 99 |
|
|
@@ -210,11 +210,11 @@ chunk = runtime.predict_chunk(agent_rgb, wrist_rgb)
|
|
| 210 |
## 6. Language Handling
|
| 211 |
|
| 212 |
When using string instructions:
|
| 213 |
-
1. If the instruction matches a canonical benchmark task, its embedding is resolved instantly (
|
| 214 |
2. If an instruction differs (e.g., *"grasp the black bowl between the plate and the ramekin"* vs. *"take the black bowl and set it onto the plate"*):
|
| 215 |
- If `parvulus_text_encoder.onnx` is present (or downloaded from Hugging Face / generated via converter), `ParvulusRuntime` encodes the prompt.
|
| 216 |
- If ONNX is not generated, `ParvulusRuntime` automatically uses PyTorch (`text_encoder.py`) without failure.
|
| 217 |
-
3. Task conditioning is set once per episode (
|
| 218 |
|
| 219 |
### Converting or Downloading the ONNX Text Encoder
|
| 220 |
The ONNX text encoder binary is not tracked. To prepare it:
|
|
|
|
| 3 |
High-performance, pure CUDA/C++ inference runtime for the **PARVULUS** 10M-parameter Flow Matching World Action Model.
|
| 4 |
|
| 5 |
- Built solely with `nvcc`, `g++`, `libcublas`, and `libcudart`.
|
| 6 |
+
- Policy natively accepts 256-dimensional continuous conditioning vectors ([256] float32). Needs to be evaluated once per episode or can be modified asynchronously.
|
| 7 |
+
- **< 8 ms per 8-step action chunk** (~>125 Hz chunk rate, >1,000 Hz action bandwidth).
|
| 8 |
- Direct C ABI shared library (`libparvulus.so`), standalone CLI binary (`parvulus_runtime`), and zero-overhead Python bindings.
|
| 9 |
|
| 10 |
---
|
|
|
|
| 90 |
| `--task_vector_file` | `path` | `""` | Load 256-dim task vector from `.bin`, `.npy`, or `.txt` |
|
| 91 |
| `--task_vector` | `string` | `""` | Comma-separated 256 floats: `"0.12,-0.05,..."` |
|
| 92 |
| `--task` | `string` | `""` | Task string (resolved via pre-cached table) |
|
| 93 |
+
| `--agent_img` | `path` | `""` | Path to raw 256 x 256 x 3 uint8 RGB agent camera frame |
|
| 94 |
+
| `--wrist_img` | `path` | `""` | Path to raw 256 x 256 x 3 uint8 RGB wrist camera frame |
|
| 95 |
+
| `--output` | `path` | `""` | File path to write predicted 8 x 7 action trajectory |
|
| 96 |
+
| `--benchmark` | `int` | `0` | Run N benchmark iterations and print latency breakdown |
|
| 97 |
|
| 98 |
### CLI Usage Examples
|
| 99 |
|
|
|
|
| 210 |
## 6. Language Handling
|
| 211 |
|
| 212 |
When using string instructions:
|
| 213 |
+
1. If the instruction matches a canonical benchmark task, its embedding is resolved instantly (< 1 µs) from `task_embeddings.bin`.
|
| 214 |
2. If an instruction differs (e.g., *"grasp the black bowl between the plate and the ramekin"* vs. *"take the black bowl and set it onto the plate"*):
|
| 215 |
- If `parvulus_text_encoder.onnx` is present (or downloaded from Hugging Face / generated via converter), `ParvulusRuntime` encodes the prompt.
|
| 216 |
- If ONNX is not generated, `ParvulusRuntime` automatically uses PyTorch (`text_encoder.py`) without failure.
|
| 217 |
+
3. Task conditioning is set once per episode (t = 0). The control loop streams only camera frames, completely bypassing any text-processing overhead.
|
| 218 |
|
| 219 |
### Converting or Downloading the ONNX Text Encoder
|
| 220 |
The ONNX text encoder binary is not tracked. To prepare it:
|