aasivas commited on
Commit
66a5ccf
·
verified ·
1 Parent(s): 1871410

Upload folder using huggingface_hub

Browse files
Files changed (2) hide show
  1. README.md +15 -15
  2. c_engine/README.md +8 -8
README.md CHANGED
@@ -23,19 +23,19 @@ pipeline_tag: robotics
23
  ## 1. Highlights
24
 
25
  - **9.98M parameters** (~20 MB in BF16/FP16, ~38 MB in FP32) for standalone robot control.
26
- - **90.88% $\pm$ 0.37%** average success across all 40 canonical LIBERO tasks ($N=6,000$ simulation rollouts).
27
  - Pinned `google-t5/t5-small` text representations with a trained 2-layer projection head.
28
  - **Multiple Execution Modes**:
29
- - **PyTorch KV-Cached**: $\approx 38.8\,\text{ms}$ ($206\,\text{Hz}$ action bandwidth).
30
- - **Standalone C++ / CUDA Engine**: $\mathbf{7.28\,\text{ms}}$ ($>1,000\,\text{Hz}$ action bandwidth, pure CUDA/cuBLAS).
31
- - **Intel Iris Xe Hybrid**: $\approx 106.6\,\text{ms}$ on low-power laptop integrated graphics (`xpu` vision + CPU ODE).
32
  - Includes optional auxiliary training decoders (`parvulus_auxiliary.safetensors`, 29.15M params) for auto-regressive visual self-rollout, motion change estimation, and self-reconstruction from noisy images.
33
 
34
  ---
35
 
36
  ## 2. Benchmark Performance
37
 
38
- Evaluated across the 4 canonical LIBERO suites ($N=6,000$ physical rollouts, 10 seeds, receding-horizon chunking $H=8, C=8$):
39
 
40
  | Benchmark Suite | Canonical Tasks | Rollout Trials | Success Rate |
41
  | :--- | :---: | :---: | :---: |
@@ -43,7 +43,7 @@ Evaluated across the 4 canonical LIBERO suites ($N=6,000$ physical rollouts, 10
43
  | **LIBERO-Object** | 10 | 1,500 | **91.13%** |
44
  | **LIBERO-Goal** | 10 | 1,500 | **89.47%** |
45
  | **LIBERO-10 (Long)** | 10 | 1,500 | **90.27%** |
46
- | **Full Benchmark Average** | **40** | **6,000** | **90.88% $\pm$ 0.37%** |
47
 
48
  ---
49
 
@@ -133,14 +133,14 @@ while not done:
133
  wristview_img = obs["robot0_eye_in_hand_image"]
134
  ```
135
 
136
- ### 3. Continuous Overlapped Pipelined Execution ($H=8, C=4$)
137
 
138
  To achieve zero-latency continuous robot motion, Parvulus supports overlapping chunk generation with robot motion execution:
139
- 1. Predict an 8-step chunk ($A_0$).
140
- 2. Execute the first 4 actions ($A_0[0:4]$).
141
- 3. At step 4, capture a fresh observation and trigger background inference for the next chunk ($A_1$).
142
- 4. Concurrently execute the remaining tail actions ($A_0[4:8]$) on the robot while the model predicts $A_1$.
143
- 5. Transition seamlessly into $A_1$ with zero robot idling.
144
 
145
  ```python
146
  from concurrent.futures import ThreadPoolExecutor
@@ -270,11 +270,11 @@ In `hybrid` mode:
270
  ## 7. Model Details & Pinned Dependencies
271
 
272
  - **Input Specifications**:
273
- - `agentview`: $[256, 256, 3]$ RGB.
274
- - `wristview`: $[256, 256, 3]$ RGB.
275
  - `instruction`: Natural language string.
276
  - **Output Action Format**:
277
- - 8-step continuous trajectory $\mathbf{A}_t \in \mathbb{R}^{8 \times 7}$ ($dx, dy, dz, d\theta_x, d\theta_y, d\theta_z, \text{gripper}$).
278
  - **Pinned Text Model**:
279
  - `google-t5/t5-small` pinned to commit revision `df1b051c49625cf57a3d0d8d3863ed4d13564fe4`.
280
 
 
23
  ## 1. Highlights
24
 
25
  - **9.98M parameters** (~20 MB in BF16/FP16, ~38 MB in FP32) for standalone robot control.
26
+ - **90.88% \\(\pm\\) 0.37%** average success across all 40 canonical LIBERO tasks (\\(N=6,000\\) simulation rollouts).
27
  - Pinned `google-t5/t5-small` text representations with a trained 2-layer projection head.
28
  - **Multiple Execution Modes**:
29
+ - **PyTorch KV-Cached**: \\(\approx 38.8\\,\\text{ms}\\) (\\(206\\,\\text{Hz}\\) action bandwidth).
30
+ - **Standalone C++ / CUDA Engine**: \\(\\mathbf{7.28\\,\\text{ms}}\\) (\\(\\gt 1,000\\,\\text{Hz}\\) action bandwidth, pure CUDA/cuBLAS).
31
+ - **Intel Iris Xe Hybrid**: \\(\approx 106.6\\,\\text{ms}\\) on low-power laptop integrated graphics (`xpu` vision + CPU ODE).
32
  - Includes optional auxiliary training decoders (`parvulus_auxiliary.safetensors`, 29.15M params) for auto-regressive visual self-rollout, motion change estimation, and self-reconstruction from noisy images.
33
 
34
  ---
35
 
36
  ## 2. Benchmark Performance
37
 
38
+ Evaluated across the 4 canonical LIBERO suites (\\(N=6,000\\) physical rollouts, 10 seeds, receding-horizon chunking \\(H=8, C=8\\)):
39
 
40
  | Benchmark Suite | Canonical Tasks | Rollout Trials | Success Rate |
41
  | :--- | :---: | :---: | :---: |
 
43
  | **LIBERO-Object** | 10 | 1,500 | **91.13%** |
44
  | **LIBERO-Goal** | 10 | 1,500 | **89.47%** |
45
  | **LIBERO-10 (Long)** | 10 | 1,500 | **90.27%** |
46
+ | **Full Benchmark Average** | **40** | **6,000** | **90.88% \\(\pm\\) 0.37%** |
47
 
48
  ---
49
 
 
133
  wristview_img = obs["robot0_eye_in_hand_image"]
134
  ```
135
 
136
+ ### 3. Continuous Overlapped Pipelined Execution (\\(H=8, C=4\\))
137
 
138
  To achieve zero-latency continuous robot motion, Parvulus supports overlapping chunk generation with robot motion execution:
139
+ 1. Predict an 8-step chunk (\\(A_0\\)).
140
+ 2. Execute the first 4 actions (\\(A_0[0:4]\\)).
141
+ 3. At step 4, capture a fresh observation and trigger background inference for the next chunk (\\(A_1\\)).
142
+ 4. Concurrently execute the remaining tail actions (\\(A_0[4:8]\\)) on the robot while the model predicts \\(A_1\\).
143
+ 5. Transition seamlessly into \\(A_1\\) with zero robot idling.
144
 
145
  ```python
146
  from concurrent.futures import ThreadPoolExecutor
 
270
  ## 7. Model Details & Pinned Dependencies
271
 
272
  - **Input Specifications**:
273
+ - `agentview`: \\([256, 256, 3]\\) RGB.
274
+ - `wristview`: \\([256, 256, 3]\\) RGB.
275
  - `instruction`: Natural language string.
276
  - **Output Action Format**:
277
+ - 8-step continuous trajectory \\(\\mathbf{A}_t \\in \\mathbb{R}^{8 \\times 7}\\) (\\(dx, dy, dz, d\\theta_x, d\\theta_y, d\\theta_z, \\text{gripper}\\)).
278
  - **Pinned Text Model**:
279
  - `google-t5/t5-small` pinned to commit revision `df1b051c49625cf57a3d0d8d3863ed4d13564fe4`.
280
 
c_engine/README.md CHANGED
@@ -3,8 +3,8 @@
3
  High-performance, pure CUDA/C++ inference runtime for the **PARVULUS** 10M-parameter Flow Matching World Action Model.
4
 
5
  - Built solely with `nvcc`, `g++`, `libcublas`, and `libcudart`.
6
- - Policy natively accepts 256-dimensional continuous conditioning vectors ($[256]\text{ float32}$). Needs to be evaluated once per episode or can be modified asynchronously.
7
- - **$\mathbf{\lt 8\,\text{ms}}$ per 8-step action chunk** (~$\gt 125\,\text{Hz}$ chunk rate, $>1,000\,\text{Hz}$ action bandwidth).
8
  - Direct C ABI shared library (`libparvulus.so`), standalone CLI binary (`parvulus_runtime`), and zero-overhead Python bindings.
9
 
10
  ---
@@ -90,10 +90,10 @@ The binary `parvulus_runtime` can be executed directly from bash or integrated i
90
  | `--task_vector_file` | `path` | `""` | Load 256-dim task vector from `.bin`, `.npy`, or `.txt` |
91
  | `--task_vector` | `string` | `""` | Comma-separated 256 floats: `"0.12,-0.05,..."` |
92
  | `--task` | `string` | `""` | Task string (resolved via pre-cached table) |
93
- | `--agent_img` | `path` | `""` | Path to raw $256 \times 256 \times 3$ uint8 RGB agent camera frame |
94
- | `--wrist_img` | `path` | `""` | Path to raw $256 \times 256 \times 3$ uint8 RGB wrist camera frame |
95
- | `--output` | `path` | `""` | File path to write predicted $8 \times 7$ action trajectory |
96
- | `--benchmark` | `int` | `0` | Run $N$ benchmark iterations and print latency breakdown |
97
 
98
  ### CLI Usage Examples
99
 
@@ -210,11 +210,11 @@ chunk = runtime.predict_chunk(agent_rgb, wrist_rgb)
210
  ## 6. Language Handling
211
 
212
  When using string instructions:
213
- 1. If the instruction matches a canonical benchmark task, its embedding is resolved instantly ($<1\,\mu\text{s}$) from `task_embeddings.bin`.
214
  2. If an instruction differs (e.g., *"grasp the black bowl between the plate and the ramekin"* vs. *"take the black bowl and set it onto the plate"*):
215
  - If `parvulus_text_encoder.onnx` is present (or downloaded from Hugging Face / generated via converter), `ParvulusRuntime` encodes the prompt.
216
  - If ONNX is not generated, `ParvulusRuntime` automatically uses PyTorch (`text_encoder.py`) without failure.
217
- 3. Task conditioning is set once per episode ($t=0$). The control loop streams only camera frames, completely bypassing any text-processing overhead.
218
 
219
  ### Converting or Downloading the ONNX Text Encoder
220
  The ONNX text encoder binary is not tracked. To prepare it:
 
3
  High-performance, pure CUDA/C++ inference runtime for the **PARVULUS** 10M-parameter Flow Matching World Action Model.
4
 
5
  - Built solely with `nvcc`, `g++`, `libcublas`, and `libcudart`.
6
+ - Policy natively accepts 256-dimensional continuous conditioning vectors (\\([256]\\text{ float32}\\)). Needs to be evaluated once per episode or can be modified asynchronously.
7
+ - **\\(\\mathbf{\\lt 8\\,\\text{ms}}\\) per 8-step action chunk** (~\\(\\gt 125\\,\\text{Hz}\\) chunk rate, \\(\\gt 1,000\\,\\text{Hz}\\) action bandwidth).
8
  - Direct C ABI shared library (`libparvulus.so`), standalone CLI binary (`parvulus_runtime`), and zero-overhead Python bindings.
9
 
10
  ---
 
90
  | `--task_vector_file` | `path` | `""` | Load 256-dim task vector from `.bin`, `.npy`, or `.txt` |
91
  | `--task_vector` | `string` | `""` | Comma-separated 256 floats: `"0.12,-0.05,..."` |
92
  | `--task` | `string` | `""` | Task string (resolved via pre-cached table) |
93
+ | `--agent_img` | `path` | `""` | Path to raw \\(256 \\times 256 \\times 3\\) uint8 RGB agent camera frame |
94
+ | `--wrist_img` | `path` | `""` | Path to raw \\(256 \\times 256 \\times 3\\) uint8 RGB wrist camera frame |
95
+ | `--output` | `path` | `""` | File path to write predicted \\(8 \\times 7\\) action trajectory |
96
+ | `--benchmark` | `int` | `0` | Run \\(N\\) benchmark iterations and print latency breakdown |
97
 
98
  ### CLI Usage Examples
99
 
 
210
  ## 6. Language Handling
211
 
212
  When using string instructions:
213
+ 1. If the instruction matches a canonical benchmark task, its embedding is resolved instantly (\\(\\lt 1\\,\\mu\\text{s}\\)) from `task_embeddings.bin`.
214
  2. If an instruction differs (e.g., *"grasp the black bowl between the plate and the ramekin"* vs. *"take the black bowl and set it onto the plate"*):
215
  - If `parvulus_text_encoder.onnx` is present (or downloaded from Hugging Face / generated via converter), `ParvulusRuntime` encodes the prompt.
216
  - If ONNX is not generated, `ParvulusRuntime` automatically uses PyTorch (`text_encoder.py`) without failure.
217
+ 3. Task conditioning is set once per episode (\\(t=0\\)). The control loop streams only camera frames, completely bypassing any text-processing overhead.
218
 
219
  ### Converting or Downloading the ONNX Text Encoder
220
  The ONNX text encoder binary is not tracked. To prepare it: