| --- |
| license: other |
| license_name: nvidia-open-model-license |
| license_link: https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/ |
| tags: [robotics, humanoid, unitree-g1, steering, loco-manipulation] |
| --- |
| # Token steering for SONIC-based G1 box loco-manipulation (R15) |
|
|
| Training-free closed-loop steering of a frozen motion-tracking foundation model: a small offset is added to SONIC's |
| 64-D post-quantizer tokens (g1 encoder) so the robot follows the planned walk-up more closely, using only odometry |
| (deployable). Bases are calibrated for the R15 policy ([`daibfc/h2r-weights`](https://huggingface.co/daibfc/h2r-weights)). |
|
|
| ## Files |
| | file | content | |
| |---|---| |
| | `bases/A_R15_omomo_nopca.npz` | the steering basis `A [2, 64]`: token change per +1 cm forward / +1 cm lateral pelvis motion (+ fit `M`, `r2`, `n`) | |
| | `calibration/omomo_pushes_r15.npz` | the 408 push experiments behind the recommended basis: `U [N, 64]` pushes, `D [N, 2]` responses (cm), `clip` | |
| | `presets/pace_walk_nopca.yaml` | the run-time rule as used in our framework | |
| | `steer_reference.py` | standalone numpy reference of the run-time rule | |
|
|
| ## Algorithm (pseudocode) |
|
|
| Calibration (offline, once per policy; R15 step 15000): |
| ```text |
| CLIPS = 68 clean OMOMO clips whose walk-up (start -> planned lift) is >= 1.5 s |
| ENVS = each clip x 2 envs (136 episodes per rollout, fixed seed -> same episode in the same env slot) |
| WINDOW = [frame 0, planned_lift - 10 frames) # walk-up only |
| |
| # 1. baseline: no push, steering logger only (gain 0) |
| for each episode i: |
| run policy; log e_base[i] = plan-tracking error of the pelvis at the end of WINDOW |
| (forward, lateral in the plan's heading frame, metres) |
| |
| # 2. pushes: 3 rollouts x 136 episodes |
| for run r in 1..3: |
| for each episode i: |
| u = randn(64); u = 0.25 * u / ||u|| # random 64-D direction, norm exactly 0.25 |
| run policy; during WINDOW add u to SONIC's post-FSQ g1-encoder tokens (64-D) every step |
| log e_push[i] at the end of WINDOW |
| if the robot fell in either run: skip |
| U.append(u); D.append(100 * (e_base[i] - e_push[i])) # cm the push moved the pelvis (fwd, lat) |
| |
| # 3. fit (408 pairs) |
| M = ((U^T U + 1.0 * I)^-1 U^T D)^T # ridge, 2 x 64: cm of pelvis motion per unit token push |
| A = pinv(M)^T # 2 x 64: minimum-norm token push per +1 cm fwd / +1 cm lat |
| ``` |
|
|
| Run time (every control step, 50 Hz, before the policy forward; deployable, odometry only): |
| ```text |
| e = R(plan_heading)^T * ( (plan_pelvis_xy(t) - plan_pelvis_xy(0)) - (odom_xy(t) - odom_xy(0)) ) # metres |
| v = clip(160 * e, -15, +15) |
| if ||e|| > 0.5 m or t >= planned_lift: v = 0 # guard + walk-only gate |
| tokens_g1 += v @ A # added to SONIC's post-FSQ g1 tokens |
| ``` |
|
|
| Why pushes: a push is random and independent of the robot's state (a randomized intervention), so U -> D is the causal |
| effect of a token change. Without pushes (fitting the same ridge on natural run-to-run token variation), the fit |
| learns SONIC's own reaction to being off-plan (tokens change BECAUSE the robot is behind), so the inverse points the |
| wrong way: 0% pickup in closed loop (table below). |
|
|
| ## Download |
| ```bash |
| pip install -U huggingface_hub |
| huggingface-cli download daibfc/h2r-steering --local-dir h2r-steering # this basis + push data + rule |
| huggingface-cli download daibfc/h2r-weights --local-dir h2r-weights # R15 policy it is calibrated for |
| huggingface-cli download daibfc/h2r-benchmark-feasible43 --repo-type dataset --local-dir h2r-benchmark-feasible43 |
| ``` |
| ```python |
| import numpy as np |
| A = np.load("h2r-steering/bases/A_R15_omomo_nopca.npz")["A"] # [2, 64] |
| z = np.load("h2r-steering/calibration/omomo_pushes_r15.npz") # U [408, 64], D [408, 2] (cm), clip [408] |
| M = np.linalg.solve(z["U"].T @ z["U"] + np.eye(64), z["U"].T @ z["D"]).T |
| assert np.allclose(np.linalg.pinv(M).T, A, atol=1e-6) # refit reproduces A |
| ``` |
|
|
| ## Benchmark results (HOI2ROI feasible43: 43 cases × 8 episodes = 344, paired episodes, R15, deployable odometry, full-size box) |
| | steering | pickup | success | falls | |
| |---|---|---|---| |
| | none | 29.4% | 20.1% | 9 | |
| | previous basis (PCA-16, not released) | 43.6% | 35.8% | 9 | |
| | **this basis (no PCA)** | **54.1%** | 36.3% | 20 | |
| | PCA-8 / 16 / 32 bases (same pushes, PCA coordinates) | 26.7 / 41.0 / 37.2% | 20.1 / 27.0 / 26.2% | 33 / 12 / 8 | |
| | sham: random directions, same size as no-PCA | 30.2% | 18.3% | 9 | |
| | observational: same fit on unpushed run-to-run variation | 0.0% | 0.0% | 59 | |
|
|
| Findings: (1) interventions are necessary: a basis fitted on natural variation steers the wrong way; (2) the |
| calibrated directions matter, not the push size (sham); (3) offline fit quality (R²) does not rank bases: PCA bases fit |
| better offline but steer worse in closed loop; (4) the no-PCA basis has the smallest lateral error at the end of the |
| walk-up (5.7 vs 7.4 cm for the previous basis). Open: more falls than the previous basis (20 vs 9). |
|
|
| Benchmark: [`daibfc/h2r-benchmark-feasible43`](https://huggingface.co/datasets/daibfc/h2r-benchmark-feasible43). |
| The bases are specific to R15 (the residual changes how tokens move the robot); recalibrate for another policy. |
| No benchmark (HOI2ROI) clip was used for calibration. |
|
|
| ## Bases for the degen_v2 hybrid |
| | file | basis | |
| |---|---| |
| | `bases/A_sonic_omomo_iso64.npz` | base SONIC (R15 step 15000, residual off), 64-D iso pushes on 68 OMOMO clips. **Used by the degen_v2 19k hybrid** (58.7% pickup / 40.7% success on HOI2ROI feasible43, box 0.7601) | |
| | `bases/A_degen_v2_19k_omomo_iso64.npz` | degen_v2 step 19000 itself, same calibration; worse than the base-SONIC basis in the hybrid (shipped for reference) | |
| |
| Same rule and preset (`presets/pace_walk_nopca.yaml`, override `sources.0.A`). Calibration used OMOMO only, never the benchmark. |
| |
