Upload tinyvla2/RESULTS.md with huggingface_hub
638924d verified about 2 months ago
46.8 kB
# TinyVLA-2 A/B/C — final results (held-out jaco, integrated-trajectory endpoint error)

Test split jaco[956:976] (n=20), GT trajectories ~210mm, zero-prediction floor ~128mm.

| jaco FT episodes | A (native+ID) | B (canonical+ID) | C (canonical+morph) |
|---|---|---|---|
| 0 (zero-shot)    | 1377 mm | 155 mm | 178 mm |
| 100              | 6919 mm | 215 mm | 218 mm |
| 500              | 5667 mm | 198 mm | 197 mm |

## Verdict
1. **Canonical action space ≫ native (10-45×)** at every budget — decisive. FK-unified
   base-frame EE actions are what enable cross-embodiment transfer. Native is not just
   worse, it DIVERGES under few-shot FT (fresh embodiment-ID + native action dist).
2. **Morphology tokens ≈ learned ID (C ≈ B)** — no measurable advantage once actions
   are canonical. The morphology hypothesis (C>B) is NOT supported.
3. **Short few-shot FT (5k steps) barely helps** — pretrained canonical already
   transfers zero-shot (~155mm vs 128mm floor); the fresh-ID re-learn even hurts B at
   N=100. Protocol caveat: warm-start the new ID, more steps, tune LR.

Honest takeaway: **the win is action-space unification, not morphology conditioning.**

## TidyBot (wheeled mobile manipulator) — B vs C morphology test (endpoint mm)
Held-out mobile morphology (is_mobile=1, holonomic), test eps [95:110], n=15.
| N eps | B (canon+ID) | C (canon+morph) |
|---|---|---|
| 0 (zero-shot) | 352 | 419 |
| 30 | 332 | 383 |
| 80 | 335 | **329** |
C starts WORSE (OOD descriptor extrapolation) but adapts far steeper (419→329, −90mm)
vs B's plateau (352→335, −17mm), OVERTAKING B by N=80 on both endpoint (329<335) and
rotation (24°<28°). First support for morphology conditioning: it helps on a genuinely
NEW morphology (mobile), unlike held-out arms (jaco: C≈B, both flat). Margin small
(~2%, n=15) — crossover needs confirmation (more N, seeds).
## Morphology-diversity continue-train (user idea 3) — held-out jaco zero-shot
| model | training morphologies | morph→fast | morph→slow | jaco endpoint |
|---|---|---|---|---|
| C           | 3 (arms only)        | ✓ | ✗ | 178 mm |
| C-diverse   | 8 (arms + mobile)    | ✓ | ✗ | **150 mm** (−16%) |
| C-diverse-slow | 8                 | ✓ | ✓ | (training) |
Morphology diversity in continue-training improves held-out-arm transfer 178→150mm:
the descriptor→conditioning MLP only generalizes once it has seen diverse morphologies
(arm-only C's encoder was useless/harmful on new morphologies). Caveat: TidyBot
(331mm) is now IN C-diverse training, so it's not a clean OOD-mobile test; a strict
OOD-mobile claim needs a held-out mobile robot (leave-one-out or a new dataset).
## Slow-path morphology injection (idea: feed morphology to Qwen prefix too)
| model | morph→fast | morph→slow | jaco endpoint |
|---|---|---|---|
| C-diverse      | ✓ | ✗ | 150 mm |
| C-diverse-slow | ✓ | ✓ | 162 mm |
No benefit (slightly worse, within n=20 noise). Fresh morph_to_lm projector (17K
params) likely underfit in 25k steps; injecting into the frozen LM prefix perturbs
the semantic latent without clear gain on a held-out arm. Idea sound, didn't pay off here.
## TinyVLA-2 overall verdict
1. Canonical action space >> native for cross-embodiment transfer (10-45x). DECISIVE.
2. Morphology conditioning ~= learned ID on held-out ARMS — UNLESS the descriptor
   encoder is trained on DIVERSE morphologies, which improves held-out transfer
   (178->150mm). On a mobile morphology (TidyBot) C adapts steeper than B (overtakes by N=80).
3. Slow-path morphology injection: no clear benefit here.
Takeaway: unify actions (big win); morphology conditioning helps only with morphological
diversity in training.
## ZERO-SHOT new-robot via descriptor (no fine-tuning) — the core promise
Descriptor sensitivity: swapping the descriptor (same observations) changes predictions
by ~0.5 RMS (normalized) → the model genuinely READS the descriptor zero-shot.
Held-out jaco, C-diverse, NO fine-tuning, endpoint error by descriptor given:
| descriptor fed | endpoint |
|---|---|
| jaco (correct)  | 159.6 mm |
| so101 (wrong arm) | 166.0 mm |
| zeros (none)    | 167.6 mm |
| uav (alien)     | 178.2 mm |
Correct new-robot descriptor → best zero-shot control (no FT). Describing a robot
correctly beats a wrong/absent/alien descriptor. Modest (~5-11%) but consistent.
## LeKiwi (wheeled mobile manipulator, SO-100 arm + kiwi base) — ZERO-SHOT, no fine-tuning
Arm-EE endpoint error on held-out LeKiwi split (C-diverse, descriptor swapped, NO FT):
| descriptor fed | endpoint |
|---|---|
| zero-floor      | 241 mm |
| lekiwi (correct, is_mobile=1) | 295 mm |
| uav (alien)     | 363 mm |
| so101 (same arm but is_mobile=-1) | 533 mm |
Correct descriptor beats the SAME-ARM-but-non-mobile descriptor by 45% (295 vs 533) —
the model genuinely uses the mobility bit zero-shot (mobile base couples arm-in-frame
motion). BUT correct-descriptor (295) still > zero-floor (241): zero-shot control of a
genuinely new mobile platform isn't achieved — needs base dims in the action head
(our 8-dim output has no vx/vy/omega) + some adaptation. Descriptor makes it "less
wrong", not yet working. LeKiwi has no sim env, so this is offline; closed-loop would
need building a MuJoCo LeKiwi scene.
## Text robot-prompt (Qwen slow-path) x numeric descriptor ablation — held-out LeKiwi
C-mega (9 max-diverse robots, resume from C-diverse, 20k steps), zero-shot on LeKiwi:
| condition | endpoint |
|---|---|
| zero-floor | 241 mm |
| numeric=lekiwi + text=NONE   | **298.5 mm** (best; ~= prior C-diverse 295mm) |
| numeric=lekiwi + text=lekiwi | 314.6 mm (text HURTS slightly, +16mm) |
| numeric=none + text=none    | 413.4 mm |
| numeric=none + text=lekiwi  | 522.8 mm (text alone WORSE than nothing) |
| numeric=so101(wrong) + text=lekiwi | 565.2 mm (conflicting signals, worst) |
**Text prompt does NOT help, and actively hurts when used alone or combined.**
Numeric descriptor remains the only working conditioning channel. Likely cause: the
text channel is genuinely new (this run), LM is frozen, and 20k continue-train steps
weren't enough to learn to use it — it just perturbs the readout-token aggregation
as noise. The numeric MLP path has accumulated training since the original C
pretrain and works. Conclusion: keep numeric morphology descriptors; the text-prompt
idea needs either LM unfreezing, far more steps, or a different injection point to
pay off (not validated here).
9-robot diversity (adding jaco+lekiwi-adjacent training) did not improve over the
8-robot C-diverse baseline on LeKiwi (298 vs 295mm, within noise) — diminishing
returns past ~8 morphologies at this training budget.
## Morphology via shared Qwen (text, unfrozen LM) — held-out LeKiwi
Replaced the MLP-over-numbers morphology encoder with the SAME Qwen doing a
text-only pass over a robot description ("Robot: LeKiwi, a 5-DOF SO-100 arm on a
three-wheel holonomic mobile base."), AND unfroze the LM (249M params, 0.1x lr),
resumed from C-diverse (8 robots), 20k steps.
| condition | endpoint |
|---|---|
| zero-floor | 241 mm |
| desc=none (empty)         | **294.1 mm** (best) |
| desc=lekiwi (correct)     | 314.1 mm |
| desc=so101 (wrong, non-mobile) | 464.8 mm |
Absolute level unchanged vs prior best (294-314 vs 295-298mm) — neither LM-unfreeze
nor Qwen-encoding gave a breakthrough at this budget. BUT the correct-vs-wrong gap
widened to 32% (314 vs 465mm) vs the numeric-MLP's ~10% (150 vs 166mm on jaco) —
the Qwen text encoder is more sensitive to descriptor CONTENT, a sign real language
understanding is engaged (unlike the frozen-LM C-mega experiment where wrong beat
none, i.e. noise). Oddity: desc=none beats the correct description — LeKiwi's
specific phrasing/combination of words may be out-of-distribution for the 8 training
descriptions the (still fairly short, 20k-step) fine-tune has seen.
Net: promising SENSITIVITY signal, not yet a WIN on absolute error. Would need
either more diverse text descriptions in training, more steps, or investigating
the desc=none anomaly to turn this into a clear improvement.
## In-context few-shot demo conditioning (K=3 real exemplars, NO gradient updates) — held-out LeKiwi
DemoConditioner: K=3 (obs,action) pairs -> spatial-encoder mean-pool + action summary -> MLP -> tokens,
composed with morph_qwen text channel. Trained 20k steps (resumed from C-qwen-morph).

| condition | endpoint |
|---|---|
| REAL LeKiwi demos + desc=lekiwi | 324.7 mm |
| REAL LeKiwi demos + desc=none   | 290.6 mm |
| GARBAGE/noise demos + desc=lekiwi | 322.5 mm |
| GARBAGE/noise demos + desc=none   | 288.5 mm |

**Real demos ≈ garbage demos (within noise) — the demo conditioner learned to
IGNORE support-example content entirely.** Gradient reaches demo_encoder during
training (verified), but the model found it easier to route around this channel
than to use it, likely because during training support pairs were sampled from
the SAME source as the query — redundant with the already-available text/numeric
identity signal, so no incentive to learn content-dependence. Net: dead channel,
clear negative result. Would need support examples that carry information NOT
available elsewhere (e.g. from a genuinely different episode/task, or examples
that vary the dynamics being asked about) to give the model a reason to attend
to them.
## Demo conditioning v2 — exemplars from a DIFFERENT TASK of the same robot
Fix attempt for v1's dead channel: sample support pairs cross-task (so they carry
robot-dynamics info the identity channels don't duplicate). Held-out LeKiwi, no FT:
| condition | endpoint |
|---|---|
| zero-floor | 241 mm |
| GARBAGE demos + desc=none | **260.9 mm** (best) |
| REAL LeKiwi demos + desc=none | 263.6 mm |
| OTHER-ROBOT (jaco) demos + desc=none | 265.9 mm |
| REAL demos + desc=lekiwi | 294.0 mm |
| GARBAGE demos + desc=lekiwi | 294.2 mm |
TWO findings:
1. **Absolute level improved a lot**: 261-266mm vs 288-295mm for every prior
   variant — closest to the 241mm floor in the whole series. But this came from
   TRAINING WITH the extra (noisy) channel acting as regularization/augmentation,
   NOT from using demo content.
2. **The channel is still content-blind**: real (263.6) ≈ garbage (260.9) ≈
   other-robot (265.9), all within noise. Cross-task sampling did NOT break the
   shortcut — the model still ignores WHAT is in the exemplars.
Conclusion: in-context demo conditioning does not work in this architecture even
with informative exemplars. Its only benefit is incidental regularization. Two
plausible remaining causes: (a) the mean-pool + 3-step action summary destroys
the information, (b) a single MLP token per exemplar is too weak a pathway —
proper in-context learning likely needs the exemplars inside the attention stream
(e.g. as extra tokens the expert cross-attends over), not squeezed through an MLP.
## Demo conditioning v3 — high-capacity encoder (24x params) — held-out LeKiwi
Widened every stage: spatial group-pools (not global mean), 8 action keyframes
(not 3), 3-layer 4x-wide MLP + LayerNorm, 4 tokens/exemplar; 0.7M -> 16.9M params.
| condition | v3 | v2 (for reference) |
|---|---|---|
| REAL LeKiwi demos + desc=none | 294.8 mm | 263.6 mm |
| GARBAGE demos + desc=none | 291.3 mm | 260.9 mm |
| OTHER-ROBOT demos + desc=none | 293.2 mm | 265.9 mm |
**Capacity was NOT the problem.** Third independent attempt (v1 same-task, v2
cross-task, v3 high-capacity) and the channel is still content-blind: real ≈
garbage ≈ other-robot within noise. v3 is also worse in absolute terms than v2
(294 vs 261mm) because it had to resume from C-qwen-morph (v2 weights were
shape-incompatible with the new encoder), losing v2's incidental regularization.
Conclusion after three attempts: the failure is ARCHITECTURAL PLACEMENT, not
capacity or sampling. Demo tokens sit in the expert's conditioning set alongside
~150 other tokens; the flow head predicts actions from the CURRENT observation and
has no mechanism to COMPARE the current scene against exemplars. Real in-context
learning needs exemplars in the attention stream where query tokens can attend to
them selectively (RoboCat-style separate demo cross-attention, or explicit
retrieval), not "placed alongside" where the network marginalizes them as constant
noise. BEST CHECKPOINT OF THE SERIES REMAINS v2 (261mm on LeKiwi zero-shot).
## DEMO-ONLY — all identity channels removed (5th and final in-context attempt)
Removed embodiment-ID, morphology descriptor AND robot text; the K=4 demo exemplars
in the LM attention stream were the ONLY cue for which robot this is (per-source
action normalization creates the pressure: correct prediction requires knowing the
robot's action scale, available only from demos). Held-out LeKiwi:
| condition | step 5k | step 10k |
|---|---|---|
| REAL LeKiwi demos | 376.1 mm | 323.6 mm |
| GARBAGE/noise demos | 393.4 mm | 322.1 mm |
| OTHER-ROBOT (jaco) demos | 330.3 mm | 294.8 mm |
At 5k a spread finally appeared (63mm — the first time any demo variant showed
content-sensitivity) but by 10k it COLLAPSED back: real (323.6) ≈ garbage (322.1).
Other-robot demos "win" only because they regress toward the training distribution,
not because demos are read. Training stopped at 10k (no point burning 7 more hours).
### In-context demo conditioning: FINAL VERDICT after 5 architectures
1. MLP encoder, same-task exemplars — dead (real ≈ garbage)
2. MLP encoder, cross-task exemplars — dead (but +regularization: 295→261mm)
3. High-capacity encoder (24x params, spatial groups) — dead
4. Demos inside the LM attention stream (rich-slow, dual-rate preserved) — dead
5. Demos as the ONLY identity channel — brief flicker at 5k, dead by 10k
Neither sampling, capacity, architectural placement, nor removing all alternatives
makes this model read demonstration content. Plausible remaining explanation: the
flow head predicts from the CURRENT observation, and nothing in the training
objective ever *requires* comparing the current scene to an exemplar — the robot is
visually identifiable in-frame (an identity leak we cannot remove), so the model
always has a cheaper route than reading demos.
**Working mechanism remains the numeric morphology descriptor (C-diverse: 150mm on
held-out jaco). Best LeKiwi zero-shot: C-incontext-v2 at 261mm.**
## Reality check: absolute performance across robots (C-diverse, best morphology ckpt)
Endpoint error vs the per-robot zero-prediction floor (ratio<1 = better than
predicting mean motion). Absolute mm are NOT comparable across robots (different
motion scales) — the ratio is.
| robot | status | endpoint | floor | ratio |
|---|---|---|---|---|
| stretch (mobile) | in training | 110 mm | 139 mm | **0.79** |
| xarm | in training | 171 mm | 179 mm | 0.95 |
| dlr_edan (wheelchair) | in training | 326 mm | 340 mm | 0.96 |
| ur5 | in training | 229 mm | 183 mm | 1.25 |
| jaco (arm) | HELD-OUT | 161 mm | 146 mm | 1.10 |
| LeKiwi (mobile) | HELD-OUT | 294 mm | 239 mm | 1.23 |
| **mean, in-training** | | | | **0.99** |
| **mean, held-out** | | | | **1.17** |

**IMPORTANT CAVEAT FOR THE WHOLE TinyVLA-2 SERIES:** this model barely beats the
trivial "predict mean motion" baseline even on robots it trained on (mean ratio
0.99). Every morphology comparison in this document (178 vs 150mm, canonical vs
native, demo channels) is therefore a comparison between variants of a WEAK model.
The RELATIVE conclusions stand — they were measured under identical conditions —
but nobody should read these numbers as "this is a working cross-embodiment
controller". For reference, TinyVLA v1 reached 86% LIBERO success with 40k steps on
ONE domain with an unfrozen LM; here we spread 40k+20k steps across 8 morphologies
with a mostly-frozen backbone. Getting absolute competence would need far more
compute per embodiment, not more conditioning tricks.
Interesting detail: held-out jaco (1.10) beats in-training ur5 (1.25) — robot
SIMILARITY matters more than mere presence in the training mix, which is mildly
supportive of the morphology-conditioning idea.

## SCALED C — the fix that actually mattered (data + budget, not conditioning tricks)
Full Bridge (53K eps) + full RT-1 (87K eps) → pool 4.4M → 9.1M frames; 60k steps
(was 20k); LM unfrozen at 0.1x lr. Same C setup (canonical actions + numeric
morphology descriptor). Ratio = endpoint error / per-robot zero-prediction floor
(<1.0 beats "predict mean motion").

| robot | C-diverse | 20k | 40k | 60k final |
|---|---|---|---|---|
| jaco | 1.10 | 0.80 | 0.69 | **0.69** |
| xarm | 0.95 | 0.66 | 0.62 | **0.64** |
| dlr_edan | 0.96 | 0.77 | 0.71 | **0.69** |
| stretch | 0.79 | 0.71 | 0.66 | **0.69** |
| ur5 | 1.25 | 1.27 | 1.24 | **1.22** |
| LeKiwi (HELD-OUT mobile) | 1.23 | 1.24 | 1.13 | **1.07** |
| **mean in-training** | **0.99** | 0.85 | 0.81 | **0.81** |
| **mean HELD-OUT** | **1.17** | 1.02 | 0.91 | **0.88** |
**This is the main positive result of TinyVLA-2.** The model went from barely
matching the trivial baseline (0.99) to clearly beating it (0.81 / 0.88). Five
conditioning tricks (text prompts, Qwen-encoded morphology, demo conditioning x3
architectures) moved nothing; scaling data + steps + unfreezing the LM moved
everything. In-training saturates by 40k (0.81) — further gains would need
unfreezing the vision tower or a bigger expert, not more steps. Held-out is still
improving at 60k (0.91→0.88), and LeKiwi — the only true unseen morphology — has
not saturated (1.24→1.13→1.07), suggesting cross-morphology transfer is the
slowest-converging capability and would keep improving with more budget.
ur5 stays hardest (1.22): its canonicalization is fine, it is simply the fastest
robot in the mix (9.9mm/step vs 5.4 for jaco, 15x the per-step rotation).
---
# CORRECTION (2026-08-28): the offline eval fed the model the wrong proprioception
Every offline number above this line was produced by an eval harness that passed
`observation.state` **raw**, truncated to 6 dims. Training normalizes it with the
dataset's own mean/std and pads to `max_state_dim` (`mixture.CanonicalSource.
__getitem__` → `_norm` → `_pad`). For joint-angle datasets the raw values are
O(100) where the model expects O(1), so the policy was evaluated with a
proprioceptive input tens of sigma outside its training distribution, and mobile
robots additionally lost their base channels to the `[:6]` truncation.
The fix is `src/tinyvla/data/eval_utils.py::StateAdapter`, now used by every eval
script. Controlled re-measurement (same checkpoint, same episodes, same sampled
frames — only the state input differs) is in `scripts/` and reproduces below.

## Corrected: tv2_C_scaled (60k) across robots

| robot | status | as-published | corrected | Δ |
|---|---|---|---|---|
| stretch (mobile) | in training | 0.69 | **0.11** | −0.58 |
| xarm | in training | 0.64 | **0.13** | −0.51 |
| dlr_edan (wheelchair) | in training | 0.69 | **0.23** | −0.46 |
| ur5 | in training | 1.22 | **0.90** | −0.32 |
| jaco (arm) | HELD-OUT | 0.69 | **0.28** | −0.41 |
| LeKiwi (mobile) | HELD-OUT | 1.07 | **1.67** | **+0.60** |
| **mean, in-training** | | 0.81 | **0.34** | −0.47 |
| **mean, HELD-OUT** | | 0.88 | **0.97** | +0.09 |
## What this changes about the project's conclusions
**The "weak model" caveat was wrong.** The model is far more competent than
reported: 0.34 on robots it trained on, not 0.81. It was never barely beating the
"predict mean motion" baseline — it beats it by ~3x. The paragraph above claiming
"nobody should read these numbers as a working cross-embodiment controller" was an
artefact of the harness, not a property of the model.
**The held-out claim splits in two, and only half survives.** Transfer to an unseen
*arm* is genuinely strong: jaco 0.28, better than in-training ur5 (0.90). Transfer
to an unseen *mobile base* got worse under correct measurement: LeKiwi 1.67, i.e.
**worse than predicting mean motion**. The honest statement is now: morphology
conditioning transfers across arms of different geometry, and fails on a base type
never seen in training.
The mechanism is consistent: with garbage proprioception the model had to ignore
the state channel entirely, which flattered LeKiwi (whose proprio it cannot
interpret anyway) and penalised every robot whose state was informative. Correct
proprio helps wherever the model learned to read it and actively misleads it on the
one morphology class it never saw.
**Relative conclusions between conditioning variants are NOT automatically safe.**
They were all measured under the same bug, so the comparison was internally
consistent, but the bug suppressed the state channel — precisely one of the inputs a
conditioning mechanism could have exploited. The in-context/demo experiments are
being re-measured; see below.
## Re-measured after the fix: does any conditioning conclusion change?
| experiment | condition | as-published | corrected |
|---|---|---|---|
| C-qwen-morph (LeKiwi) | desc=lekiwi (correct) | 294 mm | 503.9 mm |
| | desc=so101 (wrong robot) | 302 mm | 519.4 mm |
| | desc=none | 294 mm | 756.4 mm |
| C-incontext-v3 (LeKiwi) | REAL demos K=3 | 439 mm | 439.0 mm |
| | GARBAGE demos | 431 mm | 429.4 mm |
| | OTHER-ROBOT (jaco) demos | — | 437.0 mm |
| demo-only (LeKiwi, 10k) | REAL demos K=4 | 261 mm | 490.5 mm |
| | GARBAGE/noise demos | 259 mm | 518.7 mm |
| | OTHER-ROBOT (jaco) demos | — | 489.5 mm |
**The in-context negative result survives the fix, and is now better supported.**
Real exemplars remain indistinguishable from garbage (439.0 vs 429.4; garbage is
marginally *better*), and — the sharper test — real LeKiwi demos are
indistinguishable from demos of a completely different robot (490.5 vs 489.5). The
model is reading demonstration *format*, never demonstration *content*.
One thing the fix does buy: the descriptor channel now clearly matters. With correct
proprioception, dropping the morphology descriptor costs 503.9 → 756.4 mm, where
under the bug it cost nothing (294 → 294). So the numeric descriptor is a working
channel that the broken harness had rendered invisible.
`tv2_C_incontext` and `tv2_C_incontext_v2` could NOT be re-measured: widening the
demo encoder ("increase MLP capacity") changed its weight shapes (512x1048 →
2048x4160), so those checkpoints no longer load against current code. Their
published numbers stand as measured under the bug and should not be compared with
anything in this section.

# LIBERO closed-loop benchmark of the canonical (v2) line

The only benchmark comparable to v1 (86% spatial) and to the literature. The v2
model predicts base-frame EE deltas, not LIBERO's OSC action units, so it needs a
decoder — and the decoder needs its own correctness gate.

## Pipeline
- Sidecar: LIBERO state is `[pos(3), rotvec(3), grip(2)]` — neither euler nor
  quaternion, so `build_ee_sidecar.py` gained `--rot-format rotvec`. 273,465 frames.
- Decoder fit from ground truth: per-dim scale `[80.4, 76.9, 78.8]` (position) and
  `[7.57, 7.67, 8.60]` (rotation); correlations with the GT action 0.93 / 0.96 /
  0.96 / 0.95 / 0.84 / 0.98. Gripper is absolute openness, decoded by a threshold at
  0.879 (92.5% sign agreement vs 86.8% for a linear map).

## Round-trip gate (GT actions through encode → decode → env)
| decode | success |
|---|---|
| raw GT actions (ceiling) | **4/5** |
| scaled: per-step delta x fitted gain | **1/5** |
| servo, kp=60 | 3/5 |
| servo, kp=80 | 3/5 |
| **servo, kp=120** | **4/5** |
| servo, kp=160 | 3/5 |

The naive decode loses three quarters of the ceiling. Reason: a canonical delta is
*achieved* motion, while an OSC action is a *command* to a P-controller that never
fully closes the gap in one step, so replaying achieved motion as a command
systematically under-shoots. Integrating the deltas into an absolute desired pose
and commanding the error against the *measured* pose removes the lag and recovers
the full ceiling. **Anyone deploying a canonical-action policy on a position-servo
robot needs this; the scaled decode looks reasonable offline and fails on hardware.**

## Results (30k fine-tune complete)

Pipeline ceilings first, so the model number can be read against what the action
space and decoder actually permit:

| what is being replayed | success |
|---|---|
| raw GT actions (env ceiling) | 9/10 |
| canonical encode→decode, GT gripper | **9/10** |
| canonical encode→decode, decoded gripper | 8/10 |

**The canonical motion representation is lossless** — routing GT through
base-frame EE deltas and back through the servo decoder costs nothing (9/10 either
way). The whole ~10-point loss comes from the gripper: canonical stores absolute
openness, and turning that back into LIBERO's binary open/close command is only
92.5% accurate per step. That is the one place the canonical space is lossy, and it
is fixable (predict the command channel directly rather than openness).

| model | LIBERO-spatial (n=50) |
|---|---|
| zero-shot, no LIBERO training (offline ratio to zero-floor) | 0.79 |
| canonical FT 5k steps | 40% |
| **canonical FT 30k, full rate** | **70%** |
| canonical FT 30k, dual-rate (latent @1 Hz) | 68% |
| decoded-pipeline ceiling | 80% |
| v1, native actions, 40k steps (reference) | 86% |

**70% against an 80% decoder ceiling**: the policy captures ~88% of what the
pipeline allows. Against v1's 86% in the native action space, canonical actions cost
16 points, of which ~10 are the gripper decode and ~6 are the policy.

**Dual-rate survives the change of action space** (68% vs 70%, indistinguishable at
n=50). The v1 claim — a semantic latent refreshed at 1 Hz is enough, the fast path
carries the rest — holds for a cross-embodiment canonical policy too, which is the
result that makes the v2 line deployable at v1's compute profile.

Zero-shot 0.79 means the cross-embodiment checkpoint beat the trivial baseline on a
robot and domain it had never seen, before any LIBERO training at all.

## RoboCasa continue-train, re-measured (step_10000, corrected harness)
RoboCasa365 (sim kitchens, PandaOmron mobile manipulator, 1.5M frames) at weight
0.50, old robots replayed at 0.50, continued from C-scaled. Both columns measured
with the corrected `StateAdapter`; the run was stopped at ~18k and only
`step_10000` was kept.

| robot | C-scaled (base) | +RoboCasa 10k | Δ |
|---|---|---|---|
| RoboCasa (NEW domain) | — | **0.47** | new |
| xarm | 0.13 | 0.17 | +0.04 |
| dlr_edan | 0.23 | 0.26 | +0.03 |
| stretch | 0.11 | 0.25 | **+0.14** |
| ur5 | 0.90 | 0.96 | +0.06 |
| jaco (held-out arm) | 0.28 | 0.32 | +0.04 |
| LeKiwi (held-out base) | 1.67 | **1.60** | −0.07 |
| mean, old in-training | 0.34 | 0.41 | +0.07 |
**This reverses what we reported under the buggy harness.** The old story was
"RoboCasa integrates well, old robots improve 0.81→0.76, but held-out LeKiwi
collapses 1.07→1.58, so weight 0.5 is too high". Correctly measured, LeKiwi does not
collapse — it is unchanged to slightly better (1.67→1.60) — and the cost lands
instead as mild, uniform forgetting on the robots that were already learned
(mean 0.34→0.41, worst case stretch 0.11→0.25, which more than doubles).
So the trade-off is real but benign and ordinary: a genuinely new domain reaches
0.47 for ~20% relative degradation spread across the old robots, with no effect on
cross-morphology transfer. Weight 0.5 is defensible; the earlier "too high,
catastrophic" conclusion was an artefact.
Calibrate 0.47 honestly: it is **mid-pack**, not best-in-project. Corrected
in-training ranking is xarm 0.17 < stretch 0.25 < dlr_edan 0.26 < **RoboCasa 0.47**
< ur5 0.96. RoboCasa only looked best-in-project under the buggy harness, where
every other robot sat at 0.7-1.2. It is also the least-trained entry here — 10k
continue-train steps on a 1.5M-frame domain, versus 60k for the rest — so 0.47 is a
floor on what this domain can reach, not a converged value.

# Joint training with Unitree humanoid/arm + navigation packs (tv2_joint, 40k)
3.25M new samples added to the 9.1M-frame canonical pool: Unitree G1/Z1 with
native joint-position actions (7/14/16/26/28 channels) and five navigation packs
with 3-dim waypoints/step-deltas. One flow head serves all three action spaces via
the per-sample `action_dim_mask`. Weights: old pool 0.50, unitree 0.30, nav 0.20.
Resumed from a dimension-expanded checkpoint (state 16→32, action 8→32,
embodiments 16→24, all new rows/columns zero), which is also the control column
below: zero-initialised output rows mean "no capability on the new dims yet".
## What the new data bought (ratio to each space's trivial baseline)
Nav actions are deltas → floor is predicting zero motion. Unitree actions are
ABSOLUTE joint positions → integrating is meaningless, so the floor is "hold
still", i.e. predict the current joint configuration for the whole chunk.
| embodiment | space | before | after |
|---|---|---|---|
| G1 Dex3 ToastedBread | joint | 2.76 | **0.65** |
| G1 Dex1 DualArm | joint | 2.47 | **0.78** |
| Z1 StackBox | joint | 1.09 | **0.53** |
| scand (social nav) | nav | 1.08 | **0.33** |
| tartandrive (ATV) | nav | 0.91 | **0.38** |
| sacson (indoor nav) | nav | 2.18 | **0.73** |
| go-stanford (nav) | nav | 1.17 | **0.76** |
| recon (outdoor nav) | nav | 1.24 | 1.04 |
| **mean, joint** | | 2.11 | **0.65** |
| **mean, nav** | | 1.32 | **0.65** |
Both new modalities are genuinely learned: a humanoid controlled in 28-channel
joint space and five navigation robots all beat their own trivial baselines from a
model that started at or below those baselines.
**recon is the exception and the interesting one**: it is the LARGEST nav pack
(598k samples, the biggest nav weight) and it is the only embodiment that barely
moved (1.24 → 1.04). Size did not help it. Outdoor exploration from a single
camera plausibly has the weakest action-from-vision determinism in the pool — the
same "hardest robot" slot ur5 occupies among the arms.
## What it cost the canonical line
| robot | C-scaled | +joint | Δ |
|---|---|---|---|
| xarm | 0.13 | 0.12 | −0.01 |
| stretch | 0.11 | 0.14 | +0.03 |
| dlr_edan | 0.23 | 0.28 | +0.05 |
| ur5 | 0.90 | 1.02 | +0.12 |
| jaco (HELD-OUT arm) | 0.28 | 0.35 | +0.07 |
| LeKiwi (HELD-OUT base) | 1.67 | **2.10** | **+0.43** |
| **mean, in-training** | 0.34 | 0.39 | +0.05 |
| **mean, HELD-OUT** | 0.97 | 1.22 | +0.25 |

In-training degradation is mild (0.34 → 0.39) and matches what RoboCasa cost at a
comparable new-domain weight. Held-out is where it hurts, and one result there is
worth stating plainly:

**Adding 1.45M frames across five mobile navigation robots made zero-shot transfer
to a mobile manipulator WORSE (LeKiwi 1.67 → 2.10).** The intuition that "more
wheeled bases should help the wheeled held-out robot" is wrong here. LeKiwi is an
SO-100 arm on a kiwi base, and what the nav packs teach is base motion with no arm
at all (their descriptors are literally arm_dof 0, reach 0, gripper 0). The model
appears to be learning "mobile ⇒ do not manipulate", which is the opposite of what
LeKiwi needs. This is the third consecutive measurement in which LeKiwi degrades
as the pool grows (1.07 buggy → 1.67 corrected → 2.10 joint), and the first where
the added data was specifically of its own base class.
So the honest summary: joint training is a clear win for the new embodiments and a
measurable, not catastrophic, tax on the old ones — except for cross-morphology
transfer to a mobile manipulator, which it actively damages.
# GR00T / physical_ai_ft (B200) — first evaluation of a run that was never measured
`AlexWortega/tinyvla-b200-artifacts` shipped a 60k-step, 8×B300 finetune on the
`nvidia/PhysicalAI-Robotics-GR00T-X-Embodiment-Sim` collection (163 LeRobot v2.x
datasets, 214.7M source frames, streamed from the hub). It had **no training log
for the successful run** (the only log in the artifacts is a SIGSEGV from a
different launch), no loss curve, and no metric of any kind. This is its first
evaluation: `scripts/eval_physical_ai.py`, streaming, nothing written to disk.
Regime differs from the whole v2 line — `conditioning: id`, `action_space: native`,
`max_state_dim: 256`, `max_action_dim: 64` — and `state_proj` / `action_in` /
`action_out` were RE-INITIALISED at resume. It is a new action head on the
C-scaled trunk, so **no number here is comparable to our 0.34/0.28**.

## Three things found before any number was produced

**1. The published artifact is a trap.** 167 of 489 tensors carry torch.compile's
`expert._orig_mod.` prefix. LeRobot loads with `strict=False` and only *logs*
missing keys, and `FlowMatchingExpert` zero-inits `action_out` — so loading it the
normal way yields an expert that returns the sampler's own initial Gaussian noise.
Not a crash: a plausible-looking bad number. Measured: un-rekeyed error 7.74 vs
rekeyed 3.33 on the same batch. Fixed locally (`scripts/rekey_checkpoint.py`,
489/489 `strict=True`); the hub artifact is left as-is and documented here.

**2. The realized training mixture is 96.5% one family, not the intended 49%.**
`HubEpisodeStream` samples a *dataset per episode* and then streams the whole
episode. r1_pro episodes are ~37× longer than everyone else's, so the `sqrt(frames)`
weights were multiplied by episode length:
| family | intended (weight) | **realized (samples)** | P(episode unseen) |
|---|---|---|---|
| r1_pro | 49.1% | **96.5%** | **0.0%** |
| single_panda | 19.1% | 1.4% | 94.0% |
| gr1 | 16.6% | 1.0% | 96.2% |
| g1 | 6.4% | 0.3% | 91.2% |
| bi_panda_hand | 4.6% | 0.4% | 88.6% |
| bi_panda_grip | 4.2% | 0.3% | 91.1% |
This is a **sampler bug, not a config choice**: weighting by episode when episode
lengths differ by 37× silently converts a diversity mixture into single-task
training. It also hands us a real held-out set for free — five of six families were
evaluated on episodes ~90% likely never streamed.
**3. Action semantics are per-dim, not per-family.** r1_pro's `action[3:6]` *is*
`state[6:9]` exactly (offset by 3 in the state layout); gr1 mixes 13 absolute dims
with 10 deltas and 21 constants; single_panda is pure delta with 5 dead dims. The
baseline is therefore searched per dim in raw units over lags 0-2, not assumed. It
matters enormously: on r1_pro's absolute dims, hold-still scores 1.08 while
predict-the-mean scores 3.79 — using the latter would have manufactured a 3.5×
"win" out of nothing.

## Results (displacement error ÷ strongest trivial floor; <1 beats it)

4-seed averaged ODE. n = episodes, CI = bootstrap over episodes.

| family | subset | dims | err | B0 mean | B1 hold-still | B2 oracle | **ratio** | 95% CI |
|---|---|---|---|---|---|---|---|---|
| gr1 | abs | 13 | 1.68 | 3.40 | 3.11 | 2.96 | **0.54** | [0.48, 0.65] |
| gr1 | delta | 10 | 19.7 | 57.6 | — | 48.9 | **0.34** | [0.30, 0.40] |
| bi_panda_grip | delta | 14 | 27.8 | 46.8 | — | 93.4 | **0.59** | [0.57, 0.61] |
| bi_panda_hand | abs | 4 | 0.69 | 1.73 | 1.64 | 1.52 | **0.42** | [0.28, 0.45] |
| bi_panda_hand | delta | 16 | 33.0 | 64.4 | — | 84.1 | **0.51** | [0.44, 0.56] |
| single_panda | delta | 6 | 25.7 | 37.1 | — | 52.4 | **0.69** | [0.63, 0.76] |
| r1_pro *(in-training)* | abs | 17 | 0.74 | 3.79 | 1.08 | 1.06 | 0.69 | [0.62, 0.76] |
| r1_pro *(in-training)* | delta | 4 | 13.1 | 45.2 | — | 26.4 | **0.29** | [0.25, 0.35] |
| g1 | abs | 16 | 2.16 | 3.68 | 3.88 | 3.72 | **0.59** | [0.57, 0.81] |
| g1 | delta | 22 | 69.2 | 93.9 | — | 77.1 | **0.74** | [0.63, 0.85] |
**The checkpoint works.** Every family beats its own trivial baseline, and for five
of six that is on episodes ~90% likely never seen in training. Best transfer is on
the delta subsets (0.29-0.59); worst is single_panda delta (0.69) and g1 delta (0.74).

**Read the absolute-dim rows with suspicion.** On every ABS subset the oracle
"repeat the first action" lands within 10% of the trivial floor (r1_pro 1.06 vs
1.08; gr1 2.96 vs 3.11; g1 3.72 vs 3.68). There is almost no headroom between doing
nothing and the best any dynamics-free predictor can do, so a ratio of 0.6-0.7 there
is a much weaker claim than the same number on a delta subset. r1_pro's 0.69 — the
family that got 96.5% of training — is the least meaningful number in the table.

## Controls

- **Rekey (B)**: as-published 7.74 vs rekeyed 3.33 — the rename is load-bearing.
- **Flow loss (E)**: `L/L₀ = 0.124` on gr1. The expert learned the velocity field,
  independent of the ODE and of the metric design. This is the closest thing to the
  missing training log.
- **Vision vs proprioception (D)**: swapping in a donor observation moves the
  prediction by **0.184 for state** and only **0.042 for images**, against a
  prediction scale of 0.404 — and the donor states differed by just 0.047. The model
  is roughly 4× more sensitive to proprioception than to visual content. Swapping the
  embodiment id is the biggest perturbation of all (ratio 0.245 → 0.747), so under
  `conditioning: id` the identity token is doing much of the work.
- **ODE sampling (C)**: single-draw seed sensitivity `S = 1.22` — two draws disagree
  with each other as much as either disagrees with truth. The 10-step Euler
  integration does not contract the initial noise, so a single sample is
  noise-dominated. Averaging helps and saturates: gr1 abs 0.539 → 0.475 → 0.423 →
  0.419 and gr1 delta 0.236 → 0.198 → 0.176 → 0.168 at 1 / 4 / 16 / 64 seeds. **The
  headline table above uses 4 seeds and is therefore ~12% pessimistic**; the
  converged values are ~0.42 and ~0.17 for gr1.

## Caveats

- r1_pro starts are confined to the first 600 strided frames of an episode
  (prefix decode, to avoid pulling 190 MB and decoding 26k frames per episode), so
  its numbers cover the approach phase of the task, not the whole task.
- Nothing is strictly held out — no exclusion list exists — the hold-out claim is the
  probabilistic one in the table above, and it assumes the 8-GPU launch recorded in
  `metadata/commands.txt`. On a 1-GPU run every P(unseen) rises (r1_pro 0.0% → 0.8%).
- Normalized units throughout: per-dataset z-scores make absolute magnitudes
  incomparable across datasets, but the ratio is comparable. No mm are reported for
  the panda families — their native layouts are an unknown mix and no FK exists.

## GR00T checkpoint on held-out LeKiwi — zero-shot to an unseen embodiment

LeKiwi (SO-100 arm on a 3-wheel kiwi base, 9 native dims: 5 joint targets + gripper
+ x_mm/y_mm/theta) is not one of the six GR00T families and has no embodiment id.
Under `conditioning: id` + `action_space: native` there is no reason its dim k means
what any GR00T robot's dim k means, so the checkpoint was given every advantage:
each trained id 10..15 tried and the **best** reported, plus id 0 as the "no
matching identity" reference. Same 4-seed ODE averaging, same per-dim baseline
search, held-out tail episodes.

What id 0 actually is matters, and it is not what it first looks like. `tv2_C_scaled`
is variant **C = `conditioning: morph`**, so its `embodiment_emb` table never received
a gradient in that run: rows 0-9 have per-row std 0.93-1.05, norm ~22.6 (= sqrt(512))
and pairwise cosine -0.03, i.e. they are exactly `nn.Embedding`'s default N(0,1)
initialisation. The GR00T run then moved only rows 10-15, and only by 0.006-0.036
mean-abs on entries of scale ~1. **So id 0 is a pure random 512-d vector, and ids
10-15 are random vectors plus a small trained nudge.**

The dim search recovers the layout correctly without being told: 5 arm joints and
x_mm/y_mm map to their own state dims (ABS), gripper and theta come out DELTA.

| emb id | abs (7 dims) | delta (2 dims) |
|---|---|---|
| 10 gr1 | 1.22 | 1.07 |
| 11 bi_panda_grip | 1.21 | 1.09 |
| 12 bi_panda_hand | 1.33 | 1.56 |
| 13 single_panda | 1.24 | 1.08 |
| 14 r1_pro | 1.23 | 1.07 |
| **15 g1** | **1.18** | **0.88** |
| **0 (never trained)** | **1.22** | **0.92** |
| oracle repeat-A₀ | 0.91 | 0.54 |

**It does not transfer.** On the absolute dims every id lands at 1.18-1.33 — worse
than simply holding the current pose for 5 seconds. Only the delta pair (gripper,
theta) creeps under 1.0, and even there a dynamics-free predictor that repeats the
first ground-truth action scores 0.54 against the model's 0.88. The model loses to
a trivial predictor on both subsets.

**The control is the interesting part: a pure random identity vector (id 0:
1.22 / 0.92) does as well as the best trained one (id 15: 1.18 / 0.88).**

That is NOT because the id channel is inert. On an in-distribution gr1 batch the
channel is clearly functional — error vs ground truth is 0.223 with the correct id
10, 0.263 with another trained id, 0.299 and 0.331 with random ids 0 and 1. The
model reads the code and the right code is measurably better.

The channel is a **lookup table with no entry for LeKiwi and no way to synthesise
one**. Swapping between two random codes changes the prediction by 0.174 and between
two trained codes by 0.194 — comparable magnitudes, so there is no special "trained
subspace" a new robot could be projected into. For an unseen robot every available
code is somebody else's, and a random one is no worse.

That looks like the argument for the morphology descriptor — a 16-number physical
descriptor can be *computed* for an unseen robot where a learned id can only be looked
up. **So it was tested, and the argument does not survive the test.**

### Converting the GR00T checkpoint to morphology conditioning does NOT fix LeKiwi

The checkpoint has no `morph_encoder` at all (the run discarded C-scaled's, since its
config is `conditioning: id`). Two steps were tried:

1. **Graft only** — set `conditioning: morph`, transplant tv2_C_scaled's trained
   morph encoder. Result 1.30 / 1.11, *worse* than id. Expected: the expert spent 60k
   steps reading a 1-token id and has never seen 2 morph tokens from another model's
   encoder. (All six ids give identical numbers here, which confirms the morph path is
   actually active and the id is ignored.)
2. **Train it to read the descriptor** — 6k streaming steps with the six GR00T family
   descriptors (`scripts/train_groot_morph.py`), everything else already trained.

| conditioning | abs (7 dims) | delta (2 dims) |
|---|---|---|
| id, best of six (oracle) | 1.18 | 0.88 |
| morph, grafted only | 1.30 | 1.11 |
| morph, 1k steps | **1.13** | 0.83 |
| morph, 2k steps | 1.16 | 1.00 |
| morph, 4k steps | **1.12** | 0.87 |
| morph, 6k steps | 1.17 | 0.93 |
| oracle repeat-A₀ | 0.91 | 0.54 |

**The improvement is 1.18 -> 1.12 at best, and it is not monotonic** (1.13, 1.16,
1.12, 1.17 across checkpoints) — that spread is noise, not convergence. Both
conditioning schemes stay above 1.0 on the absolute dims, i.e. worse than holding the
current pose, and both lose to a dynamics-free predictor that repeats the first
ground-truth action.

So the honest conclusion is narrower than the one above: **being computable is
necessary but not sufficient.** A descriptor gives you a well-defined input for an
unseen robot, which an id cannot; it does not by itself give you transfer. This model
was trained on six morphologies — three panda variants, two humanoids and an R1-Pro —
none of them a small 5-DOF arm on a wheeled base, and no conditioning scheme extracts
a LeKiwi policy from that.

**Confound, stated rather than buried:** the morph run had to reweight the mixture,
because the per-episode sampler starved the loader (every draw pulled a ~190 MB
r1_pro episode and the first batch never arrived). The correction over-shot — realized
shares became gr1 0.67 / panda_single 0.23 / g1 0.04 / bimanual 0.035 / r1_pro 0.02 —
so the morph model saw a different mixture than the id model. This is not a matched
pair, and a matched-pair version would need retraining the id model on the same
reweighted stream.
That id 15 (g1: a humanoid arm commanded in joint space) is the best of the six is
the one sensible signal in the table — it is the closest morphology to an SO-100 arm
also commanded in joint space.
Not comparable to our LeKiwi numbers (C-scaled 1.67, +joint 2.10): those are
integrated endpoint error in canonical base-frame EE space, these are native joint
space. The only shared, comparable statement is that **both lines fail on LeKiwi**,
each against its own trivial floor.
## Closed-loop eval was single-draw, and that was costing more than any ablation
`predict_action_chunk` starts from `torch.randn` and nothing seeded it, so every
closed-loop rollout was a lottery. Caught by accident: the SAME checkpoint with the
SAME settings scored **4/20 and then 8/20** on consecutive runs. Every previously
reported closed-loop number was one draw from that distribution.
Fixed by averaging the flow ODE over 4 noise draws per decision and seeding each
decision deterministically (`--pred-seeds`, default 4). The eval is now reproducible
— three consecutive runs returned exactly 70.0%.
**How much it was costing depends on how converged the checkpoint is:**
| checkpoint | single draw | 4-seed averaged |
|---|---|---|
| new run, 5k steps (n=20) | 20% / 40% (two runs) | **70%** |
| old run, 5k steps (n=20) | 40% | **60%** |
| old run, 30k steps (n=50) | 70% | 68% |
At 30k the averaging changes nothing (70 vs 68, within noise) — the flow field is
sharp enough that one draw is already near the conditional mean. At 5k it is worth
20-30 points, because an under-trained field is diffuse and the noise compounds over
a 500-step rollout. **So the published 70% for the converged model stands**, but
every intermediate-checkpoint number in this document was noise-dominated, and the
dual-rate comparison (70 vs 68) was never precise enough to support a claim either
way.
## Gripper command channel — the honest verdict so far
The ceiling gain is real and proven: GT round-trip 8/10 with a decoded command vs
**9/10 with the carried command**, matching raw GT exactly.
Whether the policy converts that ceiling into success is not yet established:
| comparison (all 4-seed averaged) | result |
|---|---|
| new run @5k vs old run @5k, n=20 | 70% vs 60% |
| within the new checkpoint: decode `canon` vs `threshold`, n=20 | 70% vs 70% |
+10 points at a matched step is two episodes out of twenty — not significant. And
the decode-mode ablation on the *same* checkpoint shows no difference at all, which
argues the channel is not the binding constraint at this stage: the model's predicted
openness is already good enough that thresholding it matches reading the command.
The decisive number is the new run at 30k against the old run's 68% at n=50, and that
needs the training to finish (currently ~10k/30k).