Add section 12: RoboCasa fine-tune (0/15 -> 7/15 on trained task, 0/45 held-out); qualify section 11's rotation-frame hypothesis
3938c9e verified | # TinyVLA — corrected evaluation and ablations | |
| Measured independently of the numbers in the model card, on a re-implemented | |
| harness that separates the three rates and logs them instead of inferring them. | |
| All closed-loop numbers below are **paired**: every configuration runs the same | |
| (task, init state) episodes, and the flow-matching noise is seeded per | |
| (episode, step), so two rows differ only by the thing being ablated. | |
| Hardware: 4x Tesla V100-SXM2-32GB (sm_70), fp32, driver 535 / CUDA 12.2, | |
| torch 2.6.0+cu124, transformers 5.5.4, lerobot 0.6.1. Suite: `libero_spatial`. | |
| Generated 2026-08-25. | |
| **The headline dual-rate comparison in the model card is not reproduced.** With the | |
| clock fixed and 500 paired episodes per configuration, no refresh rate differs | |
| significantly from any other. What does survive is the efficiency claim, and it | |
| survives causally: see sections 3 and 4. | |
| --- | |
| ## 1. The refresh schedule was derived from the wrong frame rate | |
| `HuggingFaceVLA/libero` declares `fps = 10`, but its frames are native 20 Hz. The | |
| dataset has 1693 episodes and 273,465 frames, i.e. **161.5 frames per episode**; | |
| LIBERO demos run about 8 s, giving 20.2 Hz, which matches the environment | |
| (`lerobot.envs.configs` sets `fps: int = 20 # Must match robosuite's default | |
| control_freq (20 Hz)` and `LiberoEnv(control_freq=20)`). | |
| Every place that expresses staleness in seconds divides by the declared 10: | |
| | Site | What it controls | Effect | | |
| |---|---|---| | |
| | `scripts/eval_libero.py` (`fps = meta.fps`) | closed-loop refresh schedule | refreshes **2x more often** than the flag says | | |
| | `src/tinyvla/data/mixture.py` (`staleness_max_s * ds.fps`) | training staleness augmentation | nominal 2 s is **0.99 real seconds** | | |
| | `scripts/eval_offline.py` (`int(stale_s * ds.fps)`) | offline staleness curve | `--stale-s 2` is **1 real second** | | |
| Consequences for claims in the card: | |
| * the offline result "stale-latent 2 s == fresh (0.772 vs 0.772)" was measured at | |
| **1 real second**; | |
| * the released checkpoint has never been trained on staleness beyond ~1 real | |
| second, so 2 s is outside its training distribution. | |
| Verified by frame count, not by reading: with the declared fps a nominal | |
| `staleness_max_s=2.0` samples a maximum delay of 20 frames (1.00 s at 20 Hz); | |
| with the true rate it samples 40 frames (2.00 s). | |
| ### The published command does not enable dual-rate | |
| With `--n-action-steps 10 --refresh-s 1.0`, the refresh period works out to 10 | |
| environment steps — exactly the chunk replan interval. Counting the actual calls | |
| over two episodes: | |
| | `--refresh-s` | `_semantic_latent` calls | chunk replans | ratio | | |
| |---|---|---|---| | |
| | 1.0 | 20 | 20 | **1.00** | | |
| | 4.0 | 5 | 20 | 0.25 | | |
| So the two rows of the card's headline table run the semantic path at the same | |
| cadence and differ only in the random draw. | |
| --- | |
| ## 2. Reproducing the shipped configuration | |
| The published command, re-run through the new harness with its rates measured | |
| (100 episodes, init states 0-9): | |
| | | | | |
| |---|---| | |
| | success | **87/100 = 87.0%** | | |
| | environment | 20.0 Hz | | |
| | fast path (chunk replan) | 2.079 Hz | | |
| | semantic refresh | 2.079 Hz | | |
| | mean latent age | 0.0 s | | |
| That is at or slightly above the higher of the card's two published numbers | |
| (86 and 79), while running both paths at ~2 Hz, which is what the flags actually | |
| produce. The conventions are therefore preserved end to end; what changed is that | |
| the rates are now measured rather than assumed. | |
| --- | |
| ## 3. Rates separated, 500 paired episodes each | |
| `libero_spatial` has 50 init states per task, so 10 tasks x 50 states = 500 | |
| episodes is the full deterministic sweep of the suite. Reference row is | |
| full-rate; `p` is an exact two-sided McNemar test on the discordant pairs. | |
| | Configuration | Success | 95% CI | fast Hz | sem Hz | mean age | sem calls/ep | delta | paired 95% CI | flips | McNemar p | | |
| |---|---|---|---|---|---|---|---|---|---|---| | |
| | **full-rate** | **426/500 = 85.2%** | 81.8–88.0 | 20.00 | 20.00 | 0.00 s | 171.0 | — | — | — | — | | |
| | **dual-rate 5 Hz** | **437/500 = 87.4%** | 84.2–90.0 | 20.00 | 5.06 | 0.07 s | 41.1 | +2.2 | +0.0…+4.4 | +21/-10 | 0.071 | | |
| | **dual-rate 2 Hz** | **425/500 = 85.0%** | 81.6–87.9 | 20.00 | 2.07 | 0.22 s | 17.7 | -0.2 | -2.8…+2.2 | +20/-21 | 1.000 | | |
| | **dual-rate 1 Hz** | **422/500 = 84.4%** | 81.0–87.3 | 20.00 | 1.08 | 0.46 s | 9.2 | -0.8 | -3.2…+1.6 | +17/-21 | 0.627 | | |
| | **dual-rate 0.5 Hz** | **426/500 = 85.2%** | 81.8–88.0 | 20.00 | 0.58 | 0.93 s | 4.7 | +0.0 | -3.0…+3.0 | +30/-30 | 1.000 | | |
| | **open-loop chunk** | **437/500 = 87.4%** | 84.2–90.0 | 2.08 | 2.08 | 0.00 s | 16.5 | +2.2 | -1.0…+5.6 | +41/-30 | 0.235 | | |
| "flips" counts episodes this configuration wins that full-rate lost, and the reverse. | |
| **No difference is significant.** The largest gap (dual-rate 5 Hz, +2.2 points) | |
| has p = 0.071. Per-task success agrees within +/-2 episodes on all ten tasks. The | |
| "dual-rate beats full-rate" claim, and the temporal-regularization explanation | |
| offered for it, are not supported at this sample size. | |
| **What is supported:** dropping the semantic path from 20 Hz to 0.5 Hz cuts | |
| semantic calls from 171 to 4.7 per episode, a **36x reduction**, at a mean latent | |
| age of 0.93 s, and costs exactly nothing (85.2% vs 85.2%, 30 flips each way, | |
| p = 1.0). That is the useful half of the architectural claim, now measured | |
| causally rather than inferred from a two-row table. | |
| --- | |
| ## 4. Success against measured compute | |
| Amortized with measured per-call cost and the rates each run actually ran at. | |
| | Configuration | Success | fast Hz | sem Hz | GFLOPs / robot-second | vs full-rate | compute ms / robot-second | real time on one V100 | | |
| |---|---|---|---|---|---|---|---| | |
| | full-rate | 85.2% | 20.00 | 20.00 | 4493 | 1.00x | 6140 | no | | |
| | dual-rate 5 Hz | 87.4% | 20.00 | 5.06 | 3134 | 0.70x | 3632 | no | | |
| | dual-rate 2 Hz | 85.0% | 20.00 | 2.07 | 2862 | 0.64x | 3130 | no | | |
| | dual-rate 1 Hz | 84.4% | 20.00 | 1.08 | 2771 | 0.62x | 2963 | no | | |
| | dual-rate 0.5 Hz | 85.2% | 20.00 | 0.58 | 2726 | 0.61x | 2879 | no | | |
| | open-loop chunk | 87.4% | 2.08 | 2.08 | 467 | 0.10x | 638 | yes | | |
| Two things fall out of this table. | |
| **Caching the semantic latent saves at most ~39% of compute, not 97%,** because the | |
| floor is set by the fast path (section 6). | |
| **Plain action chunking beats the dual-rate decomposition on both axes.** Executing | |
| 10 actions per replan is 9.6x cheaper than the best dual-rate | |
| configuration, scores 2.2 points higher, and is the only configuration that runs in | |
| real time on this hardware. Since the open-loop baseline exists precisely to | |
| separate "the gain comes from dual-rate" from "the gain comes from chunking", on | |
| this benchmark the answer is chunking. | |
| **Neither the semantic rate (20 -> 0.5 Hz) nor the replan rate (20 -> 2 Hz) | |
| changes success.** On its own that is ambiguous: it could mean semantics tolerate | |
| staleness, or it could mean nothing in this scene changes fast enough to matter. | |
| Section 5 separates the two, and the answer is the first one - the suite is not | |
| quasi-static at all, it is specifically insensitive to *semantic* staleness. | |
| --- | |
| ## 5. The asymmetry: staleness on each path, separately | |
| Constant-age probes rather than a periodic cache, so the age on the x-axis is the | |
| age, not the mean of a sawtooth. 150 paired episodes per cell, compared against the | |
| same episodes of the full-rate run. Proprioception is fresh throughout. | |
| **Staling the semantic latent while geometry stays fresh:** | |
| | latent age | Success | 95% CI | delta | paired 95% CI | flips | McNemar p | | |
| |---|---|---|---|---|---|---| | |
| | 0.25 s | 128/150 = 85.3% | 78.8–90.1 | +1.3 | -4.0…+6.7 | +9/-7 | 0.8036 | | |
| | 0.5 s | 129/150 = 86.0% | 79.5–90.7 | +2.0 | -3.3…+7.3 | +10/-7 | 0.6291 | | |
| | 1 s | 128/150 = 85.3% | 78.8–90.1 | +1.3 | -4.0…+6.7 | +9/-7 | 0.8036 | | |
| | 2 s | 126/150 = 84.0% | 77.3–89.0 | +0.0 | -5.3…+5.3 | +8/-8 | 1.0000 | | |
| | 4 s | 125/150 = 83.3% | 76.6–88.4 | -0.7 | -6.0…+4.7 | +8/-9 | 1.0000 | | |
| **Staling the geometry while the semantic latent stays fresh:** | |
| | frame age | Success | 95% CI | delta | paired 95% CI | flips | McNemar p | | |
| |---|---|---|---|---|---|---| | |
| | 0.25 s | 35/150 = 23.3% | 17.3–30.7 | -60.7 | -68.7…-52.0 | +2/-93 | 0.0000*** | | |
| | 0.5 s | 8/150 = 5.3% | 2.7–10.2 | -78.7 | -85.3…-72.0 | +0/-118 | 0.0000*** | | |
| | 1 s | 4/150 = 2.7% | 1.0–6.7 | -81.3 | -87.3…-74.7 | +1/-123 | 0.0000*** | | |
| | 2 s | 2/150 = 1.3% | 0.4–4.7 | -82.7 | -88.7…-76.0 | +0/-124 | 0.0000*** | | |
| | 4 s | 0/150 = 0.0% | 0.0–2.5 | -84.0 | -89.3…-78.0 | +0/-126 | 0.0000*** | | |
| **Both stale together:** | |
| | both aged | Success | 95% CI | delta | paired 95% CI | flips | McNemar p | | |
| |---|---|---|---|---|---|---| | |
| | 0.5 s | 6/150 = 4.0% | 1.8–8.5 | -80.0 | -86.7…-72.7 | +2/-122 | 0.0000*** | | |
| | 2 s | 1/150 = 0.7% | 0.1–3.7 | -83.3 | -89.3…-77.3 | +0/-125 | 0.0000*** | | |
| **Controls - is the semantic path used at all?** | |
| | control | Success | 95% CI | delta | paired 95% CI | flips | McNemar p | | |
| |---|---|---|---|---|---|---| | |
| | latent zeroed | 16/150 = 10.7% | 6.7–16.6 | -73.3 | -80.7…-66.0 | +1/-111 | 0.0000*** | | |
| | latent from a pixel-shuffled frame | 127/150 = 84.7% | 78.0–89.6 | +0.7 | -6.0…+7.3 | +13/-12 | 1.0000 | | |
| ### Reading | |
| At four seconds of staleness the geometric path costs **84.0 points** | |
| while the semantic path costs **0.7**, which is not distinguishable from | |
| zero - no semantic cell reaches significance at any age out to 4 s, and their | |
| paired intervals all straddle 0. (A ratio of the two is not quoted: the denominator | |
| is consistent with zero, so it would be arbitrarily large and meaningless.) Every | |
| geometric cell is significant at p < 0.0001, and a quarter of a second - five frames | |
| at 20 Hz - already costs 60.7 points. The both-stale | |
| column tracks the geometry column, so geometric staleness accounts for the damage | |
| and semantic staleness adds nothing on top. | |
| The control rules out the obvious alternative explanation. Zeroing the latent | |
| costs 73.3 points, so the policy is not ignoring the semantic path - it needs | |
| it, it just does not need it recent. That is the claim this architecture rests on, | |
| and here it is separated from "the benchmark is static" and from "the latent is | |
| decorative". | |
| **A second control narrows what the semantic path is actually doing.** Shuffling | |
| the pixels of the frame fed to it - destroying all spatial structure while keeping | |
| the same pixel statistics - costs 0.7 points and is not significant | |
| (p = 1.00). Zeroing the latent outright costs 73.3. So whatever the | |
| semantic path contributes is not fine-grained scene layout: something coarse | |
| (a colour or object cue, or the interaction with the language tokens) survives a | |
| full spatial scramble, but nothing survives replacing the latent with zeros. | |
| ### Is the geometric collapse about old pixels, or a self-inconsistent observation? | |
| The grid above stales the pixels while proprioception stays fresh, which hands the | |
| policy an observation whose parts disagree in time. If *that* disagreement were the | |
| problem, holding proprioception back by the same amount - keeping the observation | |
| internally consistent, which is what caching the vision tower on a real robot would | |
| actually look like - should help. | |
| | age | pixels-only stale | pixels + proprio stale | delta | flips | McNemar p | | |
| |---|---|---|---|---|---| | |
| | 0.25 s | 23.3% | 14.0% | -9.3 | +6/-20 | 0.0094** | | |
| | 0.5 s | 5.3% | 0.0% | -5.3 | +0/-8 | 0.0078** | | |
| | 1.0 s | 2.7% | 0.7% | -2.0 | +1/-4 | 0.3750 | | |
| All 3 ages tested, all in the same direction: making the observation internally consistent does not help, and at 2/3 of them it is significantly *worse*. So the collapse is not explained by the observation disagreeing with itself — staling proprioception on top of the pixels adds damage rather than removing it. Holding proprioception back to make a cached vision tower's input "agree with itself" is not a viable mitigation: geometric staleness costs, and adding more of it costs more. | |
| --- | |
| ## 6. System profile | |
| Measured on an idle machine, batch 1, fp32, 30 iterations (5 on CPU). No edge | |
| device is attached, so CPU stands in as the constrained-device proxy; no | |
| Jetson or Raspberry Pi number is reported because none was measured. | |
| | Path | Params | GFLOPs / call | V100 p50 | V100 p95 | CPU p50 | | |
| |---|---|---|---|---|---| | |
| | semantic (slow) | 604.5M | 91 | 168.0 ms | 171.67 ms | 412.63 ms | | |
| | fast (spatial + expert) | 38.9M | 134 | 138.11 ms | 141.23 ms | 577.49 ms | | |
| | both (no cache) | — | 225 | 324.06 ms | 329.96 ms | 982.55 ms | | |
| | vision tower, one camera | 100.6M | 45 | 18.95 ms | 19.26 ms | 146.0 ms | | |
| Peak VRAM 2.61 GB. | |
| **Parameter count and FLOPs point in opposite directions.** The 38.9M fast path | |
| costs more per call (134 GFLOPs) than the 604.5M semantic path | |
| (91 GFLOPs), because the fast path pushes **both cameras through the shared | |
| 100.6M vision tower on every step** (2 x 45 | |
| GFLOPs). The tower's parameters are counted under the semantic branch; its FLOPs | |
| are paid by the fast branch. | |
| So the "~140M online per tick" figure is accurate as a parameter count and | |
| misleading as a cost: at 20 Hz control the fast path alone needs | |
| 2782 ms of compute per robot second on a V100. The remaining | |
| headroom is in the visual encoder, not in the language stack. | |
| --- | |
| ## 7. Adaptive refresh — threshold calibration | |
| A drift gate that refreshes when the pooled cam0 spatial tokens have moved away | |
| from those at the last refresh (plus a proprio term and age bounds). The signal | |
| is free: `_encode_spatial` already caches that projection. Calibration so the gate | |
| can be compared against fixed schedules at a **matched number of semantic calls** | |
| rather than at a matched frequency: | |
| | threshold tau | measured semantic Hz | | |
| |---|---| | |
| | 0.05 | 5.24 | | |
| | 0.15 | 2.79 | | |
| | 0.30 | 1.55 | | |
| | 0.50 | 1.16 | | |
| ### Fixed vs. adaptive, at a matched budget | |
| 150 paired episodes per gate, same episodes as `full_rate`'s first 150. `random` | |
| uses the same expected call budget as `fixed` (both target ~1 Hz) - the point of | |
| including it is that an adaptive gate beating a *fixed* schedule is a weak result | |
| on its own, since simply refreshing at different points in the episode could look | |
| like it "learned something" even from noise; `random` is the actual null hypothesis | |
| a real adaptive gate has to beat. | |
| One bug found and fixed while building this table, worth recording: the model's | |
| own `_conditioning` silently recomputes the semantic latent internally whenever | |
| `batch["semantic_latent"]` is absent (which is exactly the state before an | |
| episode's first scheduled refresh), and that internal recompute was being counted | |
| by the same instrumentation that counts the harness's own scheduled calls. Every | |
| schedule except `random` happens to force a refresh within its first `max_period_s` | |
| regardless (so this was invisible for them), but `random` has no such cap and | |
| could go arbitrarily long without a hit - one unlucky early stretch of an episode | |
| meant every single step silently double-counted as a "semantic call", which is | |
| what produced 3.5-4x inflated rates on the first pass. Fixed by forcing an | |
| explicit bootstrap call at step 0 of every episode (matching what every other | |
| schedule already guaranteed) and capping `random`'s tail at 4 s like the others. | |
| Regression-checked against the periodic-schedule numbers in section 3 (bit-for-bit | |
| identical) before any of the numbers below were produced. | |
| | gate | measured Hz | calls/ep | Success | 95% CI | delta | flips | McNemar p | | |
| |---|---|---|---|---|---|---|---| | |
| | fixed 1 Hz | 1.08 | 9.1 | 128/150 = 85.3% | [78.8,90.1] | +1.3 | +9/-7 | 0.8036 | | |
| | random ~1 Hz (null hypothesis) | 1.18 | 10.0 | 129/150 = 86.0% | [79.5,90.7] | +2.0 | +8/-5 | 0.5811 | | |
| | drift, tau=0.30 | 1.43 | 10.2 | 122/150 = 81.3% | [74.3,86.8] | -2.7 | +6/-10 | 0.4545 | | |
| | drift, tau=0.50 | 0.95 | 6.7 | 127/150 = 84.7% | [78.0,89.6] | +0.7 | +8/-7 | 1.0000 | | |
| | oracle (gripper transitions) | 0.51 | 5.7 | 124/150 = 82.7% | [75.8,87.9] | -1.3 | +6/-8 | 0.7905 | | |
| flips: episodes this gate wins that `full_rate` lost, and the reverse. | |
| None of the five is distinguishable from full-rate at this sample size, matching | |
| section 3's finding that semantic refresh rate does not move success on this | |
| suite - but every one of them does it at 5.7-10.2 semantic calls per episode | |
| against full-rate's 171, a 17-30x reduction. Since even `random` matches `fixed` | |
| here, the honest reading is that nothing about *when* the semantic path refreshes | |
| matters on `libero_spatial` at all (consistent with section 5's causal finding | |
| that only spatial staleness costs anything) - which also means this suite cannot | |
| show whether the drift signal is doing anything useful; that question needs the | |
| perturbation suite (dynamic scene changes), not this one. | |
| --- | |
| ## 8. Not yet included | |
| Running or pending, and deliberately not summarized here until finished: the | |
| perturbation suite (instruction swap, paraphrase, mid-episode instruction change, | |
| occlusion, latency ramp), retrained staleness and latency-metadata variants, and | |
| same-harness baselines beyond the checkpoints in section 9. Also running now: | |
| `tinyvla2/pretrain_C` fine-tuned on LIBERO with the same recipe as section 10's | |
| `pretrain_B` run (the morphology-conditioning caveat from section 9 applies here | |
| too - this fine-tune only exercises the embodiment-ID path, `morph_encoder` is | |
| still dropped on load). The fresh/stale grid (section 5), the consistent-lag check | |
| (section 5, three ages), and the gate comparison (section 7) are all done: 12 grid | |
| cells, two controls, three consistency cells, five gates - 500 + 12x150 + 2x150 + | |
| 3x150 + 5x150 = 3,500 closed-loop episodes. | |
| ## 9. Other checkpoints in the same repo, same harness | |
| Three checkpoints that predate the LIBERO fine-tune, evaluated closed-loop on | |
| `libero_spatial` with the exact same conventions and no ablation (full-rate, | |
| 100 episodes = init states 0-9). None of them were fine-tuned on LIBERO, so this | |
| sizes how much of the 85.2% full-rate number is the fine-tune rather than the | |
| shared pretrain. | |
| | Checkpoint | What it is | Success | 95% CI | | |
| |---|---|---|---| | |
| | `checkpoints/stage2_final` | SO-100/101 + Bridge mixture, frozen backbone | 0/100 = 0.0% | 0.0-3.7 | | |
| | `checkpoints/stage3_final` | + spatial distillation | 0/100 = 0.0% | 0.0-3.7 | | |
| | `tinyvla2/pretrain_A` | TinyVLA-2 ablation, variant A (native action space) | 0/100 = 0.0% | 0.0-3.7 | | |
| | `tinyvla2/pretrain_B` | TinyVLA-2 ablation, variant B (canonical action space) | 0/100 = 0.0% | 0.0-3.7 | | |
| **`tinyvla2/pretrain_A` scores 0.0% on LIBERO** (0/100), consistent with stage2/stage3: none of the three pre-fine-tune checkpoints do anything useful on this suite. This is a different model from the one the rest of this document ablates — it is the *control* arm of a separate cross-embodiment ablation (variants A/B/C), not a LIBERO checkpoint. Its own model card already reports it failing cross-embodiment transfer on held-out `jaco` (endpoint error 1392 mm vs. a 128 mm zero-prediction floor, n=8) — a result from the author, not independently reproduced here since the held-out-embodiment harness this document uses is LIBERO/MetaWorld/SO-101, not the jaco eval that number came from. `conditioning`, `morph_tokens`, and `action_space` in its config are not implemented in `modeling_tinyvla.py` as of this checkpoint family's release — they were added to `configuration_tinyvla.py` here as pass-through fields so the checkpoint would load, since variant A's own documentation states it is architecturally identical to the embodiment-ID scheme already implemented (`EmbodimentEmbedding`). Whether that also holds for variants B and C is unverified. | |
| **`tinyvla2/pretrain_B` scores 0.0% on LIBERO** (0/100). Same story as A: this is not a LIBERO checkpoint either, and pre-fine-tune performance on this suite tells you nothing about the A-vs-B question, which is about zero-shot transfer to held-out embodiments, not about LIBERO. B differs from A only in `action_space` (canonical vs. native) - same seed, mixture, weights, and step count per both checkpoints' own model cards. B's card reports a mixed zero-shot result: canonical action space is decisive on `jaco` (137 mm vs. A's catastrophic 1392 mm) but *loses* to native on `stretch` (195 mm vs. 150 mm) - so canonicalization is not a strict win, and the authors flag the comparison itself as confounded by differing normalization, deferring the decisive test to few-shot adaptation efficiency once variant C lands. Not independently reproduced here for the same reason as A: this document's harness is LIBERO/MetaWorld/SO-101, not the jaco/stretch eval those numbers came from. | |
| **`tinyvla2/pretrain_C` was not run through this harness.** Its checkpoint carries four trained tensors (`morph_encoder.mlp.{0,2}.{weight,bias}`, a 16-to-512-to-1024 MLP producing the 2 morphology-conditioning tokens the README describes) that `modeling_tinyvla.py` as published has no forward-pass code path for, and the morphology descriptor file its README points to (`configs/morphology/descriptors.yaml`) is not in the repository. Loading it anyway was tested directly: the loader logs `Unexpected key(s) ... morph_encoder.mlp.*` and silently drops all four tensors, and inference would fall back to `embodiment_emb(emb_id)` - the exact conditioning path the README states C replaces. Reporting a LIBERO number from that would measure a model C was never designed to be, not the checkpoint as released. So this document cites only what the author has already published (`tinyvla2/RESULTS.md`, `tinyvla2/fewshot_results.json`), unverified here: | |
| | jaco FT episodes | A (native+ID) | B (canonical+ID) | C (canonical+morph) | | |
| |---|---|---|---| | |
| | 0 (zero-shot) | 1377 mm | 155 mm | 178 mm | | |
| | 100 | 6919 mm | 215 mm | 218 mm | | |
| | 500 | 5667 mm | 198 mm | 197 mm | | |
| (GT trajectories ~210 mm, zero-prediction floor ~128 mm, n=20 held-out jaco episodes.) The author's own verdict: canonical action space (B, C) beats native (A) by 10-45x at every budget and is decisive; morphology tokens (C) show no measurable advantage over a learned ID (B) once actions are canonical - "the win is action-space unification, not morphology conditioning." Native additionally *diverges* under few-shot fine-tuning (A gets worse with more FT episodes, not better), which the author attributes to re-learning a fresh embodiment-ID against a native action distribution. | |
| --- | |
| ## 10. Fine-tuning pretrain_B and pretrain_C on LIBERO | |
| Section 9's `pretrain_B` row asks what a cross-embodiment pretrain is worth on | |
| LIBERO with no fine-tuning at all (nothing - 0%, same as every other pre-fine-tune | |
| checkpoint). This section asks the useful version of that question: swap the | |
| *starting checkpoint* for `ft4`'s recipe and hold everything else fixed - what does | |
| LIBERO fine-tuning look like starting from the canonical cross-embodiment pretrain | |
| instead of `stage3_distill`? | |
| **Recipe, unmodified from `configs/libero_ft4.yaml` except the resume point:** | |
| unfreeze vision tower + language stack, backbone at 0.1x lr, 40,000 steps, | |
| `resume_from: tinyvla2/pretrain_B` (and separately `pretrain_C`). | |
| `num_embodiments=16` and each pretrain's | |
| `conditioning`/`morph_tokens`/`action_space` fields carried through so the resume | |
| loads with 0 missing / 0 unexpected keys (verified directly before launching). | |
| fp16 instead of bf16 (V100, sm_70); effective batch 48 via 4-way data-parallel | |
| instead of one A6000, matching ft4's effective batch. Final training loss 0.178, | |
| comparable to ft4's reported 0.066 - 500-episode full-rate closed-loop eval, | |
| paired against the same episodes as ft4's own full-rate row (section 3). | |
| `pretrain_C` got the identical treatment. One caveat specific to C: its | |
| `morph_encoder` conditioning has **no forward-pass implementation** in this | |
| codebase, so a zero-shot evaluation of C would silently measure a *different* | |
| model than the one the checkpoint describes, and none is reported here. Fine-tuning | |
| is still legitimate — the backbone unfreezes either way, so `embodiment_emb` | |
| receives real gradient — and that is what the C row below measures. | |
| | | Success | 95% CI | delta vs. ft4 | paired 95% CI | flips | McNemar p | | |
| |---|---|---|---|---|---|---| | |
| | ft4 (from stage3_distill) | 426/500 = 85.2% | — | — | — | — | — | | |
| | **B fine-tuned (from pretrain_B)** | 345/500 = 69.0% | [64.8,72.9] | -16.2 | [-21.0,-11.4] | +40/-121 | 0.0000*** | | |
| | **C fine-tuned (from pretrain_C)** | 372/500 = 74.4% | [70.4,78.0] | -10.8 | [-15.2,-6.6] | +36/-90 | 0.0000*** | | |
| **Neither cross-embodiment pretrain makes a better starting point than | |
| stage3_distill for LIBERO specifically - both make a worse one, by wide and | |
| significant margins.** Head-to-head on the same 500 paired episodes, **C beats B by +5.4 pp** (paired 95% CI [+0.0,+10.8], McNemar p=0.0623, +111/-84 flips) - which does **not** clear significance at 0.05, so the two cross-embodiment pretrains are not separated by this evidence. This isn't | |
| one catastrophic task dragging an average down: 7/10 tasks score lower for the B-derived fine-tune than for ft4 (the other 3 are within a couple of episodes either way). Worst-hit: task8 at 9/50 vs. ft4's 39/50. Best-preserved: task5 at 49/50 vs. ft4's 42/50. Both checkpoints have identical architecture and parameter/FLOP counts (confirmed via the same profiling harness as section 6) - this is purely a pretraining-lineage effect, not a compute or capacity difference. | |
| Plausible reason: `stage3_distill` was trained specifically as *this* model's | |
| LIBERO-adjacent lineage (stage 1-3, spatial distillation against LIBERO-style | |
| labels per the model card), whereas `pretrain_B` is a from-scratch TinyVLA-2 run | |
| on a disjoint mixture (SO-100/101 + Bridge + RT-1, no LIBERO exposure anywhere in | |
| its lineage, canonical-EE-space objective) built for a different question entirely | |
| (cross-embodiment transfer to `jaco`/`stretch`, not LIBERO manipulation quality). | |
| 40k fine-tuning steps evidently were not enough to close that gap. This is a data | |
| point against "any reasonable pretrain fine-tunes to a similar place on LIBERO", | |
| not evidence about canonical vs. native action spaces (both `ft4`'s lineage and | |
| `pretrain_B` ultimately regress against LIBERO's own action convention once | |
| fine-tuning starts, per section 1's frame-rate check applying identically to | |
| both - the difference is what the backbone learned before LIBERO ever entered | |
| the picture). | |
| --- | |
| ## 11. Zero-shot transfer to RoboCasa (a genuinely different domain) | |
| Everything above is LIBERO: one fixed tabletop Panda, one scene layout family. | |
| This section asks whether `ft4` transfers with no fine-tuning at all to | |
| [RoboCasa365](https://robocasa.ai/) — PandaOmron (Franka arm on a holonomic | |
| mobile base) in procedurally-varied kitchens, a different robot, different | |
| scenes, and a different observation layout. | |
| **Conventions were verified empirically, not assumed** (`verify_robocasa.py`, | |
| `verify_robocasa_axes.py`), because a silent frame mismatch here would be | |
| indistinguishable from a policy failure: | |
| * **Gripper**: RoboCasa's -1=open/+1=closed matches LIBERO's polarity directly | |
| (measured qpos 0.039 at -1, 0.001 at +1). No sign flip. | |
| * **Action**: both use the same OSC_POSE delta controller at the same physical | |
| scaling (±0.05 m, ±0.5 rad/step), so LIBERO's unnormalized | |
| `[dpos(3), drot(3), grip(1)]` maps onto RoboCasa's `ee_pos/ee_rot/gripper` | |
| slots. `base_motion(4)` and `control_mode(1)` held at zero — arm-only. | |
| * **State**: RoboCasa's native 16-dim `agent_pos` is a *different layout* | |
| (base pose + relative EE) from the trained convention (absolute EE pose + | |
| gripper), converted via `ee_state_from_agent_pos` before normalizing. Feeding | |
| the raw vector through would put a base quaternion where the model expects an | |
| EE rotvec. | |
| * **Cameras**: `robot0_agentview_left` + `robot0_eye_in_hand`, used raw. A | |
| crosshair projection through the camera's own MuJoCo extrinsics shows +y and | |
| +z already match LIBERO's screen convention (+y→left, +z→up), while +x is a | |
| *diagonal* blend of both screen axes rather than LIBERO's clean "+x→down" — | |
| this camera looks into the kitchen at an angle instead of down a tabletop. | |
| **No correction is applied**: a reflection that fixed x would break the y/z | |
| agreement. This is a documented residual mismatch, not a fixed one. | |
| Each task runs at RoboCasa's own registered horizon (`get_task_horizon`), not a | |
| truncated budget: 900 steps for CloseFridge, 750 for OpenDrawer, 450-600 for the | |
| rest. Semantic refresh 1 Hz, `n_action_steps=1`, 15 episodes/task. | |
| | Task | Horizon | Object registry | Success | 95% Wilson | Note | | |
| |---|---|---|---|---|---| | |
| | `CloseFridge` | 900 | lightwheel | 0/15 | 0.0-20.4 | — | | |
| | `OpenDrawer` | 750 | lightwheel | 0/15 | 0.0-20.4 | — | | |
| | `NavigateKitchen` | 450 | lightwheel | 0/15 | 0.0-20.4 | requires mobile-base motion; harness holds base_motion at zero | | |
| | `TurnOnMicrowave` | 450 | lightwheel+objaverse+aigen | 0/15 | 0.0-20.4 | — | | |
| | `PickPlaceCounterToStove` | 600 | lightwheel+objaverse+aigen | 0/15 | 0.0-20.4 | — | | |
| **Zero-shot transfer is a complete failure: 0/60 across the 4 arm-only tasks (95% Wilson 0.0-6.0%).** Not a partial-credit | |
| result to be tuned up — nothing succeeds anywhere. | |
| Two failure modes are visible in the saved videos (`results/robocasa_ft4/`), and | |
| they are *behavioural*, not random flailing: the arm drifts steadily upward until | |
| it clips the ceiling, and the wrist camera rotates until it faces the floor. Both | |
| are consistent with the one convention that could **not** be pinned down | |
| empirically — the correspondence between LIBERO's rotation axes and RoboCasa's | |
| `ee_rot`. A systematic rotation-frame error produces exactly this: a constant | |
| spurious delta integrated over hundreds of steps. | |
| **Section 12 tests that hypothesis and does not confirm it.** Fine-tuning on | |
| RoboCasa's own data fixes the trained task completely but moves none of the | |
| held-out ones — which a repaired shared rotation frame should have done. Treat | |
| the paragraph above as the motivating hypothesis, not a finding. | |
| Two honest caveats on the table above: | |
| * `NavigateKitchen` is **not a measurement of the policy**. The task requires | |
| driving the mobile base across the kitchen, and this harness pins | |
| `base_motion` to zero. Its 0/15 is guaranteed by construction, so it is | |
| excluded from the aggregate. | |
| * The object registry differs by row. `TurnOnMicrowave` and | |
| `PickPlaceCounterToStove` sample object categories that do not exist in the | |
| `lightwheel` registry and **cannot be constructed** without objaverse/aigen | |
| assets, so a single registry across all five tasks was not possible without | |
| re-running the first two. The fixtures each task acts on (fridge, drawer, | |
| microwave, stove) are scene fixtures, not sampled objects; the registry | |
| affects distractor content. | |
| This is the motivation for section 12: rather than hand-tuning the one | |
| unverifiable frame convention, fine-tune on RoboCasa's own recorded data and let | |
| the model learn whatever axis mapping the dataset actually contains. | |
| --- | |
| ## 12. Fine-tuning on RoboCasa's own data | |
| Section 11 leaves one convention unverifiable by construction — the mapping | |
| between LIBERO's rotation axes and RoboCasa's `ee_rot` — and blames the zero-shot | |
| collapse on it. Rather than hand-tune that mapping, this section removes the | |
| question: fine-tune on RoboCasa's *own recorded actions*, so the model learns | |
| whatever axis convention the data actually contains. | |
| **Data**: [`pepijn223/robocasa_CloseFridge`](https://huggingface.co/datasets/pepijn223/robocasa_CloseFridge), | |
| 513 episodes / 155,443 frames at a correctly-declared 20 fps (no repeat of | |
| section 1's frame-rate bug). RoboCasa's native 16-dim `agent_pos` and 12-dim flat | |
| action are converted to this repo's canonical 8-dim state / 7-dim action schema by | |
| `robocasa_train_source.py`, and normalized with statistics computed over the whole | |
| dataset (`compute_robocasa_stats.py`), **not** LIBERO's. Feeding the raw vectors | |
| through `CanonicalSource` would have silently truncated the 12-dim action to its | |
| first 8 entries — `base_motion` + `control_mode` + a partial `ee_pos` — which is | |
| why a dedicated source class exists rather than a config flag. | |
| **Recipe**: `ft4` as the starting point, 8,000 steps at effective batch 24 | |
| (2xV100, fp16), lr 1e-4, backbone unfrozen at 0.1x — the ft4 recipe on a shorter | |
| schedule (~1.24 epochs) because this is one task with 155k frames rather than | |
| LIBERO's 500-task mixture. Final training loss ~0.35. | |
| Evaluation is the *same harness, same seeds, same object registry* as section 11's | |
| zero-shot row, with one necessary difference: the fine-tuned checkpoint is | |
| unnormalized with its own RoboCasa statistics via `--stats-json`. Scoring it with | |
| LIBERO's statistics would push actions through the wrong mean/std and report a | |
| failure that belongs to the harness, not the policy. | |
| | | Success | 95% Wilson | delta | paired 95% CI | flips | McNemar p | | |
| |---|---|---|---|---|---|---| | |
| | `ft4` zero-shot | 0/15 = 0.0% | — | — | — | — | — | | |
| | **RoboCasa fine-tune** | 7/15 = 46.7% | [24.8,69.9] | +46.7 | [+20.0,+73.3] | +7/-0 | 0.0156 | | |
| 15 paired episodes (same seeds, same object registry). Pairing verified directly: the sampled instruction string matches on 15/15 episode indices, so the two runs saw the same scenes. | |
| **Fine-tuning works, and the effect is unambiguous**: every flip goes one way | |
| (+7/-0), so this is not variance around a shared mean. The two | |
| zero-shot failure modes from section 11 — upward drift into the ceiling, wrist | |
| camera rotating to face the floor — are absent from the fine-tuned videos | |
| (`results/robocasa_ftrc/`). | |
| ### The residual failures are structured, not random | |
| | Scene variant | Instruction | Success | 95% Wilson | | |
| |---|---|---|---| | |
| | single-door fridge | `Close the fridge door.` | 7/10 | [39.7,89.2] | | |
| | double-door fridge | `Close the fridge doors.` | 0/5 | [0.0,43.4] | | |
| Fisher exact p=0.0256. **This is not a training-data gap**: both instruction variants are | |
| well represented in the fine-tuning set (84,341 frames for the single-door | |
| phrasing, 71,102 for the double-door one — 54% / 46%). The model saw plenty of | |
| double-door episodes and still closes none of them. Closing two doors requires | |
| sequencing two separate contacts; closing one does not. With n=5 double-door | |
| episodes this is a suggestive split, not a settled result — but it is the correct | |
| place to look next, and it is invisible in the 7/15 aggregate. | |
| ### It did not fix a global convention error | |
| The obvious reading of section 11 is that one bad rotation mapping broke every | |
| task, in which case fine-tuning on any single task should repair the shared | |
| representation and lift the others too. It does not. Same fine-tuned checkpoint, | |
| same harness, tasks never seen during fine-tuning: | |
| | Held-out task | Success | 95% Wilson | | |
| |---|---|---| | |
| | `OpenDrawer` | 0/15 | [0.0,20.4] | | |
| | `TurnOnMicrowave` | 0/15 | [0.0,20.4] | | |
| | `PickPlaceCounterToStove` | 0/15 | [0.0,20.4] | | |
| **no measurable transfer**: 0/45 (95% Wilson 0.0-7.9%). | |
| So the honest conclusion is narrower than "fine-tuning fixes RoboCasa transfer." | |
| Fine-tuning demonstrably solves *the task it was trained on*, moving it from | |
| categorically broken to roughly half-successful. It provides **no evidence** that | |
| the zero-shot collapse was purely a frame-convention artifact — had it been, the | |
| held-out tasks would have moved. What the model acquired looks task-specific | |
| rather than a repaired shared geometry. Distinguishing "memorized one task" from | |
| "the other tasks need capabilities beyond the convention" needs a multi-task | |
| fine-tune, which is the next experiment, not a claim this data supports. | |
| --- | |
| ## Reproduction | |
| ```bash | |
| # the published configuration, with its rates measured | |
| python eval_libero_rates.py --n-action-steps 10 --semantic-hz 2 \ | |
| --n-init-states 10 --out-json accept.json | |
| # full-rate and dual-rate, full 500-episode paired sweep | |
| ./run_sharded.sh full_rate 50 --n-action-steps 1 --semantic-hz 20 | |
| ./run_sharded.sh dual_1hz 50 --n-action-steps 1 --semantic-hz 1 | |
| # paired statistics | |
| python stats.py results/rates/full_rate.json results/rates/dual_1hz.json | |
| ``` | |
| RoboCasa (sections 11-12) runs in a separate venv — robosuite 1.5.2, versus the | |
| 1.4.0 that LIBERO's eval depends on and that must not be disturbed: | |
| ```bash | |
| source .venv_robocasa/bin/activate | |
| # zero-shot: LIBERO-trained checkpoint, LIBERO normalization (the default) | |
| python eval_robocasa.py --checkpoint tinyvla --tasks CloseFridge OpenDrawer --episodes 15 --n-action-steps 1 --refresh-s 1.0 --out-json results/rates/robocasa_ft4.json --tag robocasa_ft4 | |
| # fine-tune on RoboCasa's own data (main venv, 2-GPU DDP) | |
| accelerate launch --multi_gpu --num_processes 2 --mixed_precision fp16 tinyvla/scripts/train.py --config tinyvla/configs/robocasa_ft_closefridge.yaml | |
| # evaluate it with ITS OWN normalization — LIBERO's stats would measure noise | |
| python eval_robocasa.py --checkpoint outputs/robocasa_ft_closefridge/final --stats-json robocasa_stats.json --tasks CloseFridge --episodes 15 --n-action-steps 1 --refresh-s 1.0 --out-json results/rates/robocasa_ftrc.json --tag robocasa_ftrc | |
| ``` | |
| Tasks whose scenes sample objaverse/aigen object categories (`TurnOnMicrowave`, | |
| `PickPlaceCounterToStove`) need those registries downloaded | |
| (`python -m robocasa.scripts.download_kitchen_assets --type objs_objaverse objs_aigen`) | |
| and passed via `--obj-registries lightwheel objaverse aigen`; without them the | |
| environment cannot be constructed at all. | |