Instructions to use AlexWortega/tinyvla with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- LeRobot
How to use AlexWortega/tinyvla with LeRobot:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
ablation.md: add pretrain_B (same-harness LIBERO+profile); pretrain_C flagged as unrunnable here (missing morph-conditioning code/descriptors), cite author's own A/B/C few-shot table instead
Browse files- ablation.md +15 -2
ablation.md
CHANGED
|
@@ -8,7 +8,7 @@ All closed-loop numbers below are **paired**: every configuration runs the same
|
|
| 8 |
|
| 9 |
Hardware: 4x Tesla V100-SXM2-32GB (sm_70), fp32, driver 535 / CUDA 12.2,
|
| 10 |
torch 2.6.0+cu124, transformers 5.5.4, lerobot 0.6.1. Suite: `libero_spatial`.
|
| 11 |
-
Generated 2026-08-
|
| 12 |
|
| 13 |
**The headline dual-rate comparison in the model card is not reproduced.** With the
|
| 14 |
clock fixed and 500 paired episodes per configuration, no refresh rate differs
|
|
@@ -302,10 +302,23 @@ shared pretrain.
|
|
| 302 |
|---|---|---|---|
|
| 303 |
| `checkpoints/stage2_final` | SO-100/101 + Bridge mixture, frozen backbone | 0/100 = 0.0% | 0.0-3.7 |
|
| 304 |
| `checkpoints/stage3_final` | + spatial distillation | 0/100 = 0.0% | 0.0-3.7 |
|
| 305 |
-
| `tinyvla2/pretrain_A` | TinyVLA-2
|
|
|
|
| 306 |
|
| 307 |
**`tinyvla2/pretrain_A` scores 0.0% on LIBERO** (0/100), consistent with stage2/stage3: none of the three pre-fine-tune checkpoints do anything useful on this suite. This is a different model from the one the rest of this document ablates — it is the *control* arm of a separate cross-embodiment ablation (variants A/B/C), not a LIBERO checkpoint. Its own model card already reports it failing cross-embodiment transfer on held-out `jaco` (endpoint error 1392 mm vs. a 128 mm zero-prediction floor, n=8) — a result from the author, not independently reproduced here since the held-out-embodiment harness this document uses is LIBERO/MetaWorld/SO-101, not the jaco eval that number came from. `conditioning`, `morph_tokens`, and `action_space` in its config are not implemented in `modeling_tinyvla.py` as of this checkpoint family's release — they were added to `configuration_tinyvla.py` here as pass-through fields so the checkpoint would load, since variant A's own documentation states it is architecturally identical to the embodiment-ID scheme already implemented (`EmbodimentEmbedding`). Whether that also holds for variants B and C is unverified.
|
| 308 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 309 |
---
|
| 310 |
|
| 311 |
## Reproduction
|
|
|
|
| 8 |
|
| 9 |
Hardware: 4x Tesla V100-SXM2-32GB (sm_70), fp32, driver 535 / CUDA 12.2,
|
| 10 |
torch 2.6.0+cu124, transformers 5.5.4, lerobot 0.6.1. Suite: `libero_spatial`.
|
| 11 |
+
Generated 2026-08-22.
|
| 12 |
|
| 13 |
**The headline dual-rate comparison in the model card is not reproduced.** With the
|
| 14 |
clock fixed and 500 paired episodes per configuration, no refresh rate differs
|
|
|
|
| 302 |
|---|---|---|---|
|
| 303 |
| `checkpoints/stage2_final` | SO-100/101 + Bridge mixture, frozen backbone | 0/100 = 0.0% | 0.0-3.7 |
|
| 304 |
| `checkpoints/stage3_final` | + spatial distillation | 0/100 = 0.0% | 0.0-3.7 |
|
| 305 |
+
| `tinyvla2/pretrain_A` | TinyVLA-2 ablation, variant A (native action space) | 0/100 = 0.0% | 0.0-3.7 |
|
| 306 |
+
| `tinyvla2/pretrain_B` | TinyVLA-2 ablation, variant B (canonical action space) | 0/100 = 0.0% | 0.0-3.7 |
|
| 307 |
|
| 308 |
**`tinyvla2/pretrain_A` scores 0.0% on LIBERO** (0/100), consistent with stage2/stage3: none of the three pre-fine-tune checkpoints do anything useful on this suite. This is a different model from the one the rest of this document ablates — it is the *control* arm of a separate cross-embodiment ablation (variants A/B/C), not a LIBERO checkpoint. Its own model card already reports it failing cross-embodiment transfer on held-out `jaco` (endpoint error 1392 mm vs. a 128 mm zero-prediction floor, n=8) — a result from the author, not independently reproduced here since the held-out-embodiment harness this document uses is LIBERO/MetaWorld/SO-101, not the jaco eval that number came from. `conditioning`, `morph_tokens`, and `action_space` in its config are not implemented in `modeling_tinyvla.py` as of this checkpoint family's release — they were added to `configuration_tinyvla.py` here as pass-through fields so the checkpoint would load, since variant A's own documentation states it is architecturally identical to the embodiment-ID scheme already implemented (`EmbodimentEmbedding`). Whether that also holds for variants B and C is unverified.
|
| 309 |
|
| 310 |
+
**`tinyvla2/pretrain_B` scores 0.0% on LIBERO** (0/100). Same story as A: this is not a LIBERO checkpoint either, and pre-fine-tune performance on this suite tells you nothing about the A-vs-B question, which is about zero-shot transfer to held-out embodiments, not about LIBERO. B differs from A only in `action_space` (canonical vs. native) - same seed, mixture, weights, and step count per both checkpoints' own model cards. B's card reports a mixed zero-shot result: canonical action space is decisive on `jaco` (137 mm vs. A's catastrophic 1392 mm) but *loses* to native on `stretch` (195 mm vs. 150 mm) - so canonicalization is not a strict win, and the authors flag the comparison itself as confounded by differing normalization, deferring the decisive test to few-shot adaptation efficiency once variant C lands. Not independently reproduced here for the same reason as A: this document's harness is LIBERO/MetaWorld/SO-101, not the jaco/stretch eval those numbers came from.
|
| 311 |
+
|
| 312 |
+
**`tinyvla2/pretrain_C` was not run through this harness.** Its checkpoint carries four trained tensors (`morph_encoder.mlp.{0,2}.{weight,bias}`, a 16-to-512-to-1024 MLP producing the 2 morphology-conditioning tokens the README describes) that `modeling_tinyvla.py` as published has no forward-pass code path for, and the morphology descriptor file its README points to (`configs/morphology/descriptors.yaml`) is not in the repository. Loading it anyway was tested directly: the loader logs `Unexpected key(s) ... morph_encoder.mlp.*` and silently drops all four tensors, and inference would fall back to `embodiment_emb(emb_id)` - the exact conditioning path the README states C replaces. Reporting a LIBERO number from that would measure a model C was never designed to be, not the checkpoint as released. So this document cites only what the author has already published (`tinyvla2/RESULTS.md`, `tinyvla2/fewshot_results.json`), unverified here:
|
| 313 |
+
|
| 314 |
+
| jaco FT episodes | A (native+ID) | B (canonical+ID) | C (canonical+morph) |
|
| 315 |
+
|---|---|---|---|
|
| 316 |
+
| 0 (zero-shot) | 1377 mm | 155 mm | 178 mm |
|
| 317 |
+
| 100 | 6919 mm | 215 mm | 218 mm |
|
| 318 |
+
| 500 | 5667 mm | 198 mm | 197 mm |
|
| 319 |
+
|
| 320 |
+
(GT trajectories ~210 mm, zero-prediction floor ~128 mm, n=20 held-out jaco episodes.) The author's own verdict: canonical action space (B, C) beats native (A) by 10-45x at every budget and is decisive; morphology tokens (C) show no measurable advantage over a learned ID (B) once actions are canonical - "the win is action-space unification, not morphology conditioning." Native additionally *diverges* under few-shot fine-tuning (A gets worse with more FT episodes, not better), which the author attributes to re-learning a fresh embodiment-ID against a native action distribution.
|
| 321 |
+
|
| 322 |
---
|
| 323 |
|
| 324 |
## Reproduction
|