Hugging Face's logo

AlexWortega
/
tinyvla

Robotics
LeRobot
Safetensors
vla
libero
dual-rate
Model card Files
xet
 Community
tinyvla
File size: 4,791 Bytes
e63ca67
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
# TinyVLA-2 β€” system wiring (what connects to what)

## Data flow (one dataset β†’ model)

```
raw LeRobot dataset (per robot)
  observation.images.*   observation.state   action        task(text)
        β”‚                      β”‚                β”‚              β”‚
        β”‚      scripts/build_ee_sidecar.py  (offline, once, NO video decode)
        β”‚      β”œβ”€ SO101: FK(joint_deg) via placo+URDF  β†’ EE pose
        β”‚      └─ others: EE pose read from observation.state (euler / quat auto-detect)
        β”‚                      β–Ό
        β”‚      ~/tinyvla_data/ee_sidecar/<name>.parquet   [s_p*,s_q*, a_p*,a_q*, grip]
        β”‚                      β”‚
        β”‚      src/tinyvla/data/canonical.py  CanonicalChunkStore
        β”‚      β”œβ”€ 10 Hz resample (step = round(fps/10))
        β”‚      β”œβ”€ base-frame left-delta:  Ξ”0 = aβŠ–s ; Ξ”k = a_k βŠ– a_{k-1}
        β”‚      β”œβ”€ rotation = log(R_k R_{k-1}^T) rotvec ; gripper = q1–q99 openness
        β”‚      └─ quantile (q01/q99 β†’ [-1,1]) normalization
        β”‚                      β–Ό
        └─────► src/tinyvla/data/mixture.py  CanonicalSource  β†’  canonical action chunk (50, 7β†’pad 8)
                                             + action_dim_mask, camera_mask(cam0/cam1), embodiment_id, morphology
                       β”‚
              WeightedMixtureDataset (seed 42, temperature weights)  β†’  batch
                       β”‚
                       β–Ό
              src/tinyvla/modeling_tinyvla.py  TinyVLAPolicy
```

## Model conditioning (where each wire lands)

```
cam0 ─► Qwen3.5 vision tower ─► 64 spatial tokens ──────────────┐
cam0 + instruction ─► Qwen3.5 LM (12 layers) ─► 8 semantic latent─
proprio (state, pad 16) ─► state_proj ───────────────────────────► expert
                                                                 β”‚  cross-attn
   conditioning (ONE of):                                        β”‚  conditioning set
   β€’ variant A/B:  embodiment_id ─► nn.Embedding(16,512) ─► 1 tok β”‚
   β€’ variant C  :  morphology(16) ─► MorphologyEncoder MLP ─► 2 tok
                                                                 β–Ό
                                    FlowMatchingExpert ─► action chunk (50 Γ— 8)
                                    action_space: native (A) | canonical (B/C)
```

## Variant matrix (what each checkpoint is wired to)

| variant | conditioning | action_space | checkpoint |
|---|---|---|---|
| A | embodiment_id | native  | outputs/tv2_A_id_native/final |
| B | embodiment_id | canonical | outputs/tv2_B_id_canon/final |
| C | morphology    | canonical | outputs/tv2_C_morph_canon/final |
| C-diverse | morphology | canonical | outputs/tv2_C_diverse/final (continue-train on 8 embodiments) |

## Embodiment registry (id ↔ morph_key ↔ dataset ↔ canonicalization)

Pretrain A/B/C used 3 embodiments (so101=0, bridge=1, rt1=2). C-diverse expands to 8:

| emb_id | morph_key | dataset root | canon source | weight | role |
|---|---|---|---|---|---|
| 0 | so101 | so101_v3/* (40 sets) | FK(joints) | 0.25 | train (fixed arm 5-DOF) |
| 1 | bridge | MolmoAct2-Bridge-Dataset | EE-state (euler) | 0.18 | train (WidowX arm) |
| 2 | rt1 | MolmoAct2-RT-1-Dataset | EE-state (quat) | 0.18 | train (Google mobile) |
| 3 | ur5 | diverse/div_ur5 | EE-state | 0.10 | train (UR5 arm) |
| 4 | xarm | diverse/div_xarm | EE-state | 0.09 | train (xArm) |
| 5 | dlr_edan | diverse/div_dlr_edan | EE-state | 0.05 | train (wheelchair arm) |
| 6 | tidybot | heldout/heldout_tidybot | EE-state (quat) | 0.08 | train (holonomic mobile) |
| 7 | hello_stretch | heldout/heldout_cmu_stretch | EE-state | 0.07 | train (prismatic mobile) |
| β€” | jaco | heldout/heldout_jaco_play | EE-state | β€” | **HELD OUT** (transfer test) |
| β€” | fanuc / uav | β€” | joint (no URDF) / broken | β€” | not used |

## Key file map

| concern | file |
|---|---|
| FK (SO101 joints→EE) | src/tinyvla/data/kinematics_so101.py (+ assets/so_arm/*.urdf, meshes) |
| EE-pose sidecar builder | scripts/build_ee_sidecar.py |
| canonical action assembly | src/tinyvla/data/canonical.py |
| dataset schema / mixture | src/tinyvla/data/mixture.py (CanonicalSource, WeightedMixtureDataset) |
| morphology descriptors | configs/morphology/descriptors.yaml (+ MORPH_FIELDS in modules/embodiment.py) |
| model + conditioning switch | src/tinyvla/modeling_tinyvla.py, configuration_tinyvla.py |
| training loop | scripts/train.py |
| physical-space eval | scripts/eval_canonical.py (integrated-trajectory mm/deg) |
| few-shot grids | scripts/{fewshot_grid,tidybot_fewshot}.py |
| results | outputs/tv2_final_results.md |
```