# TinyVLA-2 — system wiring (what connects to what)

## Data flow (one dataset → model)

```
raw LeRobot dataset (per robot)
  observation.images.*   observation.state   action        task(text)
        │                      │                │              │
        │      scripts/build_ee_sidecar.py  (offline, once, NO video decode)
        │      ├─ SO101: FK(joint_deg) via placo+URDF  → EE pose
        │      └─ others: EE pose read from observation.state (euler / quat auto-detect)
        │                      ▼
        │      ~/tinyvla_data/ee_sidecar/<name>.parquet   [s_p*,s_q*, a_p*,a_q*, grip]
        │                      │
        │      src/tinyvla/data/canonical.py  CanonicalChunkStore
        │      ├─ 10 Hz resample (step = round(fps/10))
        │      ├─ base-frame left-delta:  Δ0 = a⊖s ; Δk = a_k ⊖ a_{k-1}
        │      ├─ rotation = log(R_k R_{k-1}^T) rotvec ; gripper = q1–q99 openness
        │      └─ quantile (q01/q99 → [-1,1]) normalization
        │                      ▼
        └─────► src/tinyvla/data/mixture.py  CanonicalSource  →  canonical action chunk (50, 7→pad 8)
                                             + action_dim_mask, camera_mask(cam0/cam1), embodiment_id, morphology
                       │
              WeightedMixtureDataset (seed 42, temperature weights)  →  batch
                       │
                       ▼
              src/tinyvla/modeling_tinyvla.py  TinyVLAPolicy
```

## Model conditioning (where each wire lands)

```
cam0 ─► Qwen3.5 vision tower ─► 64 spatial tokens ──────────────┐
cam0 + instruction ─► Qwen3.5 LM (12 layers) ─► 8 semantic latent┤
proprio (state, pad 16) ─► state_proj ──────────────────────────┤► expert
                                                                 │  cross-attn
   conditioning (ONE of):                                        │  conditioning set
   • variant A/B:  embodiment_id ─► nn.Embedding(16,512) ─► 1 tok │
   • variant C  :  morphology(16) ─► MorphologyEncoder MLP ─► 2 tok
                                                                 ▼
                                    FlowMatchingExpert ─► action chunk (50 × 8)
                                    action_space: native (A) | canonical (B/C)
```

## Variant matrix (what each checkpoint is wired to)

| variant | conditioning | action_space | checkpoint |
|---|---|---|---|
| A | embodiment_id | native  | outputs/tv2_A_id_native/final |
| B | embodiment_id | canonical | outputs/tv2_B_id_canon/final |
| C | morphology    | canonical | outputs/tv2_C_morph_canon/final |
| C-diverse | morphology | canonical | outputs/tv2_C_diverse/final (continue-train on 8 embodiments) |

## Embodiment registry (id ↔ morph_key ↔ dataset ↔ canonicalization)

Pretrain A/B/C used 3 embodiments (so101=0, bridge=1, rt1=2). C-diverse expands to 8:

| emb_id | morph_key | dataset root | canon source | weight | role |
|---|---|---|---|---|---|
| 0 | so101 | so101_v3/* (40 sets) | FK(joints) | 0.25 | train (fixed arm 5-DOF) |
| 1 | bridge | MolmoAct2-Bridge-Dataset | EE-state (euler) | 0.18 | train (WidowX arm) |
| 2 | rt1 | MolmoAct2-RT-1-Dataset | EE-state (quat) | 0.18 | train (Google mobile) |
| 3 | ur5 | diverse/div_ur5 | EE-state | 0.10 | train (UR5 arm) |
| 4 | xarm | diverse/div_xarm | EE-state | 0.09 | train (xArm) |
| 5 | dlr_edan | diverse/div_dlr_edan | EE-state | 0.05 | train (wheelchair arm) |
| 6 | tidybot | heldout/heldout_tidybot | EE-state (quat) | 0.08 | train (holonomic mobile) |
| 7 | hello_stretch | heldout/heldout_cmu_stretch | EE-state | 0.07 | train (prismatic mobile) |
| — | jaco | heldout/heldout_jaco_play | EE-state | — | **HELD OUT** (transfer test) |
| — | fanuc / uav | — | joint (no URDF) / broken | — | not used |

## Key file map

| concern | file |
|---|---|
| FK (SO101 joints→EE) | src/tinyvla/data/kinematics_so101.py (+ assets/so_arm/*.urdf, meshes) |
| EE-pose sidecar builder | scripts/build_ee_sidecar.py |
| canonical action assembly | src/tinyvla/data/canonical.py |
| dataset schema / mixture | src/tinyvla/data/mixture.py (CanonicalSource, WeightedMixtureDataset) |
| morphology descriptors | configs/morphology/descriptors.yaml (+ MORPH_FIELDS in modules/embodiment.py) |
| model + conditioning switch | src/tinyvla/modeling_tinyvla.py, configuration_tinyvla.py |
| training loop | scripts/train.py |
| physical-space eval | scripts/eval_canonical.py (integrated-trajectory mm/deg) |
| few-shot grids | scripts/{fewshot_grid,tidybot_fewshot}.py |
| results | outputs/tv2_final_results.md |
```