TinyVLA-2 — system wiring (what connects to what)

Data flow (one dataset → model)

raw LeRobot dataset (per robot)
  observation.images.*   observation.state   action        task(text)
        │                      │                │              │
        │      scripts/build_ee_sidecar.py  (offline, once, NO video decode)
        │      ├─ SO101: FK(joint_deg) via placo+URDF  → EE pose
        │      └─ others: EE pose read from observation.state (euler / quat auto-detect)
        │                      ▼
        │      ~/tinyvla_data/ee_sidecar/<name>.parquet   [s_p*,s_q*, a_p*,a_q*, grip]
        │                      │
        │      src/tinyvla/data/canonical.py  CanonicalChunkStore
        │      ├─ 10 Hz resample (step = round(fps/10))
        │      ├─ base-frame left-delta:  Δ0 = a⊖s ; Δk = a_k ⊖ a_{k-1}
        │      ├─ rotation = log(R_k R_{k-1}^T) rotvec ; gripper = q1–q99 openness
        │      └─ quantile (q01/q99 → [-1,1]) normalization
        │                      ▼
        └─────► src/tinyvla/data/mixture.py  CanonicalSource  →  canonical action chunk (50, 7→pad 8)
                                             + action_dim_mask, camera_mask(cam0/cam1), embodiment_id, morphology
                       │
              WeightedMixtureDataset (seed 42, temperature weights)  →  batch
                       │
                       ▼
              src/tinyvla/modeling_tinyvla.py  TinyVLAPolicy

Model conditioning (where each wire lands)

cam0 ─► Qwen3.5 vision tower ─► 64 spatial tokens ──────────────┐
cam0 + instruction ─► Qwen3.5 LM (12 layers) ─► 8 semantic latent┤
proprio (state, pad 16) ─► state_proj ──────────────────────────┤► expert
                                                                 │  cross-attn
   conditioning (ONE of):                                        │  conditioning set
   • variant A/B:  embodiment_id ─► nn.Embedding(16,512) ─► 1 tok │
   • variant C  :  morphology(16) ─► MorphologyEncoder MLP ─► 2 tok
                                                                 ▼
                                    FlowMatchingExpert ─► action chunk (50 × 8)
                                    action_space: native (A) | canonical (B/C)

Variant matrix (what each checkpoint is wired to)

variant	conditioning	action_space	checkpoint
A	embodiment_id	native	outputs/tv2_A_id_native/final
B	embodiment_id	canonical	outputs/tv2_B_id_canon/final
C	morphology	canonical	outputs/tv2_C_morph_canon/final
C-diverse	morphology	canonical	outputs/tv2_C_diverse/final (continue-train on 8 embodiments)

Embodiment registry (id ↔ morph_key ↔ dataset ↔ canonicalization)

Pretrain A/B/C used 3 embodiments (so101=0, bridge=1, rt1=2). C-diverse expands to 8:

emb_id	morph_key	dataset root	canon source	weight	role
0	so101	so101_v3/* (40 sets)	FK(joints)	0.25	train (fixed arm 5-DOF)
1	bridge	MolmoAct2-Bridge-Dataset	EE-state (euler)	0.18	train (WidowX arm)
2	rt1	MolmoAct2-RT-1-Dataset	EE-state (quat)	0.18	train (Google mobile)
3	ur5	diverse/div_ur5	EE-state	0.10	train (UR5 arm)
4	xarm	diverse/div_xarm	EE-state	0.09	train (xArm)
5	dlr_edan	diverse/div_dlr_edan	EE-state	0.05	train (wheelchair arm)
6	tidybot	heldout/heldout_tidybot	EE-state (quat)	0.08	train (holonomic mobile)
7	hello_stretch	heldout/heldout_cmu_stretch	EE-state	0.07	train (prismatic mobile)
—	jaco	heldout/heldout_jaco_play	EE-state	—	HELD OUT (transfer test)
—	fanuc / uav	—	joint (no URDF) / broken	—	not used

Key file map

concern	file
FK (SO101 joints→EE)	src/tinyvla/data/kinematics_so101.py (+ assets/so_arm/*.urdf, meshes)
EE-pose sidecar builder	scripts/build_ee_sidecar.py
canonical action assembly	src/tinyvla/data/canonical.py
dataset schema / mixture	src/tinyvla/data/mixture.py (CanonicalSource, WeightedMixtureDataset)
morphology descriptors	configs/morphology/descriptors.yaml (+ MORPH_FIELDS in modules/embodiment.py)
model + conditioning switch	src/tinyvla/modeling_tinyvla.py, configuration_tinyvla.py
training loop	scripts/train.py
physical-space eval	scripts/eval_canonical.py (integrated-trajectory mm/deg)
few-shot grids	scripts/{fewshot_grid,tidybot_fewshot}.py
results	outputs/tv2_final_results.md