| # TinyVLA-2 β system wiring (what connects to what) |
|
|
| ## Data flow (one dataset β model) |
|
|
| ``` |
| raw LeRobot dataset (per robot) |
| observation.images.* observation.state action task(text) |
| β β β β |
| β scripts/build_ee_sidecar.py (offline, once, NO video decode) |
| β ββ SO101: FK(joint_deg) via placo+URDF β EE pose |
| β ββ others: EE pose read from observation.state (euler / quat auto-detect) |
| β βΌ |
| β ~/tinyvla_data/ee_sidecar/<name>.parquet [s_p*,s_q*, a_p*,a_q*, grip] |
| β β |
| β src/tinyvla/data/canonical.py CanonicalChunkStore |
| β ββ 10 Hz resample (step = round(fps/10)) |
| β ββ base-frame left-delta: Ξ0 = aβs ; Ξk = a_k β a_{k-1} |
| β ββ rotation = log(R_k R_{k-1}^T) rotvec ; gripper = q1βq99 openness |
| β ββ quantile (q01/q99 β [-1,1]) normalization |
| β βΌ |
| βββββββΊ src/tinyvla/data/mixture.py CanonicalSource β canonical action chunk (50, 7βpad 8) |
| + action_dim_mask, camera_mask(cam0/cam1), embodiment_id, morphology |
| β |
| WeightedMixtureDataset (seed 42, temperature weights) β batch |
| β |
| βΌ |
| src/tinyvla/modeling_tinyvla.py TinyVLAPolicy |
| ``` |
|
|
| ## Model conditioning (where each wire lands) |
|
|
| ``` |
| cam0 ββΊ Qwen3.5 vision tower ββΊ 64 spatial tokens βββββββββββββββ |
| cam0 + instruction ββΊ Qwen3.5 LM (12 layers) ββΊ 8 semantic latentβ€ |
| proprio (state, pad 16) ββΊ state_proj βββββββββββββββββββββββββββ€βΊ expert |
| β cross-attn |
| conditioning (ONE of): β conditioning set |
| β’ variant A/B: embodiment_id ββΊ nn.Embedding(16,512) ββΊ 1 tok β |
| β’ variant C : morphology(16) ββΊ MorphologyEncoder MLP ββΊ 2 tok |
| βΌ |
| FlowMatchingExpert ββΊ action chunk (50 Γ 8) |
| action_space: native (A) | canonical (B/C) |
| ``` |
|
|
| ## Variant matrix (what each checkpoint is wired to) |
|
|
| | variant | conditioning | action_space | checkpoint | |
| |---|---|---|---| |
| | A | embodiment_id | native | outputs/tv2_A_id_native/final | |
| | B | embodiment_id | canonical | outputs/tv2_B_id_canon/final | |
| | C | morphology | canonical | outputs/tv2_C_morph_canon/final | |
| | C-diverse | morphology | canonical | outputs/tv2_C_diverse/final (continue-train on 8 embodiments) | |
|
|
| ## Embodiment registry (id β morph_key β dataset β canonicalization) |
| |
| Pretrain A/B/C used 3 embodiments (so101=0, bridge=1, rt1=2). C-diverse expands to 8: |
| |
| | emb_id | morph_key | dataset root | canon source | weight | role | |
| |---|---|---|---|---|---| |
| | 0 | so101 | so101_v3/* (40 sets) | FK(joints) | 0.25 | train (fixed arm 5-DOF) | |
| | 1 | bridge | MolmoAct2-Bridge-Dataset | EE-state (euler) | 0.18 | train (WidowX arm) | |
| | 2 | rt1 | MolmoAct2-RT-1-Dataset | EE-state (quat) | 0.18 | train (Google mobile) | |
| | 3 | ur5 | diverse/div_ur5 | EE-state | 0.10 | train (UR5 arm) | |
| | 4 | xarm | diverse/div_xarm | EE-state | 0.09 | train (xArm) | |
| | 5 | dlr_edan | diverse/div_dlr_edan | EE-state | 0.05 | train (wheelchair arm) | |
| | 6 | tidybot | heldout/heldout_tidybot | EE-state (quat) | 0.08 | train (holonomic mobile) | |
| | 7 | hello_stretch | heldout/heldout_cmu_stretch | EE-state | 0.07 | train (prismatic mobile) | |
| | β | jaco | heldout/heldout_jaco_play | EE-state | β | **HELD OUT** (transfer test) | |
| | β | fanuc / uav | β | joint (no URDF) / broken | β | not used | |
| |
| ## Key file map |
| |
| | concern | file | |
| |---|---| |
| | FK (SO101 jointsβEE) | src/tinyvla/data/kinematics_so101.py (+ assets/so_arm/*.urdf, meshes) | |
| | EE-pose sidecar builder | scripts/build_ee_sidecar.py | |
| | canonical action assembly | src/tinyvla/data/canonical.py | |
| | dataset schema / mixture | src/tinyvla/data/mixture.py (CanonicalSource, WeightedMixtureDataset) | |
| | morphology descriptors | configs/morphology/descriptors.yaml (+ MORPH_FIELDS in modules/embodiment.py) | |
| | model + conditioning switch | src/tinyvla/modeling_tinyvla.py, configuration_tinyvla.py | |
| | training loop | scripts/train.py | |
| | physical-space eval | scripts/eval_canonical.py (integrated-trajectory mm/deg) | |
| | few-shot grids | scripts/{fewshot_grid,tidybot_fewshot}.py | |
| | results | outputs/tv2_final_results.md | |
| ``` |
| |
