root card: stack_bowls section (floor 0.667 deg), the new control, and reflow the real3 row into the 4-column table
3970df9 verified | pipeline_tag: robotics | |
| tags: | |
| - openvla | |
| - vision-language-action | |
| - robotics | |
| - libero | |
| # TERRA β post-trained policies (real robot + LIBERO) | |
| OpenVLA-7B with TERRA latent-action phase 2 (`latp2_inv` labels), post-trained with a flow-matching action head and | |
| 4 query tokens. **One directory per policy**, each self-contained: merged weights (`model-*.safetensors`), the | |
| action/latent head (`action_decoder.pt` β **required**, it is not inside the merged VLA weights), | |
| `dataset_statistics.json` (action un-normalisation), `ctrl_meta.json`, processor/tokenizer, and its own `README.md` | |
| with the measured numbers and the exact deployment loop. | |
| **Code:** https://github.com/DingTianxingjian/Latent_Free β branch `newton`, public, no token needed. | |
| Real-robot deployment wrapper: `acdc_la/realrobot/piper_policy.py`; LIBERO evaluator: | |
| `acdc_la/downstream/vla/run_libero_eval_cont.py`. | |
| ```bash | |
| git clone -b newton https://github.com/DingTianxingjian/Latent_Free.git latent_free && cd latent_free | |
| huggingface-cli download Potestates/TERRA --include "<dir>/*" --local-dir . | |
| ``` | |
| --- | |
| ## Which one do I test? | |
| | I want to⦠| use | why | | |
| |---|---|---| | |
| | **press a coloured button** | **`press_button_terra_delta`** | the only policy **confirmed working on hardware** β it presses the requested button | | |
| | **pick the can, place on plate** | **`piper_pick_can_openvla_delta`** | **3x better open loop** than the arm currently on the robot (1.05 vs 3.24 deg); simplest of the equals | | |
| | the same, with our full method | `piper_pick_can_terra_delta` | statistically **indistinguishable** from the row above (1.13 vs 1.05, CI contains 0) β either is fine | | |
| | **stack the bowls** | `piper_real3_anchored_fwd` | the only *policy* for this task; `stack_bowls_openvla_delta` is a **control**, and it ties "repeat the previous command" | | |
| | **one model for both tasks** | `piper_multi_anchored_step5500` | **experimental**, and a retrain will supersede it β see its README | | |
| | **compare the latent mechanism** in sim | the `libero_goal_frac10_*` **triplet** | ours / UniVLA's / **no latent at all**, identical otherwise | | |
| | post-train a new task | `p2fft_20k` | the **base**, not a policy | | |
| > ### β Two things that will break a deployment if you skip them | |
| > 1. **Action space differs per directory.** `piper_pick_can_10k` emits **absolute** joint targets; everything with | |
| > `delta` in the name emits **target minus measured joint angle** and must be sent as `q + a[:6]` with `q` re-read | |
| > **every tick**; the `anchored` ones use `q` read **once per chunk** and a different call. Every `ctrl_meta.json` | |
| > now records `"action_space"` explicitly and `PiperPolicy` prints it on load β **check the printed line.** | |
| > 2. **Execute 1 chunk entry, then re-predict** (all per-step `delta` policies). The demos were teleoperated as a | |
| > continuous 30 Hz command, so later chunk entries are mis-anchored for an arm that settles: drift is | |
| > **0.855 deg/step on pick-can** and **~2 deg/step on press_button**, and it is opposite-signed on the two joints | |
| > that carry the motion, so it **bends the direction** instead of just scaling it. This was seen on the robot: | |
| > chunk 1 presses the button, chunk 2 "the direction is a bit off". Use an `anchored` checkpoint if you need | |
| > multi-step chunks. | |
| --- | |
| ## The three action spaces on the Piper β getting this wrong drives the arm somewhere else | |
| `ctrl_meta.json -> action_space` says which one a directory uses, `PiperPolicy` reads it and prints it on load. | |
| (It used to be *absent* on the pick-can directories and inferred from the joint means; since 2026-11-12 all three | |
| record it explicitly, so **a missing key now means an old copy** β re-download.) | |
| | `action_space` | what `chunk[k]` is | the call | how many steps per chunk | | |
| |---|---|---|---| | |
| | `absolute` | the joint target itself | send `chunk[k]` directly | β | | |
| | `delta` | target β joints measured **at step k** | `to_joint_targets(a, q)`, `q` re-read **every tick** | **1** β the teleop lead adds up linearly and with opposite signs on j1/j2 (β0.74 / +0.75 deg per step on pick-can), so the direction bends | | |
| | `delta_anchored` | `target[s+k]` β joints read **once** at the chunk's start | `chunk_targets(chunk, q)`, `q` read once per chunk; **no ensembling** (the policy disables it) | 1, then try 2 and 4: tracking drift does not exist here, what limits N is the model's own error growing with k | | |
| --- | |
| ## Inventory β which directories are **ours (TERRA)** and which are **controls / baselines** | |
| Every policy here shares the same interface (LoRA r32 + flow-matching head + 4 query tokens, chunk 12). What differs | |
| is the **base** and the **latent channel**, which is exactly what each comparison is about. | |
| ### TERRA β ours (latent channel `tc_cont`, `latp2_inv` labels) | |
| | dir | base | task / suite | number | | |
| |---|---|---|---| | |
| | `p2fft_20k` | openvla-7b, full fine-tune | **phase-2 base, not a policy** | β | | |
| | `press_button_terra_delta` | `p2fft_20k` | press_button, per-step delta | 0.918Β° β on hardware | | |
| | **`piper_real3_anchored_fwd`** | `p2fft_20k` | **THE single robot model**: all 3 tasks (press_button x2 + pick-can + stack_bowls), **anchored**, fixed window | step 6000; `unnorm_key` **per task** β see its card | | |
| | `press_button_anchored_fwd_step1500` | `p2fft_20k` | press_button, anchored, fixed window | 0.96Β° at k=0, flat in k | | |
| | `piper_pick_can_terra_delta` | `p2fft_20k` | pick-can, per-step delta | 1.126Β° | | |
| | `piper_pick_can_10k` | `p2fft_20k` | pick-can, **absolute** β superseded | 3.24Β° | | |
| | `piper_multi_anchored_step2500` / `step5500` | `p2fft_20k` | both tasks, anchored β experimental | see card | | |
| | `libero_goal_full` | LoRA-scale phase 2 | LIBERO-Goal, full split | 94.4 % | | |
| | `libero_10_full` | LoRA-scale phase 2 | LIBERO-Long, full split | 89.0 % | | |
| | `libero_10_frac10_terra_p2fft20k` | `p2fft_20k` | LIBERO-Long, 10 % (43 demos) | 60.6 % | | |
| | `libero_goal_frac10_terra` | **openvla-7b, no phase 2** | LIBERO-Goal, 10 % | 86.8 % | | |
| ### Baselines / controls β no latent, or somebody else's latent | |
| | dir | what it controls for | number | | |
| |---|---|---| | |
| | `press_button_openvla_delta` | stock openvla-7b, **`--latent none`** β the whole package vs nothing | 1.920Β° (worse than not moving) | | |
| | `piper_pick_can_openvla_delta` | the same, on pick-can | **1.054Β°** β here it is a good policy | | |
| | **`stack_bowls_openvla_delta`** | stock openvla-7b, **`--latent none`**, stack_bowls alone, per-step delta | **0.695Β°** β but *repeating the previous command* scores **0.667Β°** on the same episodes, so one step at a time it is **not distinguishable from doing the trivial thing** | | |
| | `libero_goal_frac10_none` | **no latent at all**, identical otherwise (`latent_r2` β‘ 0) | 88.6 % | | |
| | `libero_goal_frac10_univla_cont` | **UniVLA's** deployed latent through our interface | 89.4 % | | |
| > **One honest caveat about the two `*_openvla_delta` controls.** They change the base **and** the latent branch at | |
| > once (`INIT=none LATENT=none`), so where they separate, the effect belongs to *the whole package*, not specifically | |
| > to the latent branch. The missing arm that would split those two β phase-2 base with `--latent none` β has not been | |
| > run. The `libero_goal_frac10_*` triplet *is* clean in that sense (one factor, no phase 2 anywhere), and there the | |
| > answer is that the latent channel changes nothing. | |
| > | |
| > **Which base a policy came from is not in most of these files.** `ctrl_meta.json` records the latent, the head | |
| > and the dataset, but the training script never wrote the phase-2 init. The four directories published on | |
| > 2026-11-12 (`libero_goal_frac10_*`, `libero_10_frac10_terra_p2fft20k`) now carry an explicit `"init"` field; | |
| > for the others the base is stated in the table above and in each card, and nowhere in the weights. | |
| --- | |
| ## Real-robot policies β AgileX Piper | |
| Open-loop error = mean |error| per joint in **degrees** on held-out episodes never trained on, chunk sampled from | |
| noise through the deployment image path. Reproduce any row with | |
| `envs/univla_vla/bin/python -m acdc_la.realrobot.piper_policy --checkpoint <dir> --replay <rlds root>` β it is | |
| deterministic, and reproducing it is the check that your preprocessing matches ours. | |
| ### `press_button` β "press the {blue, orange, pink, red} button", 100 episodes, layout reshuffled per episode | |
| | dir | base | action | `chunk[0]` | reads the instruction? | on the robot | | |
| |---|---|---|---|---|---| | |
| | **`press_button_terra_delta`** | our phase 2 | delta | **0.918** | **yes** β output moves 0.42 deg on a colour swap; true word wins 4/5 episodes | β **presses the button** (chunk 1; slightly off centre) | | |
| | `press_button_openvla_delta` | stock openvla-7b | delta | 1.920 | **no** β output moves **2.4e-7 deg**; true word wins 1/5 = chance | β not for deployment | | |
| | *null: "stay where you are"* | | | *1.691* | | | | |
| The control is *worse than not moving*, and its invariance to language is **exact, not approximate**: its instruction | |
| signal is 0.28% of the query-token state, which is below bfloat16 machine epsilon (7.8e-3), so it cannot survive its | |
| own forward pass. With 4 colours and per-episode layouts a policy that cannot read the colour can only learn an | |
| average press β which is what its j1/j2 error looks like. Caveats: **5 held-out episodes, 3 of 4 colours, one seed | |
| per arm, offline**; and the pair changes *both* the base and the latent branch, so the effect cannot be attributed to | |
| the latent branch alone. | |
| ### `piper_pick_can` β "pick up the can and place it on the plate", 150 episodes, **one** instruction throughout | |
| | dir | base | action | `chunk[0]` | ensembled | note | | |
| |---|---|---|---|---|---| | |
| | **`piper_pick_can_openvla_delta`** | stock openvla-7b | delta | **1.054** | 0.914 | the matched control β and here a perfectly good policy | | |
| | **`piper_pick_can_terra_delta`** | our phase 2 | delta | **1.126** | 0.950 | ours; **not separated** from the row above | | |
| | `piper_pick_can_10k` | our phase 2 | **absolute** | 3.24 | 3.69 | the arm tested on the robot first; **superseded** by the two above | | |
| | *null: "stay where you are"* | | | *1.20* | | | | |
| | *null: repeat the last command* | | | *2.06* | | | | |
| | *between-demonstration floor* | | | *0.61* | | **nothing can beat this** | | |
| **Delta vs absolute is the real win here: 3.24 β ~1.05 deg, 3x**, and it moves the policy from 2.7x *worse* than | |
| doing nothing to better than it. **TERRA vs control is not a win for either side**: paired cluster bootstrap over the | |
| 8 held-out episodes (the independent unit β 353 steps is not the sample size) gives a difference of +0.072 deg, | |
| 95% CI **[-0.038, +0.189]**, sign test p = 0.727. | |
| A null here is what the mechanism's own story predicts: **all 150 episodes carry the identical instruction**, so | |
| there is nothing for a language-grounding advantage to do. The same pair separates 2.1x on `press_button`. | |
| ### `stack_bowls` β "stack the bowls", 100 episodes, **one** instruction throughout | |
| | dir | base | latent | action | `chunk[0]` | ensembled | | |
| |---|---|---|---|---|---| | |
| | `stack_bowls_openvla_delta` | stock openvla-7b | **none** | per-step delta, single task | **0.695Β°** | 0.585Β° | | |
| | `piper_real3_anchored_fwd` | `p2fft_20k` | `tc_cont` | **anchored**, 3-task mixture | not scored here | n/a (never ensemble anchored) | | |
| **The floor on this task is 0.667Β°, not 0.** On the 5 held-out episodes (184 steps) *repeating the previous | |
| command* scores **0.667Β°** and *not moving at all* **0.863Β°** β and the control's 0.695Β° is **not distinguishable** | |
| from repeat-last (+0.031Β°, 95% paired cluster bootstrap over the 5 episodes **[-0.118, +0.184]**). Only temporal | |
| ensembling beats it, by 0.079Β° [-0.147, -0.015], on an interval that one bad episode would swallow. Note the order | |
| **flips** against pick-can, where stay-still (1.20Β°) was the easier null and repeat-last (2.06Β°) the harder: here the | |
| motion is smooth and strongly drifting, so repeat-last is the one to beat. | |
| **These two directories do not pair.** The control is `stack_bowls` **alone**, **per-step** delta, **backward** | |
| sample window; `piper_real3_anchored_fwd` is a 350-episode **3-task mixture**, **chunk-anchored**, **forward** | |
| window. Only the step count, LoRA, head, query tokens, rate and chunk match, so a difference between them cannot be | |
| attributed to anything. Their `unnorm_key`s are also built from `anhdao69/data_real_4tasks`' episode order, which | |
| differs from the control's, so **the held-out 5 episodes are probably not the same 5** β check the split first. The | |
| control's per-step errors are in the code repo (`acdc_la/results/stack_bowls_openvla_delta_perstep.npz`) so a | |
| properly matched TERRA arm can be compared **paired**. Neither has been on the robot. | |
| ### `piper_multi_anchored_step2500`, `piper_multi_anchored_step5500` β β experimental | |
| One model for **both** tasks (press_button x2 + pick-can mixture), using **chunk-ANCHORED** deltas | |
| (`target[s+k] - q[s]`, `q` read once per chunk) so that executing several entries open loop is valid. Better than the | |
| per-step parameterisation from k>=2, slightly worse at k=0. Trained with the old backward sample window (magnitudes | |
| under-predicted ~0.8x). **`press_button_anchored_fwd_step1500` is the first checkpoint with the window fixed** | |
| (press-button only, snapshot of a run still training): its error stops growing with chunk position β it levels | |
| off near 3.5Β° instead of climbing to 13Β° β and it will supersede these for that task. Measured on the 5 held-out | |
| press_button episodes, mean error over the steps actually executed: **exec 2 / exec 4 = 1.50 / 2.28Β°** for the | |
| window-fixed snapshot, 1.44 / 2.65Β° for `piper_multi_anchored_step5500` (old window), 1.66 / 3.50Β° for the | |
| per-step `press_button_terra_delta` under ideal tracking. Both anchored models are **snapshots of runs still | |
| training** (1500 of 3000, and step 5500) and **neither has been on the robot**. Different deployment call | |
| (`chunk_targets`, no ensembling) β read their READMEs. | |
| --- | |
| ## LIBERO policies | |
| Standard eval, 50 trials x 10 tasks = **500 episodes**, continuous flow head: | |
| ```bash | |
| python -m acdc_la.downstream.vla.run_libero_eval_cont --pretrained_checkpoint <dir> | |
| ``` | |
| | dir | suite | demos | base | latent | success | | |
| |---|---|---|---|---|---| | |
| | `libero_goal_full` | Goal | full | LoRA-scale phase 2 | ours | **94.4 %** | | |
| | `libero_10_full` | Long | full | LoRA-scale phase 2 | ours | **89.0 %** | | |
| | `libero_10_frac10_terra_p2fft20k` | Long | 10 % (43) | **`p2fft_20k`** | ours | **60.6 %** | | |
| | `libero_goal_frac10_univla_cont` | Goal | 10 % (43) | openvla-7b | **UniVLA's** | 89.4 % | | |
| | `libero_goal_frac10_none` | Goal | 10 % (43) | openvla-7b | **none** | 88.6 % | | |
| | `libero_goal_frac10_terra` | Goal | 10 % (43) | openvla-7b | ours | 86.8 % | | |
| ### The one result in this repo that is a clean, single-factor comparison β and it is negative | |
| The three `libero_goal_frac10_*` rows differ **only** in the latent channel. The **no-latent control lands between | |
| the two latent arms** (88.6 % against 89.4 % and 86.8 %), so at 10 % Goal demos through this interface the latent | |
| channel is **decorative**: the action head reads what it needs from the visual features alone. 13 episodes in 500 is | |
| ~1.3Ο, one seed each, and a single 500-episode cell carries **Β±8 points at 95 %** once training randomness is | |
| counted β so these three are one band, not a ranking. | |
| What the triplet *does* settle: an earlier 9.2 % collapse on Goal with the same latent and no query tokens was an | |
| **interface** failure, not the latent. What it does not cover: regimes where the visual features are insufficient. | |
| LIBERO-Long at 10 % is that regime (nothing saturates there: 60.6 % for the best arm) but with one replicate per arm | |
| it still cannot separate 60.6 / 58.7 / 56.4. The place where a matched control *does* separate is the real robot: | |
| `press_button_openvla_delta` is 2.1x worse than ours and cannot read the instruction at all. | |
| Not published here: the full-split 20k-phase-2 Long arm (89.8 %, our best Long number) and the remaining 10 % Long | |
| replicates live on a cluster filesystem, not on this host. | |
| --- | |
| ## `p2fft_20k` β the phase-2 base (not a deployable policy) | |
| The **base** every TERRA policy above is post-trained from: OpenVLA-7B fully fine-tuned for 20,000 steps on OXE to | |
| predict our continuous latent action (`latp2_inv`). `vla_full.pt` (15,081,838,026 B) is the VLA state dict, and | |
| **`action_decoder.pt` (105,268,830 B) is required too** β `finetune_libero_ctrl.py:273` initialises the post-training | |
| heads from it, so a run without it starts from a random head and will not reproduce these policies. | |
| ```bash | |
| huggingface-cli download Potestates/TERRA --include "p2fft_20k/*" --local-dir . | |
| INIT=./p2fft_20k ... bash acdc_la/scripts/run_vla_post2_portable.sh | |
| ``` | |
| The matched control is `INIT=none LATENT=none`, which post-trains the official `openvla/openvla-7b` with everything | |
| else identical β that is how "what did phase 2 buy" is measured, and it is what the two `*_openvla_delta` | |
| directories are. | |
| --- | |
| ## One metric trap, worth repeating | |
| `action_l1_1step` in every `train_log.jsonl` is **oracle-conditioned**: `FlowHeads.loss` scores a one-step | |
| correction from a point that already contains the ground truth. On the pick-can pair it ranks the two arms | |
| **backwards** (TERRA 0.0563 vs control 0.0648, i.e. TERRA "13% better", while both free-running read-outs favour the | |
| control). Use it to track a single run or pick a checkpoint within one run β **never to compare two arms.** Rank with | |
| `--replay`. | |
| `HANDOFF_press_button.md` at the root of this repo is the brief for training the next real-robot task on a fresh | |
| host. The full experiment ledger, including everything retracted or superseded, is `LEDGER.md` in the code repo. | |
| --- | |
| *This card is version-controlled at `acdc_la/hf_cards/TERRA_root_README.md` in the code repo and is written | |
| from both hosts. **Merge into it, do not overwrite it** β two hosts replaced each other's version of this file | |
| on 2026-11-12 before anyone noticed.* | |