READMEs: model catalogue at the root, one README per directory
Browse files- README.md +33 -29
- libero_10_full/README.md +11 -0
- libero_goal_full/README.md +11 -0
- p2fft_20k/README.md +16 -0
- piper_multi_anchored_step2500/README.md +13 -0
- piper_multi_anchored_step5500/README.md +14 -1
README.md
CHANGED
|
@@ -1,43 +1,47 @@
|
|
| 1 |
-
# TERRA β
|
| 2 |
|
| 3 |
-
OpenVLA-7B
|
| 4 |
-
tokens.
|
| 5 |
-
|
|
|
|
| 6 |
|
| 7 |
-
**Code:** https://github.com/DingTianxingjian/Latent_Free β branch `newton`
|
| 8 |
|
| 9 |
```bash
|
| 10 |
git clone -b newton https://github.com/DingTianxingjian/Latent_Free.git latent_free && cd latent_free
|
|
|
|
| 11 |
```
|
| 12 |
|
| 13 |
-
|
| 14 |
-
|---|---|---|---|
|
| 15 |
-
| `libero_10_full` | libero_10_no_noops | full | 89.0 |
|
| 16 |
-
| `libero_goal_full` | libero_goal_no_noops | full | 94.4 |
|
| 17 |
-
| `piper_pick_can_10k` | piper_pick_can (real robot, AgileX Piper) | 142 | n/a β real robot, see its README |
|
| 18 |
-
| `press_button_terra_delta` | press_button, 4 colours (real robot, Piper) β **DELTA actions**: send `q + delta`, see its README | 95 | n/a β real robot, see its README |
|
| 19 |
-
| `press_button_openvla_delta` | the matched **control** for the row above (official openvla-7b, same recipe): 1.92 vs 0.92 deg, ignores the instruction β not for deployment | 95 | n/a |
|
| 20 |
|
| 21 |
-
|
| 22 |
-
|
| 23 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 24 |
|
| 25 |
-
##
|
| 26 |
|
| 27 |
-
|
| 28 |
-
20,000 steps on OXE to predict our continuous latent action (`latp2_inv`). `vla_full.pt` (15,081,838,026 B) is the
|
| 29 |
-
VLA state dict and **`action_decoder.pt` (105,268,830 B) is required too** β `finetune_libero_ctrl.py:273`
|
| 30 |
-
initialises the post-training heads from it, so a run without it starts from a random head.
|
| 31 |
|
| 32 |
-
```
|
| 33 |
-
|
| 34 |
-
|
| 35 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 36 |
|
| 37 |
-
The
|
| 38 |
-
|
| 39 |
|
| 40 |
-
`
|
| 41 |
-
Jetstream2 g3.xl.
|
| 42 |
|
| 43 |
-
|
|
|
|
|
|
| 1 |
+
# TERRA β model checkpoints
|
| 2 |
|
| 3 |
+
OpenVLA-7B + TERRA latent-action phase 2 (`latp2_inv` labels), post-trained with a flow-matching action head and
|
| 4 |
+
4 query tokens. **One directory = one model**, each with its own README (what it is, how to deploy it, what has been
|
| 5 |
+
measured). Every deployable directory holds merged weights (`model-*.safetensors`), the action head
|
| 6 |
+
(`action_decoder.pt`, required), `dataset_statistics.json` (un-normalisation), `ctrl_meta.json` and the processor.
|
| 7 |
|
| 8 |
+
**Code:** https://github.com/DingTianxingjian/Latent_Free β branch `newton` (deployment wrapper: `acdc_la/realrobot/piper_policy.py`).
|
| 9 |
|
| 10 |
```bash
|
| 11 |
git clone -b newton https://github.com/DingTianxingjian/Latent_Free.git latent_free && cd latent_free
|
| 12 |
+
huggingface-cli download Potestates/TERRA --include "<dir>/*" --local-dir .
|
| 13 |
```
|
| 14 |
|
| 15 |
+
## Which model? (εΏ«ιιζ©)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 16 |
|
| 17 |
+
| dir | role | task | action (dims 0-5) | how to run it on the Piper | status |
|
| 18 |
+
|---|---|---|---|---|---|
|
| 19 |
+
| `press_button_terra_delta` | **policy** | press the {blue, orange, pink, red} button | per-step **delta** `target β q` | `to_joint_targets(a, q)`, re-read `q` EVERY tick, **chunk = 1** | **pressed the button on the robot** (chunk=1; slightly off-centre). chunk β₯ 2 bends the direction β don't |
|
| 20 |
+
| `piper_multi_anchored_step5500` | **policy, one model for both tasks** | press button (4 colours) + pick up the can and place it on the plate | **chunk-anchored delta** `target[s+k] β q[s]` | read `q` ONCE per chunk, `chunk_targets(chunk, q)`, execute N = 1 / 2 / 4 | experimental, latest snapshot (step 5500); not yet run on the robot |
|
| 21 |
+
| `piper_multi_anchored_step2500` | same run, earlier snapshot | same | same | same | superseded by step 5500; kept for comparison |
|
| 22 |
+
| `piper_pick_can_10k` | policy | `pick up the can and place it on the plate` | **absolute** joint target | send `chunk[k]` directly | offline only (3.24 deg) |
|
| 23 |
+
| `press_button_openvla_delta` | **control (baseline)** | press_button | per-step delta | as `press_button_terra_delta` | official OpenVLA-7B, same recipe: 1.92 vs 0.92 deg, **ignores the colour word** β for the comparison, not for use |
|
| 24 |
+
| `libero_10_full` | policy (sim) | LIBERO-10 | LIBERO EEF delta | `run_libero_eval_cont` | 89.0 % (500 episodes) |
|
| 25 |
+
| `libero_goal_full` | policy (sim) | LIBERO-Goal | LIBERO EEF delta | `run_libero_eval_cont` | 94.4 % (500 episodes) |
|
| 26 |
+
| `p2fft_20k` | **pretrained base**, not a policy | β | β | `INIT=./p2fft_20k` for post-training | phase 2, 20k steps on OXE |
|
| 27 |
|
| 28 |
+
### The three action spaces on the Piper β getting this wrong drives the arm somewhere else
|
| 29 |
|
| 30 |
+
`ctrl_meta.json β action_space` says which one a directory uses, and `PiperPolicy` reads it.
|
|
|
|
|
|
|
|
|
|
| 31 |
|
| 32 |
+
* **absolute** (no `action_space` key): `chunk[k]` IS the joint target.
|
| 33 |
+
* **delta** (`"delta"`): `chunk[k]` = target β measured joints *at step k*. Send `q_now + chunk[k]`, `q` re-read every
|
| 34 |
+
tick. Use **chunk = 1**: the demos were teleoperated at 30 Hz with a continuous command, so the demonstrated arm
|
| 35 |
+
runs ~0.86β2 deg/step *ahead* of where an arm that settles at each 10 Hz command would be; with chunk = N that lead
|
| 36 |
+
adds up linearly and with OPPOSITE signs on j1/j2 (β0.74 / +0.75 deg per step on pick-can) β the direction bends.
|
| 37 |
+
* **delta_anchored** (`"delta_anchored"`): `chunk[k]` = target[s+k] β joints read ONCE at the chunk's start. Send
|
| 38 |
+
`q_s + chunk[k]`; the targets do not depend on how the arm tracks inside the chunk, so the tracking drift above
|
| 39 |
+
does not exist β what remains is the model's own error, which grows with k (see the directory's README).
|
| 40 |
|
| 41 |
+
The method ledger is `LEDGER.md` in the code repo. `HANDOFF_press_button.md` at the root of this repo is the brief
|
| 42 |
+
for training a new real-robot task.
|
| 43 |
|
| 44 |
+
## `p2fft_20k` β the phase-2 pretrained base
|
|
|
|
| 45 |
|
| 46 |
+
See `p2fft_20k/README.md`. Post-training from it: `INIT=./p2fft_20k ... bash acdc_la/scripts/run_vla_post2_portable.sh`;
|
| 47 |
+
the matched control is `INIT=none` (official `openvla/openvla-7b`, everything else identical).
|
libero_10_full/README.md
ADDED
|
@@ -0,0 +1,11 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# `libero_10_full` β LIBERO simulation policy (libero_10_no_noops)
|
| 2 |
+
|
| 3 |
+
TERRA phase 2 (`p2fft_20k`) post-trained on the full `libero_10_no_noops` demonstrations, 40000 steps, flow-matching head,
|
| 4 |
+
4 query tokens, chunk of 12. Merged weights; `action_decoder.pt` is the head (required).
|
| 5 |
+
|
| 6 |
+
**Standard LIBERO evaluation (500 episodes): 89.0 %.**
|
| 7 |
+
|
| 8 |
+
```bash
|
| 9 |
+
python -m acdc_la.downstream.vla.run_libero_eval_cont --pretrained_checkpoint <this dir>
|
| 10 |
+
```
|
| 11 |
+
Action: the standard LIBERO 7-D end-effector delta + gripper. Simulation only β not for the Piper.
|
libero_goal_full/README.md
ADDED
|
@@ -0,0 +1,11 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# `libero_goal_full` β LIBERO simulation policy (libero_goal_no_noops)
|
| 2 |
+
|
| 3 |
+
TERRA phase 2 (`p2fft_20k`) post-trained on the full `libero_goal_no_noops` demonstrations, 30000 steps, flow-matching head,
|
| 4 |
+
4 query tokens, chunk of 12. Merged weights; `action_decoder.pt` is the head (required).
|
| 5 |
+
|
| 6 |
+
**Standard LIBERO evaluation (500 episodes): 94.4 %.**
|
| 7 |
+
|
| 8 |
+
```bash
|
| 9 |
+
python -m acdc_la.downstream.vla.run_libero_eval_cont --pretrained_checkpoint <this dir>
|
| 10 |
+
```
|
| 11 |
+
Action: the standard LIBERO 7-D end-effector delta + gripper. Simulation only β not for the Piper.
|
p2fft_20k/README.md
ADDED
|
@@ -0,0 +1,16 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# `p2fft_20k` β the TERRA phase-2 pretrained base (NOT a deployable policy)
|
| 2 |
+
|
| 3 |
+
OpenVLA-7B **fully fine-tuned for 20,000 steps on OXE** (`acdc_oxe29` mixture) to predict our continuous latent
|
| 4 |
+
action (`latp2_inv`, dz 256, 4 query tokens). Every policy in this repo is post-trained from it.
|
| 5 |
+
|
| 6 |
+
| file | what |
|
| 7 |
+
|---|---|
|
| 8 |
+
| `vla_full.pt` (15,081,838,026 B) | the VLA state dict |
|
| 9 |
+
| `action_decoder.pt` (105,268,830 B) | the heads β **required**: `finetune_libero_ctrl.py` initialises the post-training heads from it; without it the head starts random |
|
| 10 |
+
| the rest | processor / tokenizer / `ctrl_meta.json` |
|
| 11 |
+
|
| 12 |
+
```bash
|
| 13 |
+
huggingface-cli download Potestates/TERRA --include "p2fft_20k/*" --local-dir .
|
| 14 |
+
INIT=./p2fft_20k DS=<your rlds dataset> ... bash acdc_la/scripts/run_vla_post2_portable.sh
|
| 15 |
+
```
|
| 16 |
+
The matched control (to measure what phase 2 buys) is `INIT=none`: official `openvla/openvla-7b`, same recipe.
|
piper_multi_anchored_step2500/README.md
CHANGED
|
@@ -1,5 +1,18 @@
|
|
| 1 |
# `piper_multi_anchored_step2500` β EXPERIMENTAL, Piper press-button + pick-can, CHUNK-ANCHORED delta
|
| 2 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3 |
Same base/recipe as `press_button_terra_delta` (OpenVLA-7B + TERRA phase 2 `p2fft_20k`, LoRA r32 merged, flow head,
|
| 4 |
4 query tokens), trained 2,500 steps on the mixture press_button (x2) + `jachansantiago/pick_can_place_plate_150_trimmed`.
|
| 5 |
|
|
|
|
| 1 |
# `piper_multi_anchored_step2500` β EXPERIMENTAL, Piper press-button + pick-can, CHUNK-ANCHORED delta
|
| 2 |
|
| 3 |
+
**Superseded by `piper_multi_anchored_step5500`** (same run, later). Kept only for comparison.
|
| 4 |
+
|
| 5 |
+
**One model, two tasks** β pick the task with `unnorm_key` and the instruction:
|
| 6 |
+
|
| 7 |
+
| task | `unnorm_key` | instruction |
|
| 8 |
+
|---|---|---|
|
| 9 |
+
| press button | `press_button_anc` | `press the {blue, orange, pink, red} button` |
|
| 10 |
+
| pick can | `piper_pick_can_anc` | `pick up the can and place it on the plate` (the only instruction in the data) |
|
| 11 |
+
|
| 12 |
+
**How many steps per chunk.** The per-step-delta rule "chunk = 1" (tracking drift) does NOT apply here: anchored
|
| 13 |
+
targets do not depend on how the arm tracks within the chunk. What limits N is the model's own error growing with k
|
| 14 |
+
(table below). Start with N = 1, then try 2 and 4 and compare. Offline numbers exist for press_button only.
|
| 15 |
+
|
| 16 |
Same base/recipe as `press_button_terra_delta` (OpenVLA-7B + TERRA phase 2 `p2fft_20k`, LoRA r32 merged, flow head,
|
| 17 |
4 query tokens), trained 2,500 steps on the mixture press_button (x2) + `jachansantiago/pick_can_place_plate_150_trimmed`.
|
| 18 |
|
piper_multi_anchored_step5500/README.md
CHANGED
|
@@ -1,5 +1,18 @@
|
|
| 1 |
# `piper_multi_anchored_step5500` β EXPERIMENTAL, Piper press-button + pick-can, CHUNK-ANCHORED delta
|
| 2 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3 |
Same base/recipe as `press_button_terra_delta` (OpenVLA-7B + TERRA phase 2 `p2fft_20k`, LoRA r32 merged, flow head,
|
| 4 |
4 query tokens), trained 5,500 steps on the mixture press_button (x2) + `jachansantiago/pick_can_place_plate_150_trimmed`.
|
| 5 |
|
|
@@ -20,7 +33,7 @@ while True:
|
|
| 20 |
```
|
| 21 |
Do NOT use `to_joint_targets` per tick and do not ensemble (the policy refuses / disables it).
|
| 22 |
|
| 23 |
-
## Status β why it is experimental
|
| 24 |
Held-out press_button (5 episodes), open-loop target error by chunk position k (deg, mean over joints):
|
| 25 |
|
| 26 |
| k | 0 | 1 | 2 | 3 | 7 | 11 |
|
|
|
|
| 1 |
# `piper_multi_anchored_step5500` β EXPERIMENTAL, Piper press-button + pick-can, CHUNK-ANCHORED delta
|
| 2 |
|
| 3 |
+
**Latest snapshot of this run β use this one.** Same run as `piper_multi_anchored_step2500`, 3,000 more steps.
|
| 4 |
+
|
| 5 |
+
**One model, two tasks** β pick the task with `unnorm_key` and the instruction:
|
| 6 |
+
|
| 7 |
+
| task | `unnorm_key` | instruction |
|
| 8 |
+
|---|---|---|
|
| 9 |
+
| press button | `press_button_anc` | `press the {blue, orange, pink, red} button` |
|
| 10 |
+
| pick can | `piper_pick_can_anc` | `pick up the can and place it on the plate` (the only instruction in the data) |
|
| 11 |
+
|
| 12 |
+
**How many steps per chunk.** The per-step-delta rule "chunk = 1" (tracking drift) does NOT apply here: anchored
|
| 13 |
+
targets do not depend on how the arm tracks within the chunk. What limits N is the model's own error growing with k
|
| 14 |
+
(table below). Start with N = 1, then try 2 and 4 and compare. Offline numbers exist for press_button only.
|
| 15 |
+
|
| 16 |
Same base/recipe as `press_button_terra_delta` (OpenVLA-7B + TERRA phase 2 `p2fft_20k`, LoRA r32 merged, flow head,
|
| 17 |
4 query tokens), trained 5,500 steps on the mixture press_button (x2) + `jachansantiago/pick_can_place_plate_150_trimmed`.
|
| 18 |
|
|
|
|
| 33 |
```
|
| 34 |
Do NOT use `to_joint_targets` per tick and do not ensemble (the policy refuses / disables it).
|
| 35 |
|
| 36 |
+
## Status β why it is experimental
|
| 37 |
Held-out press_button (5 episodes), open-loop target error by chunk position k (deg, mean over joints):
|
| 38 |
|
| 39 |
| k | 0 | 1 | 2 | 3 | 7 | 11 |
|