Potestates commited on
Commit
7fb9a0d
Β·
verified Β·
1 Parent(s): 922d001

READMEs: model catalogue at the root, one README per directory

Browse files
README.md CHANGED
@@ -1,43 +1,47 @@
1
- # TERRA β€” post-trained policies (LIBERO, and real-robot arms)
2
 
3
- OpenVLA-7B with TERRA latent-action phase 2 (`latp2_inv` labels), post-trained with a flow-matching head and 4 query
4
- tokens. One directory per policy: merged weights (`model-*.safetensors`), action/latent heads (`action_decoder.pt`),
5
- `dataset_statistics.json` (action un-normalisation), `ctrl_meta.json`, processor / tokenizer.
 
6
 
7
- **Code:** https://github.com/DingTianxingjian/Latent_Free β€” branch `newton`, public, no token needed.
8
 
9
  ```bash
10
  git clone -b newton https://github.com/DingTianxingjian/Latent_Free.git latent_free && cd latent_free
 
11
  ```
12
 
13
- | dir | suite | demos | standard LIBERO (500 ep.) |
14
- |---|---|---|---|
15
- | `libero_10_full` | libero_10_no_noops | full | 89.0 |
16
- | `libero_goal_full` | libero_goal_no_noops | full | 94.4 |
17
- | `piper_pick_can_10k` | piper_pick_can (real robot, AgileX Piper) | 142 | n/a β€” real robot, see its README |
18
- | `press_button_terra_delta` | press_button, 4 colours (real robot, Piper) β€” **DELTA actions**: send `q + delta`, see its README | 95 | n/a β€” real robot, see its README |
19
- | `press_button_openvla_delta` | the matched **control** for the row above (official openvla-7b, same recipe): 1.92 vs 0.92 deg, ignores the instruction β€” not for deployment | 95 | n/a |
20
 
21
- Evaluate the LIBERO ones with `acdc_la.downstream.vla.run_libero_eval_cont --pretrained_checkpoint <dir>`.
22
- The real-robot directory carries its own README; the deployment wrapper, the measured chunk-length and
23
- jitter settings, and the offline check are in `acdc_la/realrobot/` in the repo.
 
 
 
 
 
 
 
24
 
25
- ## `p2fft_20k` β€” the phase-2 pretrained base
26
 
27
- The **base** the policies above are post-trained from, not a deployable policy: OpenVLA-7B fully fine-tuned for
28
- 20,000 steps on OXE to predict our continuous latent action (`latp2_inv`). `vla_full.pt` (15,081,838,026 B) is the
29
- VLA state dict and **`action_decoder.pt` (105,268,830 B) is required too** β€” `finetune_libero_ctrl.py:273`
30
- initialises the post-training heads from it, so a run without it starts from a random head.
31
 
32
- ```bash
33
- huggingface-cli download Potestates/TERRA --include "p2fft_20k/*" --local-dir .
34
- INIT=./p2fft_20k ... bash acdc_la/scripts/run_vla_post2_portable.sh
35
- ```
 
 
 
 
36
 
37
- The matched control is `INIT=none`, which post-trains the official `openvla/openvla-7b` with everything else
38
- identical β€” that is how "what did phase 2 buy" is measured.
39
 
40
- `HANDOFF_press_button.md` at the root of this repo is the brief for training the next real-robot task on a
41
- Jetstream2 g3.xl.
42
 
43
- The method ledger for all of this is `LEDGER.md` at the repository root.
 
 
1
+ # TERRA β€” model checkpoints
2
 
3
+ OpenVLA-7B + TERRA latent-action phase 2 (`latp2_inv` labels), post-trained with a flow-matching action head and
4
+ 4 query tokens. **One directory = one model**, each with its own README (what it is, how to deploy it, what has been
5
+ measured). Every deployable directory holds merged weights (`model-*.safetensors`), the action head
6
+ (`action_decoder.pt`, required), `dataset_statistics.json` (un-normalisation), `ctrl_meta.json` and the processor.
7
 
8
+ **Code:** https://github.com/DingTianxingjian/Latent_Free β€” branch `newton` (deployment wrapper: `acdc_la/realrobot/piper_policy.py`).
9
 
10
  ```bash
11
  git clone -b newton https://github.com/DingTianxingjian/Latent_Free.git latent_free && cd latent_free
12
+ huggingface-cli download Potestates/TERRA --include "<dir>/*" --local-dir .
13
  ```
14
 
15
+ ## Which model? (εΏ«ι€Ÿι€‰ζ‹©)
 
 
 
 
 
 
16
 
17
+ | dir | role | task | action (dims 0-5) | how to run it on the Piper | status |
18
+ |---|---|---|---|---|---|
19
+ | `press_button_terra_delta` | **policy** | press the {blue, orange, pink, red} button | per-step **delta** `target βˆ’ q` | `to_joint_targets(a, q)`, re-read `q` EVERY tick, **chunk = 1** | **pressed the button on the robot** (chunk=1; slightly off-centre). chunk β‰₯ 2 bends the direction β€” don't |
20
+ | `piper_multi_anchored_step5500` | **policy, one model for both tasks** | press button (4 colours) + pick up the can and place it on the plate | **chunk-anchored delta** `target[s+k] βˆ’ q[s]` | read `q` ONCE per chunk, `chunk_targets(chunk, q)`, execute N = 1 / 2 / 4 | experimental, latest snapshot (step 5500); not yet run on the robot |
21
+ | `piper_multi_anchored_step2500` | same run, earlier snapshot | same | same | same | superseded by step 5500; kept for comparison |
22
+ | `piper_pick_can_10k` | policy | `pick up the can and place it on the plate` | **absolute** joint target | send `chunk[k]` directly | offline only (3.24 deg) |
23
+ | `press_button_openvla_delta` | **control (baseline)** | press_button | per-step delta | as `press_button_terra_delta` | official OpenVLA-7B, same recipe: 1.92 vs 0.92 deg, **ignores the colour word** β€” for the comparison, not for use |
24
+ | `libero_10_full` | policy (sim) | LIBERO-10 | LIBERO EEF delta | `run_libero_eval_cont` | 89.0 % (500 episodes) |
25
+ | `libero_goal_full` | policy (sim) | LIBERO-Goal | LIBERO EEF delta | `run_libero_eval_cont` | 94.4 % (500 episodes) |
26
+ | `p2fft_20k` | **pretrained base**, not a policy | β€” | β€” | `INIT=./p2fft_20k` for post-training | phase 2, 20k steps on OXE |
27
 
28
+ ### The three action spaces on the Piper β€” getting this wrong drives the arm somewhere else
29
 
30
+ `ctrl_meta.json β†’ action_space` says which one a directory uses, and `PiperPolicy` reads it.
 
 
 
31
 
32
+ * **absolute** (no `action_space` key): `chunk[k]` IS the joint target.
33
+ * **delta** (`"delta"`): `chunk[k]` = target βˆ’ measured joints *at step k*. Send `q_now + chunk[k]`, `q` re-read every
34
+ tick. Use **chunk = 1**: the demos were teleoperated at 30 Hz with a continuous command, so the demonstrated arm
35
+ runs ~0.86–2 deg/step *ahead* of where an arm that settles at each 10 Hz command would be; with chunk = N that lead
36
+ adds up linearly and with OPPOSITE signs on j1/j2 (βˆ’0.74 / +0.75 deg per step on pick-can) β€” the direction bends.
37
+ * **delta_anchored** (`"delta_anchored"`): `chunk[k]` = target[s+k] βˆ’ joints read ONCE at the chunk's start. Send
38
+ `q_s + chunk[k]`; the targets do not depend on how the arm tracks inside the chunk, so the tracking drift above
39
+ does not exist β€” what remains is the model's own error, which grows with k (see the directory's README).
40
 
41
+ The method ledger is `LEDGER.md` in the code repo. `HANDOFF_press_button.md` at the root of this repo is the brief
42
+ for training a new real-robot task.
43
 
44
+ ## `p2fft_20k` β€” the phase-2 pretrained base
 
45
 
46
+ See `p2fft_20k/README.md`. Post-training from it: `INIT=./p2fft_20k ... bash acdc_la/scripts/run_vla_post2_portable.sh`;
47
+ the matched control is `INIT=none` (official `openvla/openvla-7b`, everything else identical).
libero_10_full/README.md ADDED
@@ -0,0 +1,11 @@
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # `libero_10_full` β€” LIBERO simulation policy (libero_10_no_noops)
2
+
3
+ TERRA phase 2 (`p2fft_20k`) post-trained on the full `libero_10_no_noops` demonstrations, 40000 steps, flow-matching head,
4
+ 4 query tokens, chunk of 12. Merged weights; `action_decoder.pt` is the head (required).
5
+
6
+ **Standard LIBERO evaluation (500 episodes): 89.0 %.**
7
+
8
+ ```bash
9
+ python -m acdc_la.downstream.vla.run_libero_eval_cont --pretrained_checkpoint <this dir>
10
+ ```
11
+ Action: the standard LIBERO 7-D end-effector delta + gripper. Simulation only β€” not for the Piper.
libero_goal_full/README.md ADDED
@@ -0,0 +1,11 @@
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # `libero_goal_full` β€” LIBERO simulation policy (libero_goal_no_noops)
2
+
3
+ TERRA phase 2 (`p2fft_20k`) post-trained on the full `libero_goal_no_noops` demonstrations, 30000 steps, flow-matching head,
4
+ 4 query tokens, chunk of 12. Merged weights; `action_decoder.pt` is the head (required).
5
+
6
+ **Standard LIBERO evaluation (500 episodes): 94.4 %.**
7
+
8
+ ```bash
9
+ python -m acdc_la.downstream.vla.run_libero_eval_cont --pretrained_checkpoint <this dir>
10
+ ```
11
+ Action: the standard LIBERO 7-D end-effector delta + gripper. Simulation only β€” not for the Piper.
p2fft_20k/README.md ADDED
@@ -0,0 +1,16 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # `p2fft_20k` β€” the TERRA phase-2 pretrained base (NOT a deployable policy)
2
+
3
+ OpenVLA-7B **fully fine-tuned for 20,000 steps on OXE** (`acdc_oxe29` mixture) to predict our continuous latent
4
+ action (`latp2_inv`, dz 256, 4 query tokens). Every policy in this repo is post-trained from it.
5
+
6
+ | file | what |
7
+ |---|---|
8
+ | `vla_full.pt` (15,081,838,026 B) | the VLA state dict |
9
+ | `action_decoder.pt` (105,268,830 B) | the heads β€” **required**: `finetune_libero_ctrl.py` initialises the post-training heads from it; without it the head starts random |
10
+ | the rest | processor / tokenizer / `ctrl_meta.json` |
11
+
12
+ ```bash
13
+ huggingface-cli download Potestates/TERRA --include "p2fft_20k/*" --local-dir .
14
+ INIT=./p2fft_20k DS=<your rlds dataset> ... bash acdc_la/scripts/run_vla_post2_portable.sh
15
+ ```
16
+ The matched control (to measure what phase 2 buys) is `INIT=none`: official `openvla/openvla-7b`, same recipe.
piper_multi_anchored_step2500/README.md CHANGED
@@ -1,5 +1,18 @@
1
  # `piper_multi_anchored_step2500` β€” EXPERIMENTAL, Piper press-button + pick-can, CHUNK-ANCHORED delta
2
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3
  Same base/recipe as `press_button_terra_delta` (OpenVLA-7B + TERRA phase 2 `p2fft_20k`, LoRA r32 merged, flow head,
4
  4 query tokens), trained 2,500 steps on the mixture press_button (x2) + `jachansantiago/pick_can_place_plate_150_trimmed`.
5
 
 
1
  # `piper_multi_anchored_step2500` β€” EXPERIMENTAL, Piper press-button + pick-can, CHUNK-ANCHORED delta
2
 
3
+ **Superseded by `piper_multi_anchored_step5500`** (same run, later). Kept only for comparison.
4
+
5
+ **One model, two tasks** β€” pick the task with `unnorm_key` and the instruction:
6
+
7
+ | task | `unnorm_key` | instruction |
8
+ |---|---|---|
9
+ | press button | `press_button_anc` | `press the {blue, orange, pink, red} button` |
10
+ | pick can | `piper_pick_can_anc` | `pick up the can and place it on the plate` (the only instruction in the data) |
11
+
12
+ **How many steps per chunk.** The per-step-delta rule "chunk = 1" (tracking drift) does NOT apply here: anchored
13
+ targets do not depend on how the arm tracks within the chunk. What limits N is the model's own error growing with k
14
+ (table below). Start with N = 1, then try 2 and 4 and compare. Offline numbers exist for press_button only.
15
+
16
  Same base/recipe as `press_button_terra_delta` (OpenVLA-7B + TERRA phase 2 `p2fft_20k`, LoRA r32 merged, flow head,
17
  4 query tokens), trained 2,500 steps on the mixture press_button (x2) + `jachansantiago/pick_can_place_plate_150_trimmed`.
18
 
piper_multi_anchored_step5500/README.md CHANGED
@@ -1,5 +1,18 @@
1
  # `piper_multi_anchored_step5500` β€” EXPERIMENTAL, Piper press-button + pick-can, CHUNK-ANCHORED delta
2
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3
  Same base/recipe as `press_button_terra_delta` (OpenVLA-7B + TERRA phase 2 `p2fft_20k`, LoRA r32 merged, flow head,
4
  4 query tokens), trained 5,500 steps on the mixture press_button (x2) + `jachansantiago/pick_can_place_plate_150_trimmed`.
5
 
@@ -20,7 +33,7 @@ while True:
20
  ```
21
  Do NOT use `to_joint_targets` per tick and do not ensemble (the policy refuses / disables it).
22
 
23
- ## Status β€” why it is experimental (step 5500 numbers: pending, see LEDGER; table below is step 2500)
24
  Held-out press_button (5 episodes), open-loop target error by chunk position k (deg, mean over joints):
25
 
26
  | k | 0 | 1 | 2 | 3 | 7 | 11 |
 
1
  # `piper_multi_anchored_step5500` β€” EXPERIMENTAL, Piper press-button + pick-can, CHUNK-ANCHORED delta
2
 
3
+ **Latest snapshot of this run β€” use this one.** Same run as `piper_multi_anchored_step2500`, 3,000 more steps.
4
+
5
+ **One model, two tasks** β€” pick the task with `unnorm_key` and the instruction:
6
+
7
+ | task | `unnorm_key` | instruction |
8
+ |---|---|---|
9
+ | press button | `press_button_anc` | `press the {blue, orange, pink, red} button` |
10
+ | pick can | `piper_pick_can_anc` | `pick up the can and place it on the plate` (the only instruction in the data) |
11
+
12
+ **How many steps per chunk.** The per-step-delta rule "chunk = 1" (tracking drift) does NOT apply here: anchored
13
+ targets do not depend on how the arm tracks within the chunk. What limits N is the model's own error growing with k
14
+ (table below). Start with N = 1, then try 2 and 4 and compare. Offline numbers exist for press_button only.
15
+
16
  Same base/recipe as `press_button_terra_delta` (OpenVLA-7B + TERRA phase 2 `p2fft_20k`, LoRA r32 merged, flow head,
17
  4 query tokens), trained 5,500 steps on the mixture press_button (x2) + `jachansantiago/pick_can_place_plate_150_trimmed`.
18
 
 
33
  ```
34
  Do NOT use `to_joint_targets` per tick and do not ensemble (the policy refuses / disables it).
35
 
36
+ ## Status β€” why it is experimental
37
  Held-out press_button (5 episodes), open-loop target error by chunk position k (deg, mean over joints):
38
 
39
  | k | 0 | 1 | 2 | 3 | 7 | 11 |