| # Handoff: the `press_button` arm on a Jetstream2 g3.xl |
| |
| Written 2026-11-10 by the workstation session for whoever picks this up on Jetstream2. Everything below that is a |
| measurement was measured; everything that is a guess says so. Read the "unverified" notes before trusting a number. |
| |
| ## 0. The repository |
| |
| ```bash |
| git clone -b newton https://github.com/DingTianxingjian/Latent_Free.git latent_free && cd latent_free |
| ``` |
| |
| **It is public -- no token, no credentials.** (The first reader of this document guessed `Potestates/Latent_Free` |
| from the HuggingFace org name and got a 404; the owner is `DingTianxingjian`.) |
| |
| ## 0b. Read these first, in this order |
| |
| 1. `AGENTS.md` (repo root) -- **the experiment ledger is `LEDGER.md` at the repo root, and it must be updated before |
| you report a result to the user.** Configuration, split, budget, the number, its control, conclusion status, |
| artifact paths, next decision. Do not leave a conclusion only in chat or in a subdirectory README. |
| 2. `LEDGER.md`, the entry **2026-11-10 "FIRST PHYSICAL-ROBOT feedback on the Piper policy"**. It has everything |
| already measured about this robot and policy family. Do not re-measure it. |
| 3. `next_task.md` section 8: **never discretise our latent for the VLA; never change a shared setting for one arm |
| only.** The second one is the live constraint here -- see section 6. |
| |
| ## 1. What this run is for |
| |
| `anhdao69/press_button_task` is the **first multi-task real-robot dataset in this project**. The pick-can set has |
| exactly one task, which means the instruction carried **zero information**: the policy could ignore the language |
| entirely and nothing would change. The press-button set contains at least `"press the red button"` and |
| `"press the blue button"` -- same scene, same motion class, the only disambiguator is the word. **That is the first |
| clean test of whether this policy family reads its instruction at all**, and it is worth more than a second task |
| would be on its own. Register that as the question before you train, not after. |
| |
| ## 2. The machine: what fits on a g3.xl |
| |
| | | | |
| |---|---| |
| | g3.xl | 32 vCPU, **120 GB system RAM**, **40 GB GPU RAM** (a full A100), 64 SU/h | |
| | smaller flavors | g3.large 20 GB GPU / 60 GB RAM; g3.medium 10 GB / 30 GB. **Neither is enough.** | |
| | multi-GPU | g3.2xl / g3.4xl exist but are **not available by default** -- a justified request to help@jetstream-cloud.org | |
| |
| **Measured on the workstation (RTX PRO 6000, 2026-11-10): training takes 39.9-40.8 GB of VRAM at micro-batch 8.** |
| That does not fit in 40 GB with any headroom. So: |
| |
| ``` |
| BS=4 GA=16 estimated ~28 GB <- use this |
| BS=8 GA=8 40 GB measured <- will OOM on an A100 |
| ``` |
| |
| **The global batch must stay 64** (`BS x GA x GPUs = 64`); the launcher asserts it. Every Piper and 10 % LIBERO cell |
| in this project is 64, and changing it makes the run incomparable with all of them. The ~28 GB figure is an |
| extrapolation from one measured point (fixed ~16 GB + activations linear in batch), **not measured** -- check |
| `nvidia-smi` at step 50 and drop to BS=2 GA=32 if it is close to the limit. |
| |
| Other facts worth having before you start: |
| |
| * **Disk: plan 80-100 GB and attach a volume.** `openvla-7b` is 15 GB, the env is ~11 GB, each merged checkpoint is |
| 16 GB, the phase-2 base (if you get it) is another 15 GB. |
| * **120 GB RAM is the one place Jetstream2 beats the workstation.** The standalone merge peaks at **30.7 GB** and was |
| SIGKILLed three times on the 62 GB workstation (LEDGER 2026-11-08). On g3.xl it is a non-issue. |
| * **Measure the first 100 steps before quoting an ETA to anyone.** On 2026-11-10 the workstation estimate of 13 h |
| became 18 h because another user's job was on the same card. `train_log.jsonl` has a `seconds` field; use it. |
| |
| ## 3. Environment |
| |
| Follow **`REPRODUCE.md`, the `univla_vla` recipe** (torch 2.2.0 + cu121, `pip install -e third_party/UniVLA`, the |
| pinned `tensorflow_metadata` / `protobuf` / `wandb`). That recipe targets sm80, which **is** the A100 -- do not copy |
| this workstation's env, which is torch 2.7.1+cu128 built for Blackwell. |
| |
| Two things that save time: |
| |
| * **Training never asks for flash-attention.** Grep confirms `attn_implementation` appears only in the two inference |
| scripts (`run_libero_eval_cont.py`, `realrobot/piper_policy.py`) and both use `"sdpa"`. So you do not need to build |
| flash-attn for sm80. |
| * **Conversion needs `pyarrow` and `av`** on top of the env (`cv2` too, which the env has). The workstation does not |
| have them, which is why this dataset was not converted there. |
| |
| ## 4. The dataset, and three things that will bite you |
| |
| Verified from `meta/info.json` and `meta/stats.json` (read without downloading the videos): |
| |
| ``` |
| robot_type piper | LeRobot codebase_version v3.0 | fps 30 |
| total_episodes 100 | total_frames 5533 | total_tasks 4 |
| action float32 [7] joint_1.pos .. joint_6.pos + gripper (ABSOLUTE joint positions, degrees) |
| observation.state float32 [14] |
| observation.images.front / .wrist video 480x640x3 mp4 |
| ``` |
| |
| Same robot, same action space and same layout as `pick_can_place_plate_150_trimmed`, so |
| `acdc_la/realrobot/convert_piper_rlds.py` reads it **without a code change** to its parsing. But: |
|
|
| ### (a) `GRIP_SCALE = 8.2` is WRONG for this dataset |
| |
| `convert_piper_rlds.py:20` divides the raw gripper by 8.2 because the pick-can set's raw gripper ranges 0 .. 8.15. |
| Here it does not: |
| |
| ``` |
| action dim 6: min 0.04 max 0.06 mean 0.05 std 0.01 |
| ``` |
| |
| Pressing a button never opens the gripper. `clip(act[:,6] / 8.2, 0, 1)` maps all of it to ~0.005 -- a **constant**. |
| Numerically harmless, but that dimension carries no signal, and if the two datasets are ever trained jointly the |
| gripper calibrations are incompatible. **Make a per-dataset decision and write it in the converter's docstring.** |
| |
| ### (b) The episodes are very short, and a 12-step chunk may not fit in them |
| |
| ``` |
| 5533 frames / 30 Hz, subsampled by 3 -> ~1,844 steps at 10 Hz (pick-can: 6,462, i.e. 3.5x more) |
| 1,844 / 100 episodes -> ~18 steps = 1.8 s per episode |
| window_size 12 = 1.2 s -> two thirds of an entire episode |
| RLDSDataset is built with window_size + 2 = 14 frames per sample -> ~5 valid windows per episode |
| ``` |
| |
| **SETTLED 2026-11-10 -- it pads, nothing masks it, and for this dataset that is most of your samples.** Measured by |
| running the real chunker on synthetic trajectories (`acdc_la/realrobot/short_episode.py`): |
| |
| * The window is **backward-looking**: timestep `t` gets indices `[t-13 ... t]`, then |
| `tf.maximum(chunk_indices, 0)` (`traj_transforms.py:91`) clamps anything before the start, so **frame 0 and |
| action 0 are repeated**. Actions are clamped the same way to `[0, traj_len-1]` (`:98`). |
| * **No sample is dropped** -- one per timestep, including `t=0`, whose entire 14-frame window is frame 0 and whose |
| entire action target is action 0. A fully degenerate sample. |
| * `pad_mask` records exactly which entries are padding (`:104`) and **nothing downstream reads it**: a grep over |
| `prismatic/vla/` and `acdc_la/` finds it used only by `acdc_la/scripts/test_obs_keep.py`. (The `pad_mask_dict` |
| that *is* used is a different thing -- missing images and language.) |
| * Fraction of samples with a fully real window, by episode length in 10 Hz steps: |
|
|
| | episode length | 12 (press_button shortest) | 17 (press_button median) | 43 (pick-can median) | ~203 (LIBERO Long) | |
| |---|---|---|---|---| |
| | clean samples | **0 %** | **24 %** | 70 % | 94 % | |
|
|
| So **press_button at 10 Hz is mostly padding** and its shortest episodes yield no clean sample at all. Compute the |
| exact fraction on your copy as `sum(max(0, T_i - 13)) / sum(T_i)` over episodes and put it in the ledger entry; the |
| effective sample count is what that gives, not 1,844. |
| |
| **Do not "fix" this by lowering `window_size`** -- it is shared with every LIBERO cell and with the baseline |
| (`next_task.md` section 8). The options, in the order I would raise them with the user: |
|
|
| 1. **Accept it and report it.** The contamination is identical on both sides of every arm, so an A-vs-baseline |
| comparison on this dataset stays valid; only the absolute number stops being comparable with pick-can. |
| 2. **Keep 30 Hz for this dataset instead of subsampling by 3.** Episodes are 38-167 frames, so a 14-frame window |
| leaves 25-154 clean samples -- most samples clean. But it changes the control rate and what a 12-step chunk |
| spans (0.4 s instead of 1.2 s), so it is a design decision, not a fix. Raise it; do not take it alone. |
| 3. `chunk_filter_fn` exists in the dataset API (`dataset.py:411-420`) and could drop padded chunks, but that changes |
| the effective data for one arm. |
|
|
| `prismatic/vla/datasets/rlds/traj_transforms.py::chunk_act_obs_libero` is the function; the test is one file. |
|
|
| ### (c) Overfitting is a real risk here -- unlike on pick-can |
|
|
| ``` |
| 1,844 steps of data / global batch 64 -> 29 steps per epoch |
| 10,000 steps -> 347 epochs (pick-can at 10k was 104) |
| 95/5 split of 100 episodes -> 95 train, only 5 held out |
| ``` |
|
|
| The collaborator's "is it under- or over-trained?" question was a **negative** for pick-can (held-out L1 flat from |
| step 6,000 to 10,000 while train L1 still fell). **On this dataset it will genuinely bite.** So: |
|
|
| * Pick a smaller budget. ~3,000 steps is ~104 epochs, matching pick-can's epoch count, which is the defensible |
| choice -- not 10,000. |
| * **Actually watch the held-out curve this time**, and snapshot checkpoints: `finetune_libero_ctrl.py:496` writes |
| every save to the SAME adapter dir, so each save overwrites the last and the run keeps **no history**. Copy |
| `<run_root>/adapters/<exp_id>` after each save or you cannot pick a step afterwards. |
| * The 4 tasks are **blue / orange / pink / red button, 25 episodes each** (confirmed on the real download |
| 2026-11-10). Four colours of the same motion is a *better* language test than I assumed from two. |
|
|
| ## 5. Registering the dataset |
|
|
| `convert_piper_rlds.py` builds a TFDS builder whose **dataset name comes from the class name** (`PiperPickCan` -> |
| `piper_pick_can`), and `piper_pick_can` is registered in prismatic by `acdc_la/patches/univla_piper.patch` (image |
| keys + the `EEF_POS` contract). For a new dataset, copy the converter to `convert_press_button_rlds.py`, rename the |
| builder class to `PressButton`, and add a `press_button` entry to the patch's two tables (`OXE_DATASET_CONFIGS` in |
| `oxe/configs.py` and `OXE_STANDARDIZATION_TRANSFORMS` in `oxe/transforms.py`) -- the Piper entries can be copied |
| verbatim, both datasets have the same image keys and action contract. |
|
|
| **A gotcha I already hit:** TFDS refuses to build a builder class whose module is `__main__` |
| ("Cannot access resources for '__main__'"). Run it as an import, not as a script: |
|
|
| ```bash |
| PYTHONPATH=.:third_party/UniVLA python -c " |
| import sys; sys.argv = ['x', '--src', '<lerobot dir>', '--out', 'data/real_rlds'] |
| from acdc_la.realrobot.convert_press_button_rlds import main; main()" |
| ``` |
|
|
| The shortcut of reusing the `piper_pick_can` registration under a different `--data_root_dir` works (that is what the |
| delta copy does) but gives both datasets the same key in `dataset_statistics.json`, which is a real footgun. Register |
| a proper name. |
|
|
| ## 6. Absolute or delta actions? Do not decide this alone |
|
|
| Two 10,000-step runs are in flight on the workstation **right now** testing exactly this on pick-can, both on delta |
| actions at global batch 64 (expected to finish ~2026-11-10 22:00 UTC): |
|
|
| * `tc_cont+piper_pick_can+b64x1+...--terra_piper_delta` -- our phase-2 base |
| * `none+piper_pick_can+b64x1+...--openvla_piper_delta` -- official openvla-7b, no latent mechanism (the baseline) |
|
|
| Why: the absolute parameterisation spends a 164 deg output range on a ~2 deg/step signal (1.2 %), against a measured |
| error of 2.4-11.5 % of each joint's range -- the signal sits below the error (LEDGER 2026-11-10). The delta is |
| `action[:6] - joint_state[:6]`, target minus the step's own **measured** configuration, which is self-correcting at |
| deployment; `acdc_la/realrobot/make_delta_rlds.py` derives a delta RLDS from an absolute one in ~3 minutes with the |
| frames bit-identical. |
|
|
| **Recommendation: convert to absolute, immediately derive the delta copy, and hold the training choice** until those |
| two runs land. Then use whichever they support -- and use the SAME choice for every arm, per `next_task.md` section 8. |
| The registered prediction for the delta runs, so you can check it: ~1.0-1.5 deg open-loop (the nulls are 1.20 deg for |
| "stay where you are" and the demonstration floor is 0.61 deg; the absolute arm is 3.24 deg). |
|
|
| ## 7. Commands |
|
|
| ```bash |
| # baseline arm -- runs TODAY with no missing artifacts, openvla-7b downloads from HF |
| PY=<env>/bin/python INIT=none LATENT=none QUERY_TOKENS=4 \ |
| DS=press_button EXTRA="--data_root_dir $PWD/data/real_rlds --merge False" \ |
| SKIP_EVAL=1 GLOBAL=64 BS=4 STEPS=3000 SAVE=500 \ |
| NOTE=openvla_press_button GPUS="0" RUNROOT=<volume>/vla_runs \ |
| bash acdc_la/scripts/run_vla_post2_portable.sh |
| |
| # our arm -- needs the phase-2 base, see section 8 |
| PY=<env>/bin/python INIT=<dir containing vla_full.pt> \ |
| DS=press_button EXTRA="--data_root_dir $PWD/data/real_rlds --merge False" \ |
| SKIP_EVAL=1 GLOBAL=64 BS=4 STEPS=3000 SAVE=500 \ |
| NOTE=terra_press_button GPUS="0" RUNROOT=<volume>/vla_runs \ |
| bash acdc_la/scripts/run_vla_post2_portable.sh |
| |
| # merge separately -- NEVER in-run (--merge False above). 30.7 GB peak, 211 s. |
| <env>/bin/python -m acdc_la.scripts.merge_ctrl_run --run <run dir> [--init <phase-2 dir>] |
| |
| # offline check, in DEGREES through the deployment image path |
| PYTHONPATH=.:third_party/UniVLA <env>/bin/python -m acdc_la.realrobot.piper_policy \ |
| --checkpoint <merged run dir> --replay data/real_rlds --dataset press_button |
| ``` |
|
|
| `SKIP_EVAL=1` is required: the LIBERO simulator eval does not apply to real-robot data. |
|
|
| ## 8. Getting the phase-2 base |
|
|
| It is on HuggingFace as of 2026-11-10, in the **public** `Potestates/TERRA` repo -- **no token, no login** |
| (verified anonymously: a range request on `p2fft_20k/vla_full.pt` with no credentials returns HTTP 206): |
|
|
| ```bash |
| huggingface-cli download Potestates/TERRA --include "p2fft_20k/*" --local-dir . |
| # then: INIT=./p2fft_20k ... |
| ``` |
|
|
| (There is also a private copy at `Potestates/TERRA-phase2-base`, kept as a backup. You do not need it.) |
|
|
| **Check you got both big files before training**, and against these exact sizes: |
|
|
| ``` |
| vla_full.pt 15,081,838,026 B the base the LoRA goes on |
| action_decoder.pt 105,268,830 B REQUIRED -- finetune_libero_ctrl.py:273 initialises the post-training |
| heads from it (strict=False). Without it the heads start RANDOM and the |
| run does not reproduce the workstation's arm. |
| ``` |
|
|
| Both arms therefore need nothing from us. **Start with the baseline anyway** (section 7, `INIT=none`): it is the |
| control the other arm is measured against, and it needs only `openvla/openvla-7b`. |
|
|
| ## 9. Do not |
|
|
| * Do not change `window_size`, `lora_rank`, `learning_rate`, the global batch, or `FlowHeads`' `n_latents` for this |
| arm alone. They are shared with every LIBERO cell and with the baseline; changing one side voids the comparison. |
| * Do not run the in-run merge (`--merge False` is in the commands for a reason). |
| * Do not report `action_l1_1step` from `train_log.jsonl` as a prediction error. It is **oracle-conditioned**: |
| `FlowHeads.loss` computes `(at + (1-t)*va - actions).abs()` where `at` already contains the ground-truth action. |
| It is valid for ranking checkpoints under one fixed protocol and for comparing two arms on the SAME dataset and |
| normalisation, and for nothing else. The number a robot would see comes from `piper_policy.py --replay`. |
| * Do not quote offline L1 as if it predicted success rate. It does not. |
|
|
