HANDOFF_press_button.md

Handoff: the press_button arm on a Jetstream2 g3.xl

Written 2026-11-10 by the workstation session for whoever picks this up on Jetstream2. Everything below that is a measurement was measured; everything that is a guess says so. Read the "unverified" notes before trusting a number.

0. The repository

git clone -b newton https://github.com/DingTianxingjian/Latent_Free.git latent_free && cd latent_free

It is public -- no token, no credentials. (The first reader of this document guessed Potestates/Latent_Free from the HuggingFace org name and got a 404; the owner is DingTianxingjian.)

0b. Read these first, in this order

1. AGENTS.md (repo root) -- the experiment ledger is LEDGER.md at the repo root, and it must be updated before you report a result to the user. Configuration, split, budget, the number, its control, conclusion status, artifact paths, next decision. Do not leave a conclusion only in chat or in a subdirectory README.
2. LEDGER.md, the entry 2026-11-10 "FIRST PHYSICAL-ROBOT feedback on the Piper policy". It has everything already measured about this robot and policy family. Do not re-measure it.
3. next_task.md section 8: never discretise our latent for the VLA; never change a shared setting for one arm only. The second one is the live constraint here -- see section 6.

1. What this run is for

anhdao69/press_button_task is the first multi-task real-robot dataset in this project. The pick-can set has exactly one task, which means the instruction carried zero information: the policy could ignore the language entirely and nothing would change. The press-button set contains at least "press the red button" and "press the blue button" -- same scene, same motion class, the only disambiguator is the word. That is the first clean test of whether this policy family reads its instruction at all, and it is worth more than a second task would be on its own. Register that as the question before you train, not after.

2. The machine: what fits on a g3.xl

g3.xl	32 vCPU, 120 GB system RAM, 40 GB GPU RAM (a full A100), 64 SU/h
smaller flavors	g3.large 20 GB GPU / 60 GB RAM; g3.medium 10 GB / 30 GB. Neither is enough.
multi-GPU	g3.2xl / g3.4xl exist but are not available by default -- a justified request to help@jetstream-cloud.org

Measured on the workstation (RTX PRO 6000, 2026-11-10): training takes 39.9-40.8 GB of VRAM at micro-batch 8. That does not fit in 40 GB with any headroom. So:

BS=4 GA=16    estimated ~28 GB      <- use this
BS=8 GA=8     40 GB measured        <- will OOM on an A100

The global batch must stay 64 (BS x GA x GPUs = 64); the launcher asserts it. Every Piper and 10 % LIBERO cell in this project is 64, and changing it makes the run incomparable with all of them. The ~28 GB figure is an extrapolation from one measured point (fixed ~16 GB + activations linear in batch), not measured -- check nvidia-smi at step 50 and drop to BS=2 GA=32 if it is close to the limit.

Other facts worth having before you start:

• Disk: plan 80-100 GB and attach a volume. openvla-7b is 15 GB, the env is ~11 GB, each merged checkpoint is 16 GB, the phase-2 base (if you get it) is another 15 GB.
• 120 GB RAM is the one place Jetstream2 beats the workstation. The standalone merge peaks at 30.7 GB and was SIGKILLed three times on the 62 GB workstation (LEDGER 2026-11-08). On g3.xl it is a non-issue.
• Measure the first 100 steps before quoting an ETA to anyone. On 2026-11-10 the workstation estimate of 13 h became 18 h because another user's job was on the same card. train_log.jsonl has a seconds field; use it.

3. Environment

Follow REPRODUCE.md, the univla_vla recipe (torch 2.2.0 + cu121, pip install -e third_party/UniVLA, the pinned tensorflow_metadata / protobuf / wandb). That recipe targets sm80, which is the A100 -- do not copy this workstation's env, which is torch 2.7.1+cu128 built for Blackwell.

Two things that save time:

• Training never asks for flash-attention. Grep confirms attn_implementation appears only in the two inference scripts (run_libero_eval_cont.py, realrobot/piper_policy.py) and both use "sdpa". So you do not need to build flash-attn for sm80.
• Conversion needs pyarrow and av on top of the env (cv2 too, which the env has). The workstation does not have them, which is why this dataset was not converted there.

4. The dataset, and three things that will bite you

Verified from meta/info.json and meta/stats.json (read without downloading the videos):

robot_type piper | LeRobot codebase_version v3.0 | fps 30
total_episodes 100 | total_frames 5533 | total_tasks 4
action      float32 [7]   joint_1.pos .. joint_6.pos + gripper   (ABSOLUTE joint positions, degrees)
observation.state   float32 [14]
observation.images.front / .wrist   video 480x640x3 mp4

Same robot, same action space and same layout as pick_can_place_plate_150_trimmed, so acdc_la/realrobot/convert_piper_rlds.py reads it without a code change to its parsing. But:

(a) GRIP_SCALE = 8.2 is WRONG for this dataset

convert_piper_rlds.py:20 divides the raw gripper by 8.2 because the pick-can set's raw gripper ranges 0 .. 8.15. Here it does not:

action dim 6:   min 0.04   max 0.06   mean 0.05   std 0.01

Pressing a button never opens the gripper. clip(act[:,6] / 8.2, 0, 1) maps all of it to ~0.005 -- a constant. Numerically harmless, but that dimension carries no signal, and if the two datasets are ever trained jointly the gripper calibrations are incompatible. Make a per-dataset decision and write it in the converter's docstring.

(b) The episodes are very short, and a 12-step chunk may not fit in them

5533 frames / 30 Hz, subsampled by 3  ->  ~1,844 steps at 10 Hz   (pick-can: 6,462, i.e. 3.5x more)
1,844 / 100 episodes                  ->  ~18 steps = 1.8 s per episode
window_size 12 = 1.2 s                ->  two thirds of an entire episode
RLDSDataset is built with window_size + 2 = 14 frames per sample  ->  ~5 valid windows per episode

SETTLED 2026-11-10 -- it pads, nothing masks it, and for this dataset that is most of your samples. Measured by running the real chunker on synthetic trajectories (acdc_la/realrobot/short_episode.py):

• The window is backward-looking: timestep t gets indices [t-13 ... t], then tf.maximum(chunk_indices, 0) (traj_transforms.py:91) clamps anything before the start, so frame 0 and action 0 are repeated. Actions are clamped the same way to [0, traj_len-1] (:98).

• No sample is dropped -- one per timestep, including t=0, whose entire 14-frame window is frame 0 and whose entire action target is action 0. A fully degenerate sample.

• pad_mask records exactly which entries are padding (:104) and nothing downstream reads it: a grep over prismatic/vla/ and acdc_la/ finds it used only by acdc_la/scripts/test_obs_keep.py. (The pad_mask_dict that is used is a different thing -- missing images and language.)

• Fraction of samples with a fully real window, by episode length in 10 Hz steps:

  episode length	12 (press_button shortest)	17 (press_button median)	43 (pick-can median)	~203 (LIBERO Long)
  clean samples	0 %	24 %	70 %	94 %

So press_button at 10 Hz is mostly padding and its shortest episodes yield no clean sample at all. Compute the exact fraction on your copy as sum(max(0, T_i - 13)) / sum(T_i) over episodes and put it in the ledger entry; the effective sample count is what that gives, not 1,844.

Do not "fix" this by lowering window_size -- it is shared with every LIBERO cell and with the baseline (next_task.md section 8). The options, in the order I would raise them with the user:

1. Accept it and report it. The contamination is identical on both sides of every arm, so an A-vs-baseline comparison on this dataset stays valid; only the absolute number stops being comparable with pick-can.
2. Keep 30 Hz for this dataset instead of subsampling by 3. Episodes are 38-167 frames, so a 14-frame window leaves 25-154 clean samples -- most samples clean. But it changes the control rate and what a 12-step chunk spans (0.4 s instead of 1.2 s), so it is a design decision, not a fix. Raise it; do not take it alone.
3. chunk_filter_fn exists in the dataset API (dataset.py:411-420) and could drop padded chunks, but that changes the effective data for one arm.

prismatic/vla/datasets/rlds/traj_transforms.py::chunk_act_obs_libero is the function; the test is one file.

(c) Overfitting is a real risk here -- unlike on pick-can

1,844 steps of data / global batch 64  ->  29 steps per epoch
10,000 steps                           ->  347 epochs      (pick-can at 10k was 104)
95/5 split of 100 episodes             ->  95 train, only 5 held out

The collaborator's "is it under- or over-trained?" question was a negative for pick-can (held-out L1 flat from step 6,000 to 10,000 while train L1 still fell). On this dataset it will genuinely bite. So:

• Pick a smaller budget. ~3,000 steps is ~104 epochs, matching pick-can's epoch count, which is the defensible choice -- not 10,000.
• Actually watch the held-out curve this time, and snapshot checkpoints: finetune_libero_ctrl.py:496 writes every save to the SAME adapter dir, so each save overwrites the last and the run keeps no history. Copy <run_root>/adapters/<exp_id> after each save or you cannot pick a step afterwards.
• The 4 tasks are blue / orange / pink / red button, 25 episodes each (confirmed on the real download 2026-11-10). Four colours of the same motion is a better language test than I assumed from two.

5. Registering the dataset

convert_piper_rlds.py builds a TFDS builder whose dataset name comes from the class name (PiperPickCan -> piper_pick_can), and piper_pick_can is registered in prismatic by acdc_la/patches/univla_piper.patch (image keys + the EEF_POS contract). For a new dataset, copy the converter to convert_press_button_rlds.py, rename the builder class to PressButton, and add a press_button entry to the patch's two tables (OXE_DATASET_CONFIGS in oxe/configs.py and OXE_STANDARDIZATION_TRANSFORMS in oxe/transforms.py) -- the Piper entries can be copied verbatim, both datasets have the same image keys and action contract.

A gotcha I already hit: TFDS refuses to build a builder class whose module is __main__ ("Cannot access resources for 'main'"). Run it as an import, not as a script:

PYTHONPATH=.:third_party/UniVLA python -c "
import sys; sys.argv = ['x', '--src', '<lerobot dir>', '--out', 'data/real_rlds']
from acdc_la.realrobot.convert_press_button_rlds import main; main()"

The shortcut of reusing the piper_pick_can registration under a different --data_root_dir works (that is what the delta copy does) but gives both datasets the same key in dataset_statistics.json, which is a real footgun. Register a proper name.

6. Absolute or delta actions? Do not decide this alone

Two 10,000-step runs are in flight on the workstation right now testing exactly this on pick-can, both on delta actions at global batch 64 (expected to finish ~2026-11-10 22:00 UTC):

• tc_cont+piper_pick_can+b64x1+...--terra_piper_delta -- our phase-2 base
• none+piper_pick_can+b64x1+...--openvla_piper_delta -- official openvla-7b, no latent mechanism (the baseline)

Why: the absolute parameterisation spends a 164 deg output range on a ~2 deg/step signal (1.2 %), against a measured error of 2.4-11.5 % of each joint's range -- the signal sits below the error (LEDGER 2026-11-10). The delta is action[:6] - joint_state[:6], target minus the step's own measured configuration, which is self-correcting at deployment; acdc_la/realrobot/make_delta_rlds.py derives a delta RLDS from an absolute one in ~3 minutes with the frames bit-identical.

Recommendation: convert to absolute, immediately derive the delta copy, and hold the training choice until those two runs land. Then use whichever they support -- and use the SAME choice for every arm, per next_task.md section 8. The registered prediction for the delta runs, so you can check it: ~1.0-1.5 deg open-loop (the nulls are 1.20 deg for "stay where you are" and the demonstration floor is 0.61 deg; the absolute arm is 3.24 deg).

7. Commands

# baseline arm -- runs TODAY with no missing artifacts, openvla-7b downloads from HF
PY=<env>/bin/python INIT=none LATENT=none QUERY_TOKENS=4 \
DS=press_button EXTRA="--data_root_dir $PWD/data/real_rlds --merge False" \
SKIP_EVAL=1 GLOBAL=64 BS=4 STEPS=3000 SAVE=500 \
NOTE=openvla_press_button GPUS="0" RUNROOT=<volume>/vla_runs \
  bash acdc_la/scripts/run_vla_post2_portable.sh

# our arm -- needs the phase-2 base, see section 8
PY=<env>/bin/python INIT=<dir containing vla_full.pt> \
DS=press_button EXTRA="--data_root_dir $PWD/data/real_rlds --merge False" \
SKIP_EVAL=1 GLOBAL=64 BS=4 STEPS=3000 SAVE=500 \
NOTE=terra_press_button GPUS="0" RUNROOT=<volume>/vla_runs \
  bash acdc_la/scripts/run_vla_post2_portable.sh

# merge separately -- NEVER in-run (--merge False above). 30.7 GB peak, 211 s.
<env>/bin/python -m acdc_la.scripts.merge_ctrl_run --run <run dir> [--init <phase-2 dir>]

# offline check, in DEGREES through the deployment image path
PYTHONPATH=.:third_party/UniVLA <env>/bin/python -m acdc_la.realrobot.piper_policy \
    --checkpoint <merged run dir> --replay data/real_rlds --dataset press_button

SKIP_EVAL=1 is required: the LIBERO simulator eval does not apply to real-robot data.

8. Getting the phase-2 base

It is on HuggingFace as of 2026-11-10, in the public Potestates/TERRA repo -- no token, no login (verified anonymously: a range request on p2fft_20k/vla_full.pt with no credentials returns HTTP 206):

huggingface-cli download Potestates/TERRA --include "p2fft_20k/*" --local-dir .
# then:  INIT=./p2fft_20k  ...

(There is also a private copy at Potestates/TERRA-phase2-base, kept as a backup. You do not need it.)

Check you got both big files before training, and against these exact sizes:

vla_full.pt          15,081,838,026 B    the base the LoRA goes on
action_decoder.pt       105,268,830 B    REQUIRED -- finetune_libero_ctrl.py:273 initialises the post-training
                                         heads from it (strict=False).  Without it the heads start RANDOM and the
                                         run does not reproduce the workstation's arm.

Both arms therefore need nothing from us. Start with the baseline anyway (section 7, INIT=none): it is the control the other arm is measured against, and it needs only openvla/openvla-7b.

9. Do not

• Do not change window_size, lora_rank, learning_rate, the global batch, or FlowHeads' n_latents for this arm alone. They are shared with every LIBERO cell and with the baseline; changing one side voids the comparison.
• Do not run the in-run merge (--merge False is in the commands for a reason).
• Do not report action_l1_1step from train_log.jsonl as a prediction error. It is oracle-conditioned: FlowHeads.loss computes (at + (1-t)*va - actions).abs() where at already contains the ground-truth action. It is valid for ranking checkpoints under one fixed protocol and for comparing two arms on the SAME dataset and normalisation, and for nothing else. The number a robot would see comes from piper_policy.py --replay.
• Do not quote offline L1 as if it predicted success rate. It does not.