JingwuLuo/ur3_sim_pick_and_place_cube

π0.5 policies for UR3 cube pick-and-place: simulation pre-training and fine-tuning on real recordings

openpi π0.5 policies of a dual-arm UR3 workcell in which one arm picks one of three cubes and puts it on the table, the plate, the white box or another cube. Four groups of checkpoints:

1. Stage 1, simulation only, 500 episodes (stage1_sim500/): the action expert trained on 500 scripted MuJoCo episodes (JingwuLuo/ur3_sim_pick_and_place_cube, revision 7fa3c586).

2. Stage 1, simulation only, 1000 episodes (stage1_sim1000/): the action expert trained on 1000 scripted episodes of the same simulator (revision 5cd89567 of the same dataset) with relays between the two arms and perturbed approaches.

3. Stage 2, real recordings (stage2_lora_real/): LoRA adapters on the action expert, trained on 223 teleoperated episodes of the real workcell, starting from the stage1_sim500/ stage-1 checkpoint of the same label. See the real-robot test below: these did not perform the task on the robot.

4. Stage 2 on a mixture (stage2_mix_sim1000_real/): the full action expert, starting from the stage1_sim1000/ stage-1 checkpoint of the same label, trained on the 223 real episodes with simulated rows mixed into every batch, the simulated share falling during the run. One policy also takes the robot's state as input, switched on gradually.

Within each group the policies differ only in how the 10-point action chunk is labelled; everything else (data, model, recipe) is identical.

policy	label	points per chunk	reach of one chunk
per-frame	the next 10 control frames	10	10 frames (0.33 s)
two-phase	evenly spaced in time over 48 frames, one point per frame around grasp, lift and release	10	10 to 48 frames
curvature	curvature-resampled over 48 frames: points placed by path curvature, denser per unit of joint path in turns (for example the reversal at a grasp or release), with no extra density in time or at gripper events	10	48 frames (1.6 s)

folder	stage	steps published
stage1_sim500/<policy>/	1, 500 simulated episodes	5000, 9999 (the last)
stage1_sim1000/<policy>/	1, 1000 simulated episodes	14999 (the last)
stage2_lora_real/<policy>/	2, LoRA adapters on the real recordings	3999 (the last)
stage2_mix_sim1000_real/<policy>/	2, mixture of real and simulated rows	7999 (the last)

<policy> is per_frame, two_phase or curvature in every stage folder, and additionally curvature_state in stage2_mix_sim1000_real/ (curvature with the gated state input). Each step folder is one checkpoint, for example stage2_lora_real/curvature/3999/. The --name used when serving is the configuration name listed in the serving table below, not the folder name.

Real-robot test

Closed-loop trials on the real workcell with the policies served as described below (qualitative; success rates were not recorded):

• the stage-1 curvature policy of stage1_sim500/ (simulation only) performs the task;
• a policy trained on the real episodes only (see Results) performs the task;
• the stage-2 policies of stage2_lora_real/ (LoRA adapters) do not perform the task at full adapter scale; with the adapter update scaled by 0.5 the task sometimes succeeds.

The open-loop table below did not predict this: stage 2 has lower real held-out error than stage 1. The stage1_sim1000/ policies have not been run on the robot. No closed-loop result in the simulator is reported.

Results

The table is open loop. It covers stage1_sim500/ and stage2_lora_real/; the stage1_sim1000/ policies have not been evaluated. The evaluation feeds recorded observations to the served policy and compares its output with the recording, so it does not measure task success. Setup: 5 real held-out episodes (raw recording indices 54, 102, 122, 180, 185) and 10 simulated held-out episodes (raw indices 2, 100, 231, 234, 243, 287, 323, 325, 349, 463); neither set is in the training data of either stage. The server runs with --uniform --cap 1 so that its targets line up with the recorded frames (the deployment timing is faster than the recording and cannot be compared frame by frame); a new inference every 8 frames. Each cell is the median, in rad, of the largest error over the 12 joints:

• points: the model's 10 points against the label's 10 points, per inference;
• head: the first 8 targets of each response against the recorded actions, per inference;
• stitched: the targets of consecutive responses joined, 8 per response, against the recorded actions, per frame (padding rows excluded).

label	model	simulated held-out: points / head / stitched	real held-out: points / head / stitched
per-frame	stage 1	0.138 / 0.112 / 0.045	0.157 / 0.130 / 0.066
per-frame	stage 2	0.174 / 0.140 / 0.066	0.132 / 0.108 / 0.055
two-phase	stage 1	0.236 / 0.106 / 0.046	0.296 / 0.119 / 0.063
two-phase	stage 2	0.276 / 0.122 / 0.054	0.229 / 0.107 / 0.055
curvature	stage 1	0.472 / 0.110 / 0.053	0.418 / 0.116 / 0.061
curvature	stage 2	0.499 / 0.135 / 0.067	0.382 / 0.107 / 0.058
curvature	trained on the real episodes only, see below	0.522 / 0.154 / 0.078	0.288 / 0.087 / 0.052

• Stage 2 lowers the real held-out error of every label against stage 1 and raises the simulated held-out error.
• The curvature policy trained on the real episodes only (full action expert from pi05_base, 20,000 steps, folder 223TrialsTrainData/pi05_ur3_cube_quick_v4h5_curvature_s48_pp/19999 of JingwuLuo/pick_and_place_cube_quick_ur3) has lower real held-out error than every stage-2 policy and higher simulated held-out error.
• Open-loop error did not predict the closed-loop outcome on the real robot (see above).
• "points" compares against labels of different reach, so compare it within a label, not across labels.
• Gripper commands: every recorded open/close edge appears in the stitched trajectory of every stage-1 and stage-2 policy, on both held-out sets.

Stage 1: the simulated data

Raw episodes: JingwuLuo/ur3_sim_pick_and_place_cube, revision 7fa3c586627f34b55d179fbc4e762e5e9106ba44 (500 episodes, 81,087 steps at 30 Hz, 3.2–10.9 s per episode, median 4.9 s). The dataset card describes how the episodes were generated. The recordings used here, as the dataset card describes them:

• Scripted from the simulator's privileged state (object poses); not teleoperated and not produced by a learned policy; only successful episodes are kept. The scripted motion is faster and smoother than the teleoperated recordings of the real workcell.
• In every episode one arm moves and the other stays at its start pose (240 episodes left arm, 260 right arm). 83 episodes have a late close and regrasp: the arm reaches the grasp point with open fingers, pauses and starts lifting, the gripper closes on air too late, reopens after 0.1–0.3 s, and at least 0.7 s later the arm descends straight back and grasps the cube (two close and two open commands in these episodes). The transport distance from pick to place is short in 172 episodes, medium in 240 and long in 88.
• Cubes: pink 173, blue 180, orange 147 episodes. Where the cube starts: table 270, plate 98, white box 88, on another cube 44. Where it is put: table 160, plate 105, white box 92, on another cube 143.
• Camera set: the top camera at 240×320 and two wrist cameras at 240×424, 30 Hz, the 14-dimensional state and action of the real workcell (left joints 0–5, left gripper 6, right joints 7–12, right gripper 13; grippers 0 open, 1 closed). Both wrist cameras sit on the table side of their grippers, as on the real rig.

Preparation before training (the converted datasets are not uploaded):

1. Conversion of the HDF5 episodes to a LeRobot dataset.
2. Instructions. The episodes carry no language, so five phrasings per episode were generated from each episode's sim/task group (the arm is not named; 211 distinct sentences over the 2,500): Put the pink cube on the white box., Pick up the pink cube from the table and place it on the white box., Move the pink block from the table onto the white box., Could you pick the pink cube up from the table and set it on the white box?, pink cube from table to white box. One of the five is drawn at random per training sample.
3. Still frames. Stretches of at least 5 consecutive frames in which the action changes by less than 0.002 per frame (rad for the joints) were dropped (6,475 rows), and 60 frames of the final pose were appended to each episode as padding rows (30,000 rows), so that the policy learns to hold still after the task is done. 104,612 rows in total.
4. Keyframes (grasp, rest, lift, release) were detected per arm from the gripper signal and the arm's motion; rest is the first frame after the grasp at which the arm has stopped (joint speed below 0.003 rad per frame for 3 frames), because the simulator issues the close command while the arm is still decelerating. In all 500 episodes the number of detected grasps and releases equals the number of gripper commands the simulator recorded. The two-phase label uses the keyframes.
5. Labels. The per-frame, two-phase and curvature action-chunk labels were built from the timed episodes.
6. Held-out episodes. 10 episodes (raw indices 2, 100, 231, 234, 243, 287, 323, 325, 349, 463; 7 right arm, 3 left arm, 2 late-close; 5 short, 4 medium and 1 long transport) are excluded from training; the other 490 episodes (102,500 rows) are the training set.

Stage 1 on 1000 simulated episodes (stage1_sim1000/)

Raw episodes: JingwuLuo/ur3_sim_pick_and_place_cube, revision 5cd89567862dbe659abc900edc81b5467cb81ce1 (1000 episodes, 268,700 steps at 30 Hz, 3.4–17.2 s per episode, median 9.0 s). The dataset card describes how the episodes were generated. As the card describes them:

• Scripted from the simulator's privileged state; not teleoperated and not produced by a learned policy; only successful episodes are kept.
• 388 episodes: one arm does the task while the other stays still (right 203, left 185). 612 episodes are relays: the picking arm cannot reach the place, so it puts the cube on a handover spot in the middle of the table and returns; the other arm starts when the first one begins its return, takes the cube and puts it at the place.
• 252 episodes are perturbed: in one leg the arm is pushed 1–5 cm off its path during the approach (105) or the transport (147), stops, and corrects from there; the pushed steps are tagged.
• 236 episodes have a late close and regrasp. Pick-to-place distances are spread evenly from 0.08 to about 0.80 m.
• Cubes: pink 351, orange 323, blue 326. Camera set, state and action as in the 500-episode set.

Preparation before training (the converted datasets are not uploaded):

1. Instructions. Five phrasings per episode (231 distinct sentences over the 5,000), generated from each episode's sim/task group; the arm and the relay are not mentioned. One of the five is drawn at random per training sample.
2. Still frames. Stretches of at least 5 consecutive frames in which the action changes by less than 0.002 per frame were dropped (13,776 rows: 425 leading, 12,325 trailing, 1,026 pauses outside keyframe windows); 41,750 still frames inside keyframe windows were kept. 60 frames of the final pose were appended to each episode as padding rows (60,000 rows). 314,924 rows in total.
3. Keyframes (grasp and release of each arm). In all 1000 episodes the detected grasps and releases equal the gripper commands the simulator recorded. The two-phase label uses them.
4. Labels. The per-frame, two-phase and curvature action-chunk labels (same configurations as stage1_sim500/). Chunk points that lie on a pushed step get label weight 0 and do not enter the loss: 0.82 % (per-frame), 1.12 % (curvature) and 0.56 % (two-phase) of the points. The median reach of one chunk is 10.0 frames (per-frame), 20.3 (two-phase) and 48.0 (curvature).
5. Held-out episodes. 40 episodes, stratified (relay 27 and single arm 13; left arm 15 and right arm 25; perturbed 11; late close 8), are excluded from training; the other 960 episodes (302,218 rows) are the training set.

Training: the same as stage 1 of stage1_sim500/ except the data and the length.

item	value
start	pi05_base, action horizon 10
trained	action expert (Gemma 300M) and the action and time projections; SigLIP and Gemma 2B frozen
actions, state input	joints as deltas from the state, grippers absolute, quantile normalisation per chunk point; no state input
data	302,218 simulated rows
optimisation	batch 32, 15,000 steps (about 1.6 passes), AdamW (β2 0.95, weight decay 1e-4, gradient clipping 1.0), cosine learning rate 5e-5 to 5e-6 after 500 warm-up steps
weight averaging	EMA, decay 0.999; the saved params are the averaged weights
parallelism	one GPU per policy, no model sharding; each run trained from step 0 to step 14,999 without interruption

Batch loss in normalised space (the labels predict different quantities, so the values are not comparable); the logged loss is that of the training weights, not of the averaged weights that are saved. Every array of every published checkpoint is finite.

label	step 0	mean of steps 14,500–14,950
per-frame	0.171	0.0173
two-phase	0.170	0.0167
curvature	0.184	0.0187

The statistics in assets/ are constructed as for stage1_sim500/ (next section), from the 960 training episodes: the wrist-2 range at the last chunk point is 0.002 rad in the simulation and 0.05 (left) and 0.214–0.391 (right) after the union with the real recordings.

Stage 2 on a mixture of real and simulated rows (stage2_mix_sim1000_real/)

Starting weights: the stage1_sim1000/ stage-1 checkpoint (step 14999) of the same label. Trained: the action expert (Gemma 300M) and the action and time projections, as in stage 1; SigLIP and Gemma 2B frozen. Data: every row of a batch comes from the simulated training set of stage1_sim1000/ (960 episodes, 302,218 rows) with a probability that falls linearly from 0.5 at the first step to 0.1 at the last, and from the 223 real training episodes (62,410 rows) otherwise; each set is traversed in shuffled passes. Rows of the simulated set at pushed steps keep the zero label weight of stage 1. Prompts, labels, normalisation statistics and the 5 real held-out episodes are those of the sections above.

item	value
optimisation	batch 32, 8,000 steps, AdamW (β2 0.95, weight decay 1e-4, gradient clipping 1.0), cosine learning rate 2.5e-5 to 2.5e-6 after 300 warm-up steps
weight averaging	EMA, decay 0.999; the saved params are the averaged weights
parallelism	one GPU per policy, no model sharding

Batch loss in normalised space, over the mixed batches (the labels predict different quantities, so the values are not comparable; the logged loss is that of the training weights, not of the averaged weights that are saved). Every array of every published checkpoint is finite.

policy	step 0	mean of steps 7,500–7,950
per-frame	0.043	0.0183
two-phase	0.055	0.0156
curvature	0.074	0.0180
curvature, gated state input	0.075	0.0176

Gated state input (curvature_state/, configuration pi05_ur3_sim2_mix_curvature_s48_state_pp). pi0.5 can take the robot's state as tokens in the prompt: the 14-dimensional state, normalised with the statistics in assets/, in 256 bins (State: 12 87 ...). Stage 1 was trained without them, so during this run each training sample carries the state tokens with a probability that rises linearly from 0 at the first step to 1 at step 6,000 and is 1 for the last 2,000 steps; a sample without them has exactly the prompt of stage 1. The model is therefore always used with the state tokens after training. The state statistics of this policy differ from the others' in one entry: the range of the left arm's wrist-2 (state dimension 4, 0.0012 rad in the data) is widened to 0.05 rad around its centre, as the same joint's range is for the actions, because its encoder noise would otherwise fill the whole token range. The 14-dimensional observation/state the client sends is the one described under "Request and response"; the left arm stays parked in the real recordings, so its state tokens barely change there.

The policies of this group have no open-loop or closed-loop evaluation reported here.

Stage 2: the real recordings

Teleoperated demonstrations of the real dual-arm UR3 workcell (30 Hz, the same three cameras and 14-dimensional state and action): only the right arm moves and the left arm stays parked; an orange, blue or pink cube is put on a white box, a plate or another cube. 223 training episodes (62,410 rows) and 5 held-out episodes. The recordings of the real workcell and their preparation are described in JingwuLuo/pick_and_place_cube_quick_ur3; the 223-episode set is not uploaded. The prompts are that repository's: five phrasings per episode (480 distinct sentences), one drawn at random per sample, for example Put the orange cube on the white box.

Normalisation statistics

assets/<repo_id>/norm_stats.json inside each checkpoint holds the quantile statistics (q01, q99) the model was trained with: state (14) and actions per chunk point (10, 14), joints as deltas from the observed state, grippers absolute. The file of a stage-2 checkpoint has the same content as the file of the stage-1 checkpoint of the same label.

They are not the statistics of the simulated data alone. In the simulated episodes the wrist-2 joints (state and action dimensions 4 and 11) barely move: the q01–q99 range of their per-frame delta is 0.002 rad, against 0.21 rad for the right wrist-2 of the real recordings. Normalising real fine-tuning data with simulation-only statistics would scale those targets by about 140. For each label, the q01 of every statistic is therefore the minimum and the q99 the maximum of the simulated and the real recordings' statistics, in both stages. For the simulation-only training this means the model is trained with targets normalised by the wider range. Left-arm wrist-2 has a range floor of 0.05 rad. Do not replace this file when serving or fine-tuning.

label	left joints 0–5, range over the 10 points (rad)	right joints 7–12, range over the 10 points (rad)
per-frame	0.050–0.863	0.027–0.884
two-phase	0.050–1.933	0.113–1.848
curvature	0.050–2.550	0.102–2.506

(smallest and largest q99 − q01 over the 10 chunk points and the joints of the arm; the gripper dimensions have range 1.)

Training

item	stage 1	stage 2
start	pi05_base, action horizon 10	the stage-1 checkpoint at step 9999 of the same label
trained	action expert (Gemma 300M) and the action and time projections, about 430 M parameters; SigLIP and Gemma 2B frozen	LoRA adapters (rank 32) on the attention and feed-forward layers of the action expert only; SigLIP, Gemma 2B, the action expert's own weights and the projections frozen; the adapters start as a no-op
actions	joints as deltas from the state, grippers absolute; quantile normalisation per chunk point	same
state input	none	none
data	102,500 simulated rows	62,410 real rows
optimisation	batch 32, 10,000 steps (about 3 passes), AdamW (β2 0.95, weight decay 1e-4, gradient clipping 1.0), cosine learning rate 5e-5 to 5e-6 after 500 warm-up steps	batch 32, 4,000 steps (about 2 passes), same optimiser, cosine learning rate 1e-4 to 1e-5 after 200 warm-up steps
weight averaging	EMA, decay 0.999; the saved params are the averaged weights	EMA, decay 0.998; the LoRA arrays in the saved params are the averaged weights; see below for the frozen arrays
prompt	one of the episode's five phrasings, drawn at random per sample	same
parallelism	one GPU per policy, no model sharding	same, 1.3 s per step

Batch loss in normalised space (the three labels predict different quantities, so the values are not comparable). The logged loss is that of the training weights, not of the averaged weights that are saved.

label	stage 1: step 0 / mean of steps 9,500–9,950	stage 2: mean within 100 steps of step 100 / of step 3,900
per-frame	0.16 / 0.0138	0.064 / 0.0226
two-phase	0.17 / 0.0126	0.0665 / 0.0205
curvature	0.18 / 0.0125	0.0899 / 0.0262

Every array of every published checkpoint is finite. Each run trained from step 0 to the last step without interruption.

The frozen arrays of the stage-2 params are not the EMA of the training run. The averaging step decay * old + (1 - decay) * new runs in the dtype of the array; for the frozen bfloat16 arrays the decay 0.998 rounds to 0.99609375, so averaging an unchanged array is not the identity: all 51 frozen arrays of the averaged weights differed from their stage-1 values by about 0.3 in relative norm (the 10 LoRA arrays are float32 and unaffected; at decay 0.999 the averaging is the identity for bfloat16). The training-objective loss of the curvature policy's averaged params on real training batches was 0.2385, against 0.0265 logged. The published stage-2 params are therefore rebuilt: the 51 frozen arrays are taken from the stage-1 step-9999 checkpoint (stage 2 does not change them) and the 10 LoRA arrays from the stage-2 EMA at step 3999. The training-objective loss of the rebuilt curvature checkpoint is 0.0293 on real training batches; the other two were rebuilt the same way, and the open-loop table above was measured with the rebuilt checkpoints. The optimiser state is not published.

Contents of a checkpoint

<stage1_sim500 | stage1_sim1000 | stage2_lora_real | stage2_mix_sim1000_real>/<per_frame | two_phase | curvature | curvature_state>/<step>/
  params/                                   model weights (orbax)
  assets/<repo_id>/norm_stats.json          quantile statistics: state (14), actions per chunk point (10, 14)
  assets/<repo_id>/arc_resample.json        label configuration and robot block (30 Hz, joint velocity limits, gripper holds)
  _CHECKPOINT_METADATA

params of a stage-1 checkpoint is the EMA of the training weights of the action expert. params of a stage-2 checkpoint holds every array of the model including the LoRA adapters, so a stage-2 folder is complete and does not need the stage-1 folder to be served. The optimiser state is not included, so training cannot be resumed from these files; they can be used for serving and as the initial weights of a new run (params).

<repo_id>:

policy	stage 1 stage1_sim500/	stage 1 stage1_sim1000/	stage 2 stage2_lora_real/ and stage2_mix_sim1000_real/
per-frame	ur3-sim-cube-time-s10-train	ur3-sim2-time-s10-train	pick-and-place-cube-quick-v4h5-time-s10-train
two-phase	ur3-sim-cube-two-phase-s48-train	ur3-sim2-two-phase-s48-train	pick-and-place-cube-quick-v4h5-two-phase-s48-train
curvature	ur3-sim-cube-curvature-s48-d0-train	ur3-sim2-curvature-s48-d0-train	pick-and-place-cube-quick-v4h5-curvature-s48-d0-train

The keyframes field of arc_resample.json names the file the keyframes were read from; the path was shortened to the file name for this upload and is not used when serving. In a stage-2 file the robot block carries the real gripper's hold times (0.50 s after a close, 0.52 s after an open), which the server applies; in a stage1_sim1000/ file it carries the simulated gripper's (0.30 s after a close, 0.50 s after an open).

Serving

• NVIDIA GPU with CUDA 12. JAX preallocates 75% of GPU memory by default; set XLA_PYTHON_CLIENT_MEM_FRACTION=0.5 to share the GPU.
• openpi with the patches in luo-jingw/VLA_AGENT integrations/openpi/patches/ (all of them) applied, and VLA_AGENT (commit c3d22ea or later for the stage2_mix_sim1000_real/ policies, which need patches 0004 and 0005; commit 0baeb4d or later for the --lora serving flag of stage2_lora_real/) installed in the same environment (pip install -e <VLA_AGENT>).
• openpi fetches the PaliGemma tokenizer (gs://big_vision/paligemma_tokenizer.model) on first use; without internet access, copy ~/.cache/openpi/big_vision/ from a machine that has it (or set OPENPI_DATA_HOME).

Download one policy, stage 2 curvature as the example:

python -c "from huggingface_hub import snapshot_download; snapshot_download('JingwuLuo/ur3_sim_pick_and_place_cube', \
  allow_patterns=['stage2_lora_real/curvature/3999/**'], local_dir='ckpt')"

Serve it, from the openpi root:

export PYTHONPATH=scripts:<VLA_AGENT>/src:<VLA_AGENT>/examples/ur3_dual_arm
python <VLA_AGENT>/examples/ur3_dual_arm/run.py serve \
  --name pi05_ur3_sim_real_curvature_s48_pp \
  --repo-id pick-and-place-cube-quick-v4h5-curvature-s48-d0-train \
  --data-root /unused \
  --checkpoint ckpt/stage2_lora_real/curvature/3999 \
  --lora --init-checkpoint /unused \
  --port 8000 --cap 4 --velocity-fraction 0.5

--name is the configuration name and --repo-id the assets/ folder name of the checkpoint; --data-root is not read when serving. A stage-2 policy needs --lora, which builds the model with the LoRA adapters the weights contain; --init-checkpoint is required together with --lora but its path is not read when serving. The other policies:

policy	stage	--name	--repo-id	extra flags
per-frame	1 (stage1_sim500/)	pi05_ur3_sim_time_s10_pp	ur3-sim-cube-time-s10-train	none
two-phase	1 (stage1_sim500/)	pi05_ur3_sim_two_phase_s48_pp	ur3-sim-cube-two-phase-s48-train	none
curvature	1 (stage1_sim500/)	pi05_ur3_sim_curvature_s48_pp	ur3-sim-cube-curvature-s48-d0-train	none
per-frame	1 (stage1_sim1000/)	pi05_ur3_sim2_time_s10_pp	ur3-sim2-time-s10-train	none
two-phase	1 (stage1_sim1000/)	pi05_ur3_sim2_two_phase_s48_pp	ur3-sim2-two-phase-s48-train	none
curvature	1 (stage1_sim1000/)	pi05_ur3_sim2_curvature_s48_pp	ur3-sim2-curvature-s48-d0-train	none
per-frame	2 (stage2_mix_sim1000_real/)	pi05_ur3_sim2_mix_time_s10_pp	pick-and-place-cube-quick-v4h5-time-s10-train	none
two-phase	2 (stage2_mix_sim1000_real/)	pi05_ur3_sim2_mix_two_phase_s48_pp	pick-and-place-cube-quick-v4h5-two-phase-s48-train	none
curvature	2 (stage2_mix_sim1000_real/)	pi05_ur3_sim2_mix_curvature_s48_pp	pick-and-place-cube-quick-v4h5-curvature-s48-d0-train	none
curvature, gated state input	2 (stage2_mix_sim1000_real/)	pi05_ur3_sim2_mix_curvature_s48_state_pp	pick-and-place-cube-quick-v4h5-curvature-s48-d0-train	--state-input
per-frame	2 (stage2_lora_real/)	pi05_ur3_sim_real_time_s10_pp	pick-and-place-cube-quick-v4h5-time-s10-train	--lora --init-checkpoint /unused
two-phase	2 (stage2_lora_real/)	pi05_ur3_sim_real_two_phase_s48_pp	pick-and-place-cube-quick-v4h5-two-phase-s48-train	--lora --init-checkpoint /unused
curvature	2 (stage2_lora_real/)	pi05_ur3_sim_real_curvature_s48_pp	pick-and-place-cube-quick-v4h5-curvature-s48-d0-train	--lora --init-checkpoint /unused

No --absolute-joints flag: all of them predict deltas. Only the gated-state policy of stage2_mix_sim1000_real/ takes the state as input (--state-input); the others take none. Always pass --lora for a stage2_lora_real/ checkpoint, never for stage1_sim500/, stage1_sim1000/ or stage2_mix_sim1000_real/. Without it the server starts without an error, builds a model without adapters, and answers differently to the same input (the adapters are not applied); nothing in the log says so. The state policy of stage2_mix_sim1000_real/ must be served with --state-input (without it the prompt has no state tokens, the format of the other policies, and the answers differ); the state has to be sent as observation/state in both cases.

The two-phase and curvature policies do not output the next 10 control frames; their 10 points lie along the next 10–48 frames. The server fits a cubic spline from the observed joints through the points and re-times it with velocity timing (no joint faster than velocity_fraction × the UR3 limits, 3.14 rad/s for the base, shoulder and elbow and 6.28 rad/s for the wrists; never faster than cap × the recordings' pace; the arm holds still for 0.50 s after a gripper close and 0.52 s after an open), returning one absolute target per 30 Hz control frame. Execute the server's actions, never the raw points. Do not pass --uniform for these two policies; --uniform --cap 1 (points at fixed frame spacing) is meaningful for the per-frame policy only.

On a real arm:

• The teleoperation recordings never exceeded 2.5 rad/s; start with --velocity-fraction 0.5 and raise it after the first trials.
• The number of targets per response varies with the path; read len(actions) and execute at most that many, the first 4–8 before querying again with a fresh observation.
• Around a gripper switch the response repeats the same target for the hold time; keep sending those repeated targets, they are what makes the arm wait for the gripper. Close the right gripper when actions[:, 13] > 0.5.
• Send the measured joints as observation/state; the spline starts there, so a stale state makes the first targets jump. With asynchronous execution, drop the first d targets of a new response, d = latency × 30 frames.
• The left arm stayed parked in all stage-2 recordings. Keep it where it is (dimensions 0–6 only reproduce its parked pose).

Request and response

Client: openpi_client.websocket_client_policy.WebsocketClientPolicy(host, port).infer(obs).

key	value
observation/image	top camera, uint8 H×W×3 RGB (240×320 in the data)
observation/left_wrist_image	left wrist camera, uint8 H×W×3 (240×424)
observation/right_wrist_image	right wrist camera, uint8 H×W×3 (240×424)
observation/state	float32 (14): left joints 0–5 (rad), left gripper 6, right joints 7–12 (rad), right gripper 13; a gripper entry is the last gripper command sent (0 open, 1 closed), not the measured opening
prompt	English instruction, e.g. Put the pink cube on the white box.

Response: actions float32 (N, 14), one absolute target per 30 Hz control frame (actions[j] is the target for frame t + j + 1 after the observation); arc_points (10, 14), the model's points (absolute); arc_point_frames, the frames the points are placed at. The state is added to the predicted deltas by the server and is also the start of the spline.

The instructions do not name the arm; the model has to infer from the scene which arm moves. Name the cube's colour (pink, orange, blue), where it is (table, plate, white box, or "on the cube") and where it goes. Stage 2 saw only right-arm episodes; its behaviour on a scene where the left arm should move has not been evaluated. The stage1_sim1000/ policies saw both arms, including relays in which one arm hands the cube to the other.

Known limits

• The stage2_mix_sim1000_real/ policies are not evaluated here, neither open loop nor closed loop; the state policy has seen the state tokens only in the later part of its run.
• Stage 1 saw simulation only: no real image, joint trajectory or language annotation.
• Stage 2 saw 223 real episodes of one workcell, in which only the right arm moves.
• The idle arm's wrist-camera images in the simulation are bit-identical from frame to frame (492 runs of identical frames in the episodes, all on the idle arm), unlike a real camera.
• The mount side of the wrist cameras matches the real rig; the rest of the mount pose and the field of view come from the gripper model. The real wrist views frame the fingers differently: the fingertips appear only along the top edge, while in the simulation the gripper body fills the upper part of the image.
• Wrist-2 of both arms is almost constant in the simulation, so these dimensions carry little signal there (see above).
• The scripted motion is smoother and faster than teleoperation and contains no failures other than the deliberate late close followed by a re-grasp.
• Stage 2 raises the open-loop error on the simulated held-out episodes (table above), and the stage-2 policies did not perform the task on the real robot (see above).
• The stage1_sim1000/ policies have no open-loop or closed-loop evaluation yet; the only measurement reported for them is the training loss.