PoP on the SO-101: pretraining and fine-tuning checkpoints

PoP (Pointer Policy) is told what to act on through referents: a 2D point on the object, a 2D point on the destination and a small skill code, never a sentence. This repository holds its pretraining checkpoint (SO-100 community data, no conditioning) and the checkpoints of a real-robot comparison on four SO-101 tasks: PoP against a sentence-conditioned policy with the same network, initialization, data and steps.

Fine-tuning checkpoints and validation loss

L1 error of the EMA weights on normalized 16-step action chunks of 20 held-out demonstrations (5 per task). This is an imitation error, not a task success rate: no checkpoint has been run on the robot yet.

Run Model Initialization Frames Seed Val L1, step 40k drawer towel push cups Lowest val L1 (step)
261121-so101-pop-s0 PoP pretraining labelled 0 0.298 0.354 0.281 0.414 0.205 0.278 (4k)
261121-so101-pop-s1 PoP pretraining labelled 1 0.292 0.349 0.272 0.390 0.206 0.276 (6k)
261121-so101-pop-scratch-s0 PoP, no pretraining DINOv2 trunk only labelled 0 0.284 0.310 0.289 0.423 0.198 0.271 (4k)
261121-so101-sent-lab-s0 Sentence baseline pretraining labelled 0 0.292 0.335 0.288 0.406 0.205 0.277 (4k)
261121-so101-sent-lab-s1 Sentence baseline pretraining labelled 1 0.294 0.328 0.301 0.414 0.208 0.277 (6k)
261121-so101-sent-s0 Sentence baseline pretraining all 0 0.294 0.341 0.286 0.409 0.205 0.276 (4k)
261121-so101-sent-s1 Sentence baseline pretraining all 1 0.298 0.337 0.283 0.419 0.217 0.277 (4k)
  • "labelled": the frames with referent labels (287 of 293 training episodes); every model in the comparison uses them. "all": all 293 training episodes, an earlier variant of the sentence baseline.
  • Each folder finetune/<run>/ has step_020000.pt and step_040000.pt (the final checkpoint, the one to use), config.json and train.log. No checkpoint was selected on validation loss.

Roll-out on an SO-101 with LeRobot

code/scripts/rollout.py runs a fine-tuning checkpoint on an SO-101 follower arm through LeRobot 0.7.0 (its SO101Follower robot and OpenCV cameras). It is a standalone script, not a lerobot-rollout policy: that command takes LeRobot-format policies conditioned on a task string, and PoP takes points on the image.

The setup the checkpoints expect. They were trained on the four tmhoanggg recordings: one SO-101, a fixed camera (front) about 40 cm behind and 60 cm above the robot base looking over it at the table, a wrist camera (wrist), both 640 Γ— 480, and those objects. Expect them to work only in a setup that looks like those recordings. Another rig needs its own demonstrations, camera fit and fine-tuning (see "Another rig" below).

1. Install (Python 3.12, LeRobot 0.7.0):

uv venv -p 3.12 && source .venv/bin/activate
uv pip install "lerobot[feetech]==0.7.0"
# the window for clicking points needs OpenCV's GUI build (skip with --no-window):
uv pip uninstall opencv-python-headless && uv pip install "opencv-python<4.14"
hf download khanhnd61/pop-so101 --include "code/*" --local-dir pop-so101
cd pop-so101/code

After the OpenCV swap, uv pip check reports opencv-python-headless as missing; the GUI build provides the same cv2 module. Over SSH without a display, use --no-window.

2. Connect and calibrate the arm with LeRobot's own tools, once. The calibration id is the --robot-id of step 3.

lerobot-find-port                 # serial port of the follower arm
lerobot-find-cameras opencv       # indexes of the fixed and wrist cameras
lerobot-calibrate --robot.type=so101_follower --robot.port=/dev/ttyACM0 --robot.id=my_follower

Joints are read in degrees (use_degrees=True, LeRobot 0.7.0's default), as in the training data. A calibration that puts the joints' zero elsewhere than the recording arm's shifts every joint the policy sees.

3. Run a trial:

python scripts/rollout.py --task cups --run 261121-so101-pop-s0 \
    --port /dev/ttyACM0 --robot-id my_follower --front 0 --wrist 2
  • The script downloads finetune/<run>/step_040000.pt and the task's fitted camera (finetune/data/cameras/).
  • Points (PoP runs). A window shows the live fixed view; click the task's points in order (u = undo, Enter = done). In training each point was where the midpoint between the fingertips was when the gripper closed (object) or opened (destination), so click there, not at the object's centre. The view is live: you can put an object at its target, click, and move it back. The script prints the points as a --points '...' argument for repeat trials on the same layout and for --no-window. The points per task are in the table below.
  • Press space to start. Keys: n = next step, g = toggle "object in hand", q or Esc = stop. At the end, support the arm and press Enter: LeRobot switches the torque off when it disconnects.
  • Sentence baseline: --run 261121-so101-sent-lab-s0. No points; the task's training instruction is used.
  • Options: --max-seconds 60; --max-relative-target 10 (degrees: LeRobot clamps each command to this distance from the current joints); --save trial.npz (per tick: joints, targets, step, points); --device cpu. A policy call takes 0.15 s on a 6-core desktop CPU (i5-12400F) and 0.03 s on an RTX 3090 (median of 20, measured).
--task Steps (skill codes) Points, in order
towel fold where the fingertips close on the towel (near corner or edge); where they open (far corner)
cups pick-and-place, pick-and-place pink cup; just above the beige cup; beige cup; just above the white cup
drawer open, pick-and-place, close drawer handle (drawer closed); red block; inside the open drawer
push push where the centre of the mug should end up (the mug is found by its yellow colour)

What the script does. At 10 Hz, the training rate, it reads the joints and both cameras. Every 0.8 s it runs the policy, which returns 16 joint targets for the next 1.6 s, and sends the first 8, interpolated to 30 Hz. Frames are squashed to 256 Γ— 256 as in training. For PoP, the gripper tip comes from the joints through the SO-101 URDF and the fitted camera, and the active step follows the gripper by the rules that made the training labels: after the gripper closes on the object, the object point follows the tip; an opening ends a pick-and-place step; the drawer's open step ends when the arm lifts after pulling, and the close step aims at where the pull ended (code/popt/runtime.py).

Checked without a robot (code/scripts/check_runtime.py on recorded episodes; code 3099419):

  • On the 20 held-out episodes the roll-out code reproduces the training log's validation L1: PoP 0.2987 against 0.2984 logged, sentence baseline 0.2922 against 0.2918 (bf16 at batch 1 against batches of 128).
  • On the 307 labelled episodes, fed only the joints (and the fixed view for push), the step tracker agrees with the training labels on 100.0% (towel), 100.0% (push), 99.2% (cups) and 98.0% (drawer) of frames; 0 step switches missed; the median switch is +0.20 s (cups) and -0.05 s (drawer) from the labels'.
  • Full-size frames resized by the script differ from the training frames by 1.3–1.6 grey levels on average (the training frames were resized by the video decoder).
  • A replayed episode through a stand-in for the LeRobot robot: commands, step switches and 0.100 s ticks; LeRobot 0.7.0's SO101Follower configuration (degrees, RGB 640 Γ— 480) built without hardware.

None of this has run on a robot yet.

Another rig. The fitted cameras belong to the recording rig. For your own: record demonstrations with lerobot-record (cameras named front and wrist, 640 Γ— 480), then in code/scripts/ convert them (convert_v3.py), fit your camera (fit_camera.py; it needs the SO-ARM100 meshes, see code/assets/SOURCE.txt), make the referent labels (label_events.py; its rules cover these four tasks) and fine-tune from pretrain/261120-pretrain-community-s0/step_200000.pt (finetune.py). Each script's header gives its command. The training environment is Python 3.10 with PyTorch 2.5.1 (code/uv.lock).

Pretraining checkpoint

pretrain/261120-pretrain-community-s0/step_200000.pt: vision trunk and control head (27.74M parameters) trained by behaviour cloning on HuggingFaceVLA/community_dataset_v2 at revision 815dd11: 289 single-arm 6-joint datasets, 10,639 episodes, 1,774,216 frames at 10 Hz (49.28 h). 200k steps at batch 256, final training L1 0.125. Trained without language and without referents: the referent slots hold learned null vectors and the skill code is a reserved "unspecified" entry. data/report.json lists what was kept and dropped and why.

Fine-tuning data

Dataset Revision Instruction Episodes Train episodes / frames Val episodes / frames
tmhoanggg/so101-fold-towel cb8e3b4 fold the towel 94 89 / 10,226 5 / 647
tmhoanggg/so101-drawer-v3 537b4bf open the top drawer, put the red block inside, then close the drawer 50 45 / 11,821 5 / 1,315
tmhoanggg/so101-stack-cup-v4 f57265d stack the pink cup on the beige cup, then stack them on the white cup 120 114 / 27,309 5 / 1,396
tmhoanggg/so101-push bd41765 push the mug to the goal 50 45 / 5,352 5 / 606
  • Converted to 10 Hz and 256 Γ— 256 (aspect ratio not kept); cameras front (fixed, behind the robot base) and wrist. One normalization pooled over all four datasets (state, and actions relative to the current joints), so the normalization does not tell the policy which task it is in.
  • One joint policy over the four tasks per run.

Referent labels, made from the demonstrations without a detector or hand marks (finetune/data/):

  • Camera: one fit per dataset from its own frames: the projected surface of the SO-101 URDF (TheRobotStudio/SO-ARM100, so101_new_calib.urdf) driven by the recorded joints is matched to the orange pixels of the arm (camera pose, focal length, radial distortion, a constant offset per joint). Residual about 3 px at 256 px. cameras/*.json.
  • Points: the gripper's 3D tip from forward kinematics. Object = where the gripper closes, destination = where it opens; drawer handle = the reach before the pull; push: the mug found by colour on the table plane, destination = where it ends. Labelled training episodes: drawer-v3 39/45, fold-towel 89/89, push 45/45, stack-cup-v4 114/114.
  • Steps and skill codes: towel fold; cups pick-and-place twice; drawer open, pick-and-place, close; push.

Model

Part
Trunk DINOv2 ViT-S/14, fine-tuned, fixed and wrist view at 224 px (22.06M)
Head adaLN-zero transformer decoder, width 224, 4 layers, 7 heads (5.68M); 16 joint targets relative to the current joints, at 10 Hz
PoP conditioning Per role (object, destination): Fourier features of the point's position in the fixed view and of its offset from the projected gripper tip, an MLP and a role vector; missing destination = null vector. Skill code: 5 rows of the skill embedding. About 28M parameters in all
Sentence conditioning T5-Efficient-Tiny (11.4M, fine-tuned), mean-pooled and projected into the skill slot; referent slots at their null vectors. 39.2M parameters in all

Recipe for every fine-tuning run: 40k steps at batch 64, AdamW (0.9, 0.95), weight decay 0.05, learning rate 3e-4 (3e-5 for the trunk and the text encoder), 2k warmup then cosine to 10%, EMA 0.9995, bf16, random crop (scale 0.8–1) on the fixed view with the points moved accordingly, 1 cm of 3D noise on the referent points, L1 loss.

Code and loading

code/ is the training and roll-out package at 3099419: popt/model.py (the network: PopFinetune, pretraining PopPretrain), popt/runtime.py (Policy: a checkpoint ready to act; Camera; StepTracker), popt/so101.py (forward kinematics), popt/t5.py, scripts/ and the vendored DINOv2 code (third_party/dinov2, Apache-2.0).

import sys; sys.path.insert(0, "code")
from popt.runtime import Policy
policy = Policy("finetune/261121-so101-pop-s0/step_040000.pt", "cuda")
targets = policy.act(front_rgb, wrist_rgb, joints, policy.referent_cond("pick_place", what_xy, where_xy, tip_xy))
# (16, 6) absolute joint targets, one per 0.1 s; points in [-1, 1] image coordinates (popt.runtime.pixel_to_xy)

A checkpoint is a pickle (torch.load(..., weights_only=False)): ema (weights), norm (pooled state and action normalization), instructions (tokenized, sentence runs), config (training arguments).

Limitations

  • Validation loss only. Robot roll-outs (positions outside the training range, swapped objects) have not been run; the roll-out code is checked offline (above).
  • The validation loss is lowest at 4k–6k steps and 0.013–0.021 higher at 40k: with about 290 episodes the 40k-step budget overfits the held-out demonstrations. All runs use the same budget.
  • Labels are heuristics on the gripper signal: towel grasp points sit 1–2 cm outside the cloth (fingertip midpoint); the drawer's "close" point is where the drawer stood open, not the moving drawer front.
  • The skill codes fold, push, open and close each belong to one task, so on those tasks PoP's skill code identifies the task as the instruction does; pick-and-place is shared between the drawer and the cups.

License

Apache-2.0, as DINOv2, T5-Efficient-Tiny, HuggingFaceVLA/community_dataset_v2 and the four tmhoanggg datasets.

Downloads last month

-

Downloads are not tracked for this model. How to track
Video Preview
loading

Model tree for khanhnd61/pop-so101

Finetuned
(43)
this model

Datasets used to train khanhnd61/pop-so101