Instructions to use khanhnd61/pop-so101 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- LeRobot
How to use khanhnd61/pop-so101 with LeRobot:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
PoP on the SO-101: pretraining and fine-tuning checkpoints
PoP (Pointer Policy) is told what to act on through referents: a 2D point on the object, a 2D point on the destination and a small skill code, never a sentence. This repository holds its pretraining checkpoint (SO-100 community data, no conditioning) and the checkpoints of a real-robot comparison on four SO-101 tasks: PoP against a sentence-conditioned policy with the same network, initialization, data and steps.
Fine-tuning checkpoints and validation loss
L1 error of the EMA weights on normalized 16-step action chunks of 20 held-out demonstrations (5 per task). This is an imitation error, not a task success rate: no checkpoint has been run on the robot yet.
| Run | Model | Initialization | Frames | Seed | Val L1, step 40k | drawer | towel | push | cups | Lowest val L1 (step) |
|---|---|---|---|---|---|---|---|---|---|---|
261121-so101-pop-s0 |
PoP | pretraining | labelled | 0 | 0.298 | 0.354 | 0.281 | 0.414 | 0.205 | 0.278 (4k) |
261121-so101-pop-s1 |
PoP | pretraining | labelled | 1 | 0.292 | 0.349 | 0.272 | 0.390 | 0.206 | 0.276 (6k) |
261121-so101-pop-scratch-s0 |
PoP, no pretraining | DINOv2 trunk only | labelled | 0 | 0.284 | 0.310 | 0.289 | 0.423 | 0.198 | 0.271 (4k) |
261121-so101-sent-lab-s0 |
Sentence baseline | pretraining | labelled | 0 | 0.292 | 0.335 | 0.288 | 0.406 | 0.205 | 0.277 (4k) |
261121-so101-sent-lab-s1 |
Sentence baseline | pretraining | labelled | 1 | 0.294 | 0.328 | 0.301 | 0.414 | 0.208 | 0.277 (6k) |
261121-so101-sent-s0 |
Sentence baseline | pretraining | all | 0 | 0.294 | 0.341 | 0.286 | 0.409 | 0.205 | 0.276 (4k) |
261121-so101-sent-s1 |
Sentence baseline | pretraining | all | 1 | 0.298 | 0.337 | 0.283 | 0.419 | 0.217 | 0.277 (4k) |
- "labelled": the frames with referent labels (287 of 293 training episodes); every model in the comparison uses them. "all": all 293 training episodes, an earlier variant of the sentence baseline.
- Each folder
finetune/<run>/hasstep_020000.ptandstep_040000.pt(the final checkpoint, the one to use),config.jsonandtrain.log. No checkpoint was selected on validation loss.
Roll-out on an SO-101 with LeRobot
code/scripts/rollout.py runs a fine-tuning checkpoint on an SO-101 follower arm through LeRobot 0.7.0 (its
SO101Follower robot and OpenCV cameras). It is a standalone script, not a lerobot-rollout policy: that command
takes LeRobot-format policies conditioned on a task string, and PoP takes points on the image.
The setup the checkpoints expect. They were trained on the four tmhoanggg recordings: one SO-101, a fixed camera
(front) about 40 cm behind and 60 cm above the robot base looking over it at the table, a wrist camera (wrist), both
640 Γ 480, and those objects. Expect them to work only in a setup that looks like those recordings. Another rig needs
its own demonstrations, camera fit and fine-tuning (see "Another rig" below).
1. Install (Python 3.12, LeRobot 0.7.0):
uv venv -p 3.12 && source .venv/bin/activate
uv pip install "lerobot[feetech]==0.7.0"
# the window for clicking points needs OpenCV's GUI build (skip with --no-window):
uv pip uninstall opencv-python-headless && uv pip install "opencv-python<4.14"
hf download khanhnd61/pop-so101 --include "code/*" --local-dir pop-so101
cd pop-so101/code
After the OpenCV swap, uv pip check reports opencv-python-headless as missing; the GUI build provides the same
cv2 module. Over SSH without a display, use --no-window.
2. Connect and calibrate the arm with LeRobot's own tools, once. The calibration id is the --robot-id of step 3.
lerobot-find-port # serial port of the follower arm
lerobot-find-cameras opencv # indexes of the fixed and wrist cameras
lerobot-calibrate --robot.type=so101_follower --robot.port=/dev/ttyACM0 --robot.id=my_follower
Joints are read in degrees (use_degrees=True, LeRobot 0.7.0's default), as in the training data. A calibration that
puts the joints' zero elsewhere than the recording arm's shifts every joint the policy sees.
3. Run a trial:
python scripts/rollout.py --task cups --run 261121-so101-pop-s0 \
--port /dev/ttyACM0 --robot-id my_follower --front 0 --wrist 2
- The script downloads
finetune/<run>/step_040000.ptand the task's fitted camera (finetune/data/cameras/). - Points (PoP runs). A window shows the live fixed view; click the task's points in order (u = undo, Enter = done).
In training each point was where the midpoint between the fingertips was when the gripper closed (object) or
opened (destination), so click there, not at the object's centre. The view is live: you can put an object at its
target, click, and move it back. The script prints the points as a
--points '...'argument for repeat trials on the same layout and for--no-window. The points per task are in the table below. - Press space to start. Keys: n = next step, g = toggle "object in hand", q or Esc = stop. At the end, support the arm and press Enter: LeRobot switches the torque off when it disconnects.
- Sentence baseline:
--run 261121-so101-sent-lab-s0. No points; the task's training instruction is used. - Options:
--max-seconds 60;--max-relative-target 10(degrees: LeRobot clamps each command to this distance from the current joints);--save trial.npz(per tick: joints, targets, step, points);--device cpu. A policy call takes 0.15 s on a 6-core desktop CPU (i5-12400F) and 0.03 s on an RTX 3090 (median of 20, measured).
--task |
Steps (skill codes) | Points, in order |
|---|---|---|
towel |
fold | where the fingertips close on the towel (near corner or edge); where they open (far corner) |
cups |
pick-and-place, pick-and-place | pink cup; just above the beige cup; beige cup; just above the white cup |
drawer |
open, pick-and-place, close | drawer handle (drawer closed); red block; inside the open drawer |
push |
push | where the centre of the mug should end up (the mug is found by its yellow colour) |
What the script does. At 10 Hz, the training rate, it reads the joints and both cameras. Every 0.8 s it runs the
policy, which returns 16 joint targets for the next 1.6 s, and sends the first 8, interpolated to 30 Hz. Frames are
squashed to 256 Γ 256 as in training. For PoP, the gripper tip comes from the joints through the SO-101 URDF and the
fitted camera, and the active step follows the gripper by the rules that made the training labels: after the gripper
closes on the object, the object point follows the tip; an opening ends a pick-and-place step; the drawer's open step
ends when the arm lifts after pulling, and the close step aims at where the pull ended (code/popt/runtime.py).
Checked without a robot (code/scripts/check_runtime.py on recorded episodes; code 3099419):
- On the 20 held-out episodes the roll-out code reproduces the training log's validation L1: PoP 0.2987 against 0.2984 logged, sentence baseline 0.2922 against 0.2918 (bf16 at batch 1 against batches of 128).
- On the 307 labelled episodes, fed only the joints (and the fixed view for push), the step tracker agrees with the training labels on 100.0% (towel), 100.0% (push), 99.2% (cups) and 98.0% (drawer) of frames; 0 step switches missed; the median switch is +0.20 s (cups) and -0.05 s (drawer) from the labels'.
- Full-size frames resized by the script differ from the training frames by 1.3β1.6 grey levels on average (the training frames were resized by the video decoder).
- A replayed episode through a stand-in for the LeRobot robot: commands, step switches and 0.100 s ticks; LeRobot
0.7.0's
SO101Followerconfiguration (degrees, RGB 640 Γ 480) built without hardware.
None of this has run on a robot yet.
Another rig. The fitted cameras belong to the recording rig. For your own: record demonstrations with
lerobot-record (cameras named front and wrist, 640 Γ 480), then in code/scripts/ convert them
(convert_v3.py), fit your camera (fit_camera.py; it needs the SO-ARM100 meshes, see code/assets/SOURCE.txt),
make the referent labels (label_events.py; its rules cover these four tasks) and fine-tune from
pretrain/261120-pretrain-community-s0/step_200000.pt (finetune.py). Each script's header gives its command. The
training environment is Python 3.10 with PyTorch 2.5.1 (code/uv.lock).
Pretraining checkpoint
pretrain/261120-pretrain-community-s0/step_200000.pt: vision trunk and control head (27.74M parameters) trained by behaviour cloning
on HuggingFaceVLA/community_dataset_v2 at revision 815dd11: 289 single-arm 6-joint datasets,
10,639 episodes, 1,774,216 frames at 10 Hz (49.28 h). 200k steps at batch 256, final
training L1 0.125. Trained without language and without referents: the referent slots hold learned null vectors and the
skill code is a reserved "unspecified" entry. data/report.json lists what was kept and dropped and why.
Fine-tuning data
| Dataset | Revision | Instruction | Episodes | Train episodes / frames | Val episodes / frames |
|---|---|---|---|---|---|
tmhoanggg/so101-fold-towel |
cb8e3b4 |
fold the towel | 94 | 89 / 10,226 | 5 / 647 |
tmhoanggg/so101-drawer-v3 |
537b4bf |
open the top drawer, put the red block inside, then close the drawer | 50 | 45 / 11,821 | 5 / 1,315 |
tmhoanggg/so101-stack-cup-v4 |
f57265d |
stack the pink cup on the beige cup, then stack them on the white cup | 120 | 114 / 27,309 | 5 / 1,396 |
tmhoanggg/so101-push |
bd41765 |
push the mug to the goal | 50 | 45 / 5,352 | 5 / 606 |
- Converted to 10 Hz and 256 Γ 256 (aspect ratio not kept); cameras
front(fixed, behind the robot base) andwrist. One normalization pooled over all four datasets (state, and actions relative to the current joints), so the normalization does not tell the policy which task it is in. - One joint policy over the four tasks per run.
Referent labels, made from the demonstrations without a detector or hand marks (finetune/data/):
- Camera: one fit per dataset from its own frames: the projected surface of the SO-101 URDF (TheRobotStudio/SO-ARM100,
so101_new_calib.urdf) driven by the recorded joints is matched to the orange pixels of the arm (camera pose, focal length, radial distortion, a constant offset per joint). Residual about 3 px at 256 px.cameras/*.json. - Points: the gripper's 3D tip from forward kinematics. Object = where the gripper closes, destination = where it opens; drawer handle = the reach before the pull; push: the mug found by colour on the table plane, destination = where it ends. Labelled training episodes: drawer-v3 39/45, fold-towel 89/89, push 45/45, stack-cup-v4 114/114.
- Steps and skill codes: towel fold; cups pick-and-place twice; drawer open, pick-and-place, close; push.
Model
| Part | |
|---|---|
| Trunk | DINOv2 ViT-S/14, fine-tuned, fixed and wrist view at 224 px (22.06M) |
| Head | adaLN-zero transformer decoder, width 224, 4 layers, 7 heads (5.68M); 16 joint targets relative to the current joints, at 10 Hz |
| PoP conditioning | Per role (object, destination): Fourier features of the point's position in the fixed view and of its offset from the projected gripper tip, an MLP and a role vector; missing destination = null vector. Skill code: 5 rows of the skill embedding. About 28M parameters in all |
| Sentence conditioning | T5-Efficient-Tiny (11.4M, fine-tuned), mean-pooled and projected into the skill slot; referent slots at their null vectors. 39.2M parameters in all |
Recipe for every fine-tuning run: 40k steps at batch 64, AdamW (0.9, 0.95), weight decay 0.05, learning rate 3e-4 (3e-5 for the trunk and the text encoder), 2k warmup then cosine to 10%, EMA 0.9995, bf16, random crop (scale 0.8β1) on the fixed view with the points moved accordingly, 1 cm of 3D noise on the referent points, L1 loss.
Code and loading
code/ is the training and roll-out package at 3099419: popt/model.py (the network: PopFinetune,
pretraining PopPretrain), popt/runtime.py (Policy: a checkpoint ready to act; Camera; StepTracker),
popt/so101.py (forward kinematics), popt/t5.py, scripts/ and the vendored DINOv2 code (third_party/dinov2,
Apache-2.0).
import sys; sys.path.insert(0, "code")
from popt.runtime import Policy
policy = Policy("finetune/261121-so101-pop-s0/step_040000.pt", "cuda")
targets = policy.act(front_rgb, wrist_rgb, joints, policy.referent_cond("pick_place", what_xy, where_xy, tip_xy))
# (16, 6) absolute joint targets, one per 0.1 s; points in [-1, 1] image coordinates (popt.runtime.pixel_to_xy)
A checkpoint is a pickle (torch.load(..., weights_only=False)): ema (weights), norm (pooled state and action
normalization), instructions (tokenized, sentence runs), config (training arguments).
Limitations
- Validation loss only. Robot roll-outs (positions outside the training range, swapped objects) have not been run; the roll-out code is checked offline (above).
- The validation loss is lowest at 4kβ6k steps and 0.013β0.021 higher at 40k: with about 290 episodes the 40k-step budget overfits the held-out demonstrations. All runs use the same budget.
- Labels are heuristics on the gripper signal: towel grasp points sit 1β2 cm outside the cloth (fingertip midpoint); the drawer's "close" point is where the drawer stood open, not the moving drawer front.
- The skill codes fold, push, open and close each belong to one task, so on those tasks PoP's skill code identifies the task as the instruction does; pick-and-place is shared between the drawer and the cups.
License
Apache-2.0, as DINOv2, T5-Efficient-Tiny, HuggingFaceVLA/community_dataset_v2 and the four tmhoanggg datasets.
Model tree for khanhnd61/pop-so101
Base model
facebook/dinov2-small