Buckets:
Trained models — where they are and how to use them
Two models were trained on this machine. Both are copied into checkpoints/, which
is stable; the outputs/<date>/<time>/ originals are Hydra run directories and are
easy to lose track of.
checkpoints/
├── act_pri/ stage 1 — action-primitive classifier
│ ├── act_pri_best.pt epoch 1, val CE 0.709 <- best held-out
│ ├── act_pri_final.pt epoch 4 <- used for labelling
│ └── state_norm.json the norm stats it was trained with
└── sarm2/ stage 2 — MMoE value head
├── reward_best.pt best val MSE 0.0585 on steps_to_go
├── act_pri_best.pt the stage-1 model after joint finetuning
└── state_norm.json
Originals: outputs/2026-08-21/12-22-57/ (act_pri) and outputs/2026-08-21/14-06-07/ (sarm2).
Always pair a checkpoint with its own state_norm.json. The state normalizer is
part of the model; using the wrong one silently degrades predictions.
What each model does
| input | output | |
|---|---|---|
| act_pri | 6 frames × 3 cameras spaced frame_gap=30, + state |
one of pick_grab / fold_flatten / dummy for the last frame |
| sarm2 | [episode start] + 6 frames × 3 cameras, + state, + task text |
steps_to_go ∈ [0,1]; progress = 1 − steps_to_go |
Both use a frozen google/siglip2-base-patch16-224 backbone. frame_gap must match
between them — stage 2 feeds act_pri a slice of its own sequence
(sarm2_ws._slice_act_pri_inputs), so a mismatch is a silent train/inference bug.
Label a dataset with action primitives
Writes one primitive per frame, plus the JPEGs and JSON the review UI reads.
cd /home/ubuntu/Workspace/Thesis_Code/opensarm
source .venv/bin/activate
python tools/predict_act_pri.py \
--dataset-root /path/to/lerobot_dataset \
--out-dir act_pri_preds --session my_session \
--checkpoint checkpoints/act_pri/act_pri_final.pt --name act_pri \
--state-norm checkpoints/act_pri/state_norm.json \
--frame-gap 30
Shortcut for the three cloth_folder sessions (auto-detects the newest checkpoint and
reads frame_gap / n_obs_steps / state_norm from config/act_pri.yaml):
bash tools/run_act_pri_preds.sh # all sessions
bash tools/run_act_pri_preds.sh 20260724_171909 # one
Useful flags: --group-seconds (default 1.0 — one label per second of video),
--episodes 0-9, --no-images (skip the review JPEGs), --batch-size.
Review and correct the labels
python tools/annotate_act_pri_ui.py --labels-dir act_pri_preds --dataset-root data --port 8000
http://127.0.0.1:8000 — session dropdown, 3 camera views, the model's label and its
confidence. ←/→ step, 1/2/3 set pick_grab/fold_flatten/dummy, r
resets to the prediction. Every edit writes straight to groups/*.json,
per_frame/*.npy and act_pri.parquet — no save button. Corrected groups are
outlined in red; modified → jumps between them.
Score a dataset with SARM2 (progress)
Static plot — several episodes at once
python tools/visualise_sarm2.py \
--dataset-root /path/to/lerobot_dataset \
--split all --episodes 5 --out progress.png
Teal = predicted progress, grey dashed = the linear t/(T-1) reference, amber =
is_policy stretches if the dataset carries that column (DAgger). Also prints
mean |pred − gt| and the final progress per episode.
--episode-ids 10,16 picks specific ones, --stride trades speed for resolution
(default 5 = one prediction per 5 frames), --split val restricts to held-out episodes.
Side-by-side video
python tools/make_sarm2_video.py \
--dataset-root /path/to/lerobot_dataset \
--episodes 3 --out sarm2_videos
Runs on every frame so predictions line up 1:1 with the video, then calls the
repo's own utils.make_demo_video.produce_video. Writes per episode:
sarm2_videos/episode_XXXXXX/
├── combined_video.mp4 camera view + progress plot with a moving cursor
├── camera.mp4 the extracted camera view
├── pred.npy progress per frame
└── gt.npy linear reference
--camera observation.images.left_left_wrist switches the view;
--episode-ids 10,16 picks episodes.
Note the plot is labelled "Reward Model Prediction" — that text comes from
draw_plot_frame in the repo; the values plotted are progress.
Build a training dataset from labels
Stage 1 (whole episodes, per-frame act_pri as the CE target):
python tools/build_act_pri_clips.py --whole-episodes \
--source /path/to/session_a act_pri_labels \
--source /path/to/session_b act_pri_preds/session_b \
--repo-id <namespace>/<name> --overwrite
Add --reviewed-only to keep just the groups a human confirmed in the UI.
Stage 2 (whole-task reward + act_pri, both columns generated):
python tools/build_sarm2_dataset.py \
--source /path/to/source_dataset \
--repo-id <namespace>/<name> --overwrite \
--reward linear \
--act-pri-checkpoint checkpoints/act_pri/act_pri_final.pt \
--state-norm checkpoints/act_pri/state_norm.json --frame-gap 30
--reward linear writes reward = t/(T-1), the progress SARM2 converts internally to
steps_to_go = (T-t)/(T-1). Existing reward / progress columns and any
sarm_progress.parquet are dropped first. Videos are hardlinked, so the copy is free.
Verify before training:
python tools/check_act_pri_dataset.py --repo-id <namespace>/<name>
Retrain
python train.py --config-name act_pri # stage 1
python train.py --config-name sarm2 # stage 2
Datasets are resolved by repo_id under ~/.cache/huggingface/lerobot/, which got
cleared twice on this machine — if training reports a missing dataset, rebuild it with
the commands above. Checkpoints and the label stores in this repo were never affected.
If a dataset trains slowly, check the video GOP: random frame access costs GOP/2
decodes per frame. 100 s for 486 videos, roughly doubles the size).python tools/reencode_allintra.py --dataset-root <dir> re-encodes
every frame as a keyframe (
Caveats worth carrying forward
- act_pri's ground truth is only ~29% human-verified. Of 1,456 one-second groups,
428 were reviewed by hand; most of the rest are an earlier act_pri model's own
predictions, so the model is partly trained on its own output.
--reviewed-onlybuilds a smaller, fully human-labelled set. act_pri_best.ptis epoch 1 in both runs — validation loss rose after that.act_pri_final.pt(epoch 4) is what produced the stage-2 labels.- The grey line in the plots is not truth, it's the linear training target. On rollouts with interventions and retries, real progress is not linear, so deviation from it is expected rather than error.
Xet Storage Details
- Size:
- 7.07 kB
- Xet hash:
- 2c66293d6bb10bbf191951b98317043ebff017e7adb72fe52ff6212e72b1b095
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.