Nadeem-Shoukath115113's picture
|
download
raw
7.07 kB

Trained models — where they are and how to use them

Two models were trained on this machine. Both are copied into checkpoints/, which is stable; the outputs/<date>/<time>/ originals are Hydra run directories and are easy to lose track of.

checkpoints/
├── act_pri/                       stage 1 — action-primitive classifier
│   ├── act_pri_best.pt            epoch 1, val CE 0.709   <- best held-out
│   ├── act_pri_final.pt           epoch 4                 <- used for labelling
│   └── state_norm.json            the norm stats it was trained with
└── sarm2/                         stage 2 — MMoE value head
    ├── reward_best.pt             best val MSE 0.0585 on steps_to_go
    ├── act_pri_best.pt            the stage-1 model after joint finetuning
    └── state_norm.json

Originals: outputs/2026-08-21/12-22-57/ (act_pri) and outputs/2026-08-21/14-06-07/ (sarm2).

Always pair a checkpoint with its own state_norm.json. The state normalizer is part of the model; using the wrong one silently degrades predictions.


What each model does

input output
act_pri 6 frames × 3 cameras spaced frame_gap=30, + state one of pick_grab / fold_flatten / dummy for the last frame
sarm2 [episode start] + 6 frames × 3 cameras, + state, + task text steps_to_go ∈ [0,1]; progress = 1 − steps_to_go

Both use a frozen google/siglip2-base-patch16-224 backbone. frame_gap must match between them — stage 2 feeds act_pri a slice of its own sequence (sarm2_ws._slice_act_pri_inputs), so a mismatch is a silent train/inference bug.


Label a dataset with action primitives

Writes one primitive per frame, plus the JPEGs and JSON the review UI reads.

cd /home/ubuntu/Workspace/Thesis_Code/opensarm
source .venv/bin/activate

python tools/predict_act_pri.py \
    --dataset-root /path/to/lerobot_dataset \
    --out-dir act_pri_preds --session my_session \
    --checkpoint checkpoints/act_pri/act_pri_final.pt --name act_pri \
    --state-norm checkpoints/act_pri/state_norm.json \
    --frame-gap 30

Shortcut for the three cloth_folder sessions (auto-detects the newest checkpoint and reads frame_gap / n_obs_steps / state_norm from config/act_pri.yaml):

bash tools/run_act_pri_preds.sh                  # all sessions
bash tools/run_act_pri_preds.sh 20260724_171909  # one

Useful flags: --group-seconds (default 1.0 — one label per second of video), --episodes 0-9, --no-images (skip the review JPEGs), --batch-size.

Review and correct the labels

python tools/annotate_act_pri_ui.py --labels-dir act_pri_preds --dataset-root data --port 8000

http://127.0.0.1:8000 — session dropdown, 3 camera views, the model's label and its confidence. ←/→ step, 1/2/3 set pick_grab/fold_flatten/dummy, r resets to the prediction. Every edit writes straight to groups/*.json, per_frame/*.npy and act_pri.parquet — no save button. Corrected groups are outlined in red; modified → jumps between them.


Score a dataset with SARM2 (progress)

Static plot — several episodes at once

python tools/visualise_sarm2.py \
    --dataset-root /path/to/lerobot_dataset \
    --split all --episodes 5 --out progress.png

Teal = predicted progress, grey dashed = the linear t/(T-1) reference, amber = is_policy stretches if the dataset carries that column (DAgger). Also prints mean |pred − gt| and the final progress per episode.

--episode-ids 10,16 picks specific ones, --stride trades speed for resolution (default 5 = one prediction per 5 frames), --split val restricts to held-out episodes.

Side-by-side video

python tools/make_sarm2_video.py \
    --dataset-root /path/to/lerobot_dataset \
    --episodes 3 --out sarm2_videos

Runs on every frame so predictions line up 1:1 with the video, then calls the repo's own utils.make_demo_video.produce_video. Writes per episode:

sarm2_videos/episode_XXXXXX/
├── combined_video.mp4    camera view + progress plot with a moving cursor
├── camera.mp4            the extracted camera view
├── pred.npy              progress per frame
└── gt.npy                linear reference

--camera observation.images.left_left_wrist switches the view; --episode-ids 10,16 picks episodes.

Note the plot is labelled "Reward Model Prediction" — that text comes from draw_plot_frame in the repo; the values plotted are progress.


Build a training dataset from labels

Stage 1 (whole episodes, per-frame act_pri as the CE target):

python tools/build_act_pri_clips.py --whole-episodes \
    --source /path/to/session_a act_pri_labels \
    --source /path/to/session_b act_pri_preds/session_b \
    --repo-id <namespace>/<name> --overwrite

Add --reviewed-only to keep just the groups a human confirmed in the UI.

Stage 2 (whole-task reward + act_pri, both columns generated):

python tools/build_sarm2_dataset.py \
    --source /path/to/source_dataset \
    --repo-id <namespace>/<name> --overwrite \
    --reward linear \
    --act-pri-checkpoint checkpoints/act_pri/act_pri_final.pt \
    --state-norm checkpoints/act_pri/state_norm.json --frame-gap 30

--reward linear writes reward = t/(T-1), the progress SARM2 converts internally to steps_to_go = (T-t)/(T-1). Existing reward / progress columns and any sarm_progress.parquet are dropped first. Videos are hardlinked, so the copy is free.

Verify before training:

python tools/check_act_pri_dataset.py --repo-id <namespace>/<name>

Retrain

python train.py --config-name act_pri    # stage 1
python train.py --config-name sarm2      # stage 2

Datasets are resolved by repo_id under ~/.cache/huggingface/lerobot/, which got cleared twice on this machine — if training reports a missing dataset, rebuild it with the commands above. Checkpoints and the label stores in this repo were never affected.

If a dataset trains slowly, check the video GOP: random frame access costs GOP/2 decodes per frame. python tools/reencode_allintra.py --dataset-root <dir> re-encodes every frame as a keyframe (100 s for 486 videos, roughly doubles the size).


Caveats worth carrying forward

  • act_pri's ground truth is only ~29% human-verified. Of 1,456 one-second groups, 428 were reviewed by hand; most of the rest are an earlier act_pri model's own predictions, so the model is partly trained on its own output. --reviewed-only builds a smaller, fully human-labelled set.
  • act_pri_best.pt is epoch 1 in both runs — validation loss rose after that. act_pri_final.pt (epoch 4) is what produced the stage-2 labels.
  • The grey line in the plots is not truth, it's the linear training target. On rollouts with interventions and retries, real progress is not linear, so deviation from it is expected rather than error.

Xet Storage Details

Size:
7.07 kB
·
Xet hash:
2c66293d6bb10bbf191951b98317043ebff017e7adb72fe52ff6212e72b1b095

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.