Buckets:
| # Trained models — where they are and how to use them | |
| Two models were trained on this machine. Both are copied into `checkpoints/`, which | |
| is stable; the `outputs/<date>/<time>/` originals are Hydra run directories and are | |
| easy to lose track of. | |
| ``` | |
| checkpoints/ | |
| ├── act_pri/ stage 1 — action-primitive classifier | |
| │ ├── act_pri_best.pt epoch 1, val CE 0.709 <- best held-out | |
| │ ├── act_pri_final.pt epoch 4 <- used for labelling | |
| │ └── state_norm.json the norm stats it was trained with | |
| └── sarm2/ stage 2 — MMoE value head | |
| ├── reward_best.pt best val MSE 0.0585 on steps_to_go | |
| ├── act_pri_best.pt the stage-1 model after joint finetuning | |
| └── state_norm.json | |
| ``` | |
| Originals: `outputs/2026-08-21/12-22-57/` (act_pri) and `outputs/2026-08-21/14-06-07/` (sarm2). | |
| **Always pair a checkpoint with its own `state_norm.json`.** The state normalizer is | |
| part of the model; using the wrong one silently degrades predictions. | |
| --- | |
| ## What each model does | |
| | | input | output | | |
| |---|---|---| | |
| | **act_pri** | 6 frames × 3 cameras spaced `frame_gap=30`, + state | one of `pick_grab` / `fold_flatten` / `dummy` for the **last** frame | | |
| | **sarm2** | `[episode start] + 6 frames` × 3 cameras, + state, + task text | `steps_to_go ∈ [0,1]`; progress = `1 − steps_to_go` | | |
| Both use a frozen `google/siglip2-base-patch16-224` backbone. `frame_gap` must match | |
| between them — stage 2 feeds act_pri a slice of its own sequence | |
| (`sarm2_ws._slice_act_pri_inputs`), so a mismatch is a silent train/inference bug. | |
| --- | |
| ## Label a dataset with action primitives | |
| Writes one primitive per frame, plus the JPEGs and JSON the review UI reads. | |
| ```bash | |
| cd /home/ubuntu/Workspace/Thesis_Code/opensarm | |
| source .venv/bin/activate | |
| python tools/predict_act_pri.py \ | |
| --dataset-root /path/to/lerobot_dataset \ | |
| --out-dir act_pri_preds --session my_session \ | |
| --checkpoint checkpoints/act_pri/act_pri_final.pt --name act_pri \ | |
| --state-norm checkpoints/act_pri/state_norm.json \ | |
| --frame-gap 30 | |
| ``` | |
| Shortcut for the three cloth_folder sessions (auto-detects the newest checkpoint and | |
| reads `frame_gap` / `n_obs_steps` / `state_norm` from `config/act_pri.yaml`): | |
| ```bash | |
| bash tools/run_act_pri_preds.sh # all sessions | |
| bash tools/run_act_pri_preds.sh 20260724_171909 # one | |
| ``` | |
| Useful flags: `--group-seconds` (default 1.0 — one label per second of video), | |
| `--episodes 0-9`, `--no-images` (skip the review JPEGs), `--batch-size`. | |
| ### Review and correct the labels | |
| ```bash | |
| python tools/annotate_act_pri_ui.py --labels-dir act_pri_preds --dataset-root data --port 8000 | |
| ``` | |
| <http://127.0.0.1:8000> — session dropdown, 3 camera views, the model's label and its | |
| confidence. `←`/`→` step, `1`/`2`/`3` set `pick_grab`/`fold_flatten`/`dummy`, `r` | |
| resets to the prediction. Every edit writes straight to `groups/*.json`, | |
| `per_frame/*.npy` and `act_pri.parquet` — no save button. Corrected groups are | |
| outlined in red; `modified →` jumps between them. | |
| --- | |
| ## Score a dataset with SARM2 (progress) | |
| ### Static plot — several episodes at once | |
| ```bash | |
| python tools/visualise_sarm2.py \ | |
| --dataset-root /path/to/lerobot_dataset \ | |
| --split all --episodes 5 --out progress.png | |
| ``` | |
| Teal = predicted progress, grey dashed = the linear `t/(T-1)` reference, amber = | |
| `is_policy` stretches if the dataset carries that column (DAgger). Also prints | |
| `mean |pred − gt|` and the final progress per episode. | |
| `--episode-ids 10,16` picks specific ones, `--stride` trades speed for resolution | |
| (default 5 = one prediction per 5 frames), `--split val` restricts to held-out episodes. | |
| ### Side-by-side video | |
| ```bash | |
| python tools/make_sarm2_video.py \ | |
| --dataset-root /path/to/lerobot_dataset \ | |
| --episodes 3 --out sarm2_videos | |
| ``` | |
| Runs on **every** frame so predictions line up 1:1 with the video, then calls the | |
| repo's own `utils.make_demo_video.produce_video`. Writes per episode: | |
| ``` | |
| sarm2_videos/episode_XXXXXX/ | |
| ├── combined_video.mp4 camera view + progress plot with a moving cursor | |
| ├── camera.mp4 the extracted camera view | |
| ├── pred.npy progress per frame | |
| └── gt.npy linear reference | |
| ``` | |
| `--camera observation.images.left_left_wrist` switches the view; | |
| `--episode-ids 10,16` picks episodes. | |
| Note the plot is labelled "Reward Model Prediction" — that text comes from | |
| `draw_plot_frame` in the repo; the values plotted are progress. | |
| --- | |
| ## Build a training dataset from labels | |
| Stage 1 (whole episodes, per-frame `act_pri` as the CE target): | |
| ```bash | |
| python tools/build_act_pri_clips.py --whole-episodes \ | |
| --source /path/to/session_a act_pri_labels \ | |
| --source /path/to/session_b act_pri_preds/session_b \ | |
| --repo-id <namespace>/<name> --overwrite | |
| ``` | |
| Add `--reviewed-only` to keep just the groups a human confirmed in the UI. | |
| Stage 2 (whole-task reward + act_pri, both columns generated): | |
| ```bash | |
| python tools/build_sarm2_dataset.py \ | |
| --source /path/to/source_dataset \ | |
| --repo-id <namespace>/<name> --overwrite \ | |
| --reward linear \ | |
| --act-pri-checkpoint checkpoints/act_pri/act_pri_final.pt \ | |
| --state-norm checkpoints/act_pri/state_norm.json --frame-gap 30 | |
| ``` | |
| `--reward linear` writes `reward = t/(T-1)`, the progress SARM2 converts internally to | |
| `steps_to_go = (T-t)/(T-1)`. Existing `reward` / `progress` columns and any | |
| `sarm_progress.parquet` are dropped first. Videos are hardlinked, so the copy is free. | |
| Verify before training: | |
| ```bash | |
| python tools/check_act_pri_dataset.py --repo-id <namespace>/<name> | |
| ``` | |
| --- | |
| ## Retrain | |
| ```bash | |
| python train.py --config-name act_pri # stage 1 | |
| python train.py --config-name sarm2 # stage 2 | |
| ``` | |
| Datasets are resolved by `repo_id` under `~/.cache/huggingface/lerobot/`, which got | |
| cleared twice on this machine — if training reports a missing dataset, rebuild it with | |
| the commands above. Checkpoints and the label stores in this repo were never affected. | |
| If a dataset trains slowly, check the video GOP: random frame access costs ~GOP/2 | |
| decodes per frame. `python tools/reencode_allintra.py --dataset-root <dir>` re-encodes | |
| every frame as a keyframe (~100 s for 486 videos, roughly doubles the size). | |
| --- | |
| ## Caveats worth carrying forward | |
| - **act_pri's ground truth is only ~29% human-verified.** Of 1,456 one-second groups, | |
| 428 were reviewed by hand; most of the rest are an earlier act_pri model's own | |
| predictions, so the model is partly trained on its own output. `--reviewed-only` | |
| builds a smaller, fully human-labelled set. | |
| - **`act_pri_best.pt` is epoch 1** in both runs — validation loss rose after that. | |
| `act_pri_final.pt` (epoch 4) is what produced the stage-2 labels. | |
| - **The grey line in the plots is not truth**, it's the linear training target. On | |
| rollouts with interventions and retries, real progress is not linear, so deviation | |
| from it is expected rather than error. | |
Xet Storage Details
- Size:
- 7.07 kB
- Xet hash:
- 2c66293d6bb10bbf191951b98317043ebff017e7adb72fe52ff6212e72b1b095
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.