Nadeem-Shoukath115113's picture
|
download
raw
7.07 kB
# Trained models — where they are and how to use them
Two models were trained on this machine. Both are copied into `checkpoints/`, which
is stable; the `outputs/<date>/<time>/` originals are Hydra run directories and are
easy to lose track of.
```
checkpoints/
├── act_pri/ stage 1 — action-primitive classifier
│ ├── act_pri_best.pt epoch 1, val CE 0.709 <- best held-out
│ ├── act_pri_final.pt epoch 4 <- used for labelling
│ └── state_norm.json the norm stats it was trained with
└── sarm2/ stage 2 — MMoE value head
├── reward_best.pt best val MSE 0.0585 on steps_to_go
├── act_pri_best.pt the stage-1 model after joint finetuning
└── state_norm.json
```
Originals: `outputs/2026-08-21/12-22-57/` (act_pri) and `outputs/2026-08-21/14-06-07/` (sarm2).
**Always pair a checkpoint with its own `state_norm.json`.** The state normalizer is
part of the model; using the wrong one silently degrades predictions.
---
## What each model does
| | input | output |
|---|---|---|
| **act_pri** | 6 frames × 3 cameras spaced `frame_gap=30`, + state | one of `pick_grab` / `fold_flatten` / `dummy` for the **last** frame |
| **sarm2** | `[episode start] + 6 frames` × 3 cameras, + state, + task text | `steps_to_go ∈ [0,1]`; progress = `1 − steps_to_go` |
Both use a frozen `google/siglip2-base-patch16-224` backbone. `frame_gap` must match
between them — stage 2 feeds act_pri a slice of its own sequence
(`sarm2_ws._slice_act_pri_inputs`), so a mismatch is a silent train/inference bug.
---
## Label a dataset with action primitives
Writes one primitive per frame, plus the JPEGs and JSON the review UI reads.
```bash
cd /home/ubuntu/Workspace/Thesis_Code/opensarm
source .venv/bin/activate
python tools/predict_act_pri.py \
--dataset-root /path/to/lerobot_dataset \
--out-dir act_pri_preds --session my_session \
--checkpoint checkpoints/act_pri/act_pri_final.pt --name act_pri \
--state-norm checkpoints/act_pri/state_norm.json \
--frame-gap 30
```
Shortcut for the three cloth_folder sessions (auto-detects the newest checkpoint and
reads `frame_gap` / `n_obs_steps` / `state_norm` from `config/act_pri.yaml`):
```bash
bash tools/run_act_pri_preds.sh # all sessions
bash tools/run_act_pri_preds.sh 20260724_171909 # one
```
Useful flags: `--group-seconds` (default 1.0 — one label per second of video),
`--episodes 0-9`, `--no-images` (skip the review JPEGs), `--batch-size`.
### Review and correct the labels
```bash
python tools/annotate_act_pri_ui.py --labels-dir act_pri_preds --dataset-root data --port 8000
```
<http://127.0.0.1:8000> — session dropdown, 3 camera views, the model's label and its
confidence. `←`/`→` step, `1`/`2`/`3` set `pick_grab`/`fold_flatten`/`dummy`, `r`
resets to the prediction. Every edit writes straight to `groups/*.json`,
`per_frame/*.npy` and `act_pri.parquet` — no save button. Corrected groups are
outlined in red; `modified →` jumps between them.
---
## Score a dataset with SARM2 (progress)
### Static plot — several episodes at once
```bash
python tools/visualise_sarm2.py \
--dataset-root /path/to/lerobot_dataset \
--split all --episodes 5 --out progress.png
```
Teal = predicted progress, grey dashed = the linear `t/(T-1)` reference, amber =
`is_policy` stretches if the dataset carries that column (DAgger). Also prints
`mean |pred − gt|` and the final progress per episode.
`--episode-ids 10,16` picks specific ones, `--stride` trades speed for resolution
(default 5 = one prediction per 5 frames), `--split val` restricts to held-out episodes.
### Side-by-side video
```bash
python tools/make_sarm2_video.py \
--dataset-root /path/to/lerobot_dataset \
--episodes 3 --out sarm2_videos
```
Runs on **every** frame so predictions line up 1:1 with the video, then calls the
repo's own `utils.make_demo_video.produce_video`. Writes per episode:
```
sarm2_videos/episode_XXXXXX/
├── combined_video.mp4 camera view + progress plot with a moving cursor
├── camera.mp4 the extracted camera view
├── pred.npy progress per frame
└── gt.npy linear reference
```
`--camera observation.images.left_left_wrist` switches the view;
`--episode-ids 10,16` picks episodes.
Note the plot is labelled "Reward Model Prediction" — that text comes from
`draw_plot_frame` in the repo; the values plotted are progress.
---
## Build a training dataset from labels
Stage 1 (whole episodes, per-frame `act_pri` as the CE target):
```bash
python tools/build_act_pri_clips.py --whole-episodes \
--source /path/to/session_a act_pri_labels \
--source /path/to/session_b act_pri_preds/session_b \
--repo-id <namespace>/<name> --overwrite
```
Add `--reviewed-only` to keep just the groups a human confirmed in the UI.
Stage 2 (whole-task reward + act_pri, both columns generated):
```bash
python tools/build_sarm2_dataset.py \
--source /path/to/source_dataset \
--repo-id <namespace>/<name> --overwrite \
--reward linear \
--act-pri-checkpoint checkpoints/act_pri/act_pri_final.pt \
--state-norm checkpoints/act_pri/state_norm.json --frame-gap 30
```
`--reward linear` writes `reward = t/(T-1)`, the progress SARM2 converts internally to
`steps_to_go = (T-t)/(T-1)`. Existing `reward` / `progress` columns and any
`sarm_progress.parquet` are dropped first. Videos are hardlinked, so the copy is free.
Verify before training:
```bash
python tools/check_act_pri_dataset.py --repo-id <namespace>/<name>
```
---
## Retrain
```bash
python train.py --config-name act_pri # stage 1
python train.py --config-name sarm2 # stage 2
```
Datasets are resolved by `repo_id` under `~/.cache/huggingface/lerobot/`, which got
cleared twice on this machine — if training reports a missing dataset, rebuild it with
the commands above. Checkpoints and the label stores in this repo were never affected.
If a dataset trains slowly, check the video GOP: random frame access costs ~GOP/2
decodes per frame. `python tools/reencode_allintra.py --dataset-root <dir>` re-encodes
every frame as a keyframe (~100 s for 486 videos, roughly doubles the size).
---
## Caveats worth carrying forward
- **act_pri's ground truth is only ~29% human-verified.** Of 1,456 one-second groups,
428 were reviewed by hand; most of the rest are an earlier act_pri model's own
predictions, so the model is partly trained on its own output. `--reviewed-only`
builds a smaller, fully human-labelled set.
- **`act_pri_best.pt` is epoch 1** in both runs — validation loss rose after that.
`act_pri_final.pt` (epoch 4) is what produced the stage-2 labels.
- **The grey line in the plots is not truth**, it's the linear training target. On
rollouts with interventions and retries, real progress is not linear, so deviation
from it is expected rather than error.

Xet Storage Details

Size:
7.07 kB
·
Xet hash:
2c66293d6bb10bbf191951b98317043ebff017e7adb72fe52ff6212e72b1b095

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.