30.2 GB
6,432 files
Updated about 1 month ago
Name
Size
act_pri_viz
PLAN_bimanual_fold_tshirt_local.md6.75 kB
xet
README.md11.8 kB
xet
README_MODELS.md7.07 kB
xet
act_pri_common.py29.4 kB
xet
annotate_act_pri_ui.py33.3 kB
xet
build_act_pri_clips.py29.1 kB
xet
build_sarm2_cloth.py15.1 kB
xet
build_sarm2_dataset.py17.2 kB
xet
build_sarm2_from_rollout_dirs.py17.4 kB
xet
check_act_pri_dataset.py8.84 kB
xet
export_act_pri_lerobot_dataset.py10.3 kB
xet
label_act_pri_qwen.py29.8 kB
xet
label_for_spiral.py24.3 kB
xet
lerobot_to_rollout_dirs.py7.29 kB
xet
make_data_all_videos.py3.52 kB
xet
make_data_all_videos.sh2.83 kB
xet
make_linear_reward_dataset.py3.94 kB
xet
make_reward_collage.py5.4 kB
xet
make_sarm2_video.py20.3 kB
xet
merge_lerobot_datasets.py8.66 kB
xet
patch_progress_gripper.py24.5 kB
xet
plot_spiral_labels.py3.47 kB
xet
predict_act_pri.py14.6 kB
xet
reencode_allintra.py3.53 kB
xet
reencode_dense_gop.py3.27 kB
xet
render_label_review.py4.32 kB
xet
resume_sarm2.sh907 Bytes
xet
run_act_pri_preds.sh1.87 kB
xet
seed_rollout_labels_from_preds.py4.04 kB
xet
subset_lerobot_dataset.py8.08 kB
xet
train_sarm2_cloth.sh1.4 kB
xet
verify_act_pri_dataset.py10.8 kB
xet
verify_gripper_reward.py6.4 kB
xet
visualise_sarm2.py12.1 kB
xet
README.md

Action-primitive labeling with Qwen3-VL-32B-Thinking

Tools for producing SARM2 action-primitive (act_pri) labels for a LeRobot dataset, plus a browser UI to review and correct them.

Built for the SO-101 bimanual T-shirt folding dataset, restricted to four primitives:

label meaning SARM2 index MoE gate
pick_grab an arm approaches / reaches for / pinches / grabs the cloth 7 0
fold_flatten an arm folds a sleeve onto the shirt, or folds / flattens the bottom half 2 4
place an arm places or settles cloth in the target area 8 1
dummy arms idle, parked, retreating, not manipulating the cloth 21 7

The indices are positions in the 22-entry task_list of config/act_pri.yaml, which is the vocabulary FrameGapLeRobotDataset expects in the act_pri column (lerobot/common/datasets/rm_act_pri_lerobot_dataset.py). prick_grab is accepted everywhere as an alias for pick_grab.


How episodes are proposed

The annotator no longer pre-splits episodes into fixed 1-second chunks. Instead, it samples the complete episode into a temporal storyboard and asks Qwen3-VL-32B-Thinking to return only variable-length semantic segments. Qwen is explicitly told that the number of segments is unknown, that a typical episode may have only 3-5 segments, and that it must create a boundary only when the primitive changes.

The original Qwen proposal is stored separately as metadata, while the editable human segments remain the source of truth for the final per-frame act_pri export.


1. Label the dataset

cd /home/ubuntu/Workspace/Thesis_Code/opensarm
source .venv/bin/activate

python tools/label_act_pri_qwen.py \
    --dataset-root data \
  --out-dir act_pri_labels

The model is configured in one place inside tools/act_pri_common.py via ANNOTATOR_MODEL_ID and is currently fixed to Qwen/Qwen3-VL-32B-Thinking. No 8B fallback path is kept.

Useful flags

flag default what it does
--dataset-root data LeRobot dataset dir (v2.x and v3.0 layouts both supported)
--out-dir act_pri_labels label store, shared with the review UI
--episodes all all, 0-4, 0,3,7
--sample-frames 48 how many frames from the complete episode are shown to Qwen via storyboard pages
--save-storyboards off save the storyboard pages under images/episode_XXXXXX/ for debugging
--overwrite off relabel episodes that already have labels
--local-files-only off never contact the network

2. Review and correct the predictions

python tools/annotate_act_pri_ui.py --labels-dir act_pri_labels --dataset-root data --port 8000

Open http://127.0.0.1:8000.

  • one large video player
  • one full-episode semantic timeline with draggable boundaries
  • one label dropdown for the selected segment
  • double-click the timeline to split at the current frame
  • every change is written to disk immediately

Keyboard: Space play/pause, Left and Right step one frame, A approve.

The UI reads and writes exactly the same files the CLI produces, so corrections feed straight into training.


3. Build the stage-1 (act_pri) training dataset

workspace/act_pri_ws.py does not read an act_pri column. It reads the label from task_index mapped through cfg.model.task_list:

dataset : seq_item["task"] = task_list[task_index]     rm_act_pri_lerobot_dataset.py:226
training: act_pri_idx_gt   = task_list.index(task)     act_pri_ws.py:214

One label per sampled window means every episode must be a single-primitive clip, with task_index set to that primitive's index in the 22-entry SARM2 vocabulary. build_act_pri_clips.py run-length encodes the reviewed per-frame labels into such clips and writes a LeRobot v2.0 dataset:

python tools/build_act_pri_clips.py \
    --dataset-root data \
    --labels-dir act_pri_labels \
    --repo-id Nadeem-Shoukath115113/so101_tshirt_act_pri_clips

Add --dry-run to see the clip plan without writing, --overwrite to rebuild.

It writes to ~/.cache/huggingface/lerobot/<repo-id>/ (where get_valid_episodes and LeRobotDataset resolve a repo id):

meta/info.json | episodes.jsonl | tasks.jsonl | stats.json
data/chunk-000/episode_XXXXXX.parquet     one clip per file
videos/chunk-000/<camera>/episode_XXXXXX.mp4   h264, all-intra (gop=1)
state_norm.json                           cfg.general.state_norm_path

Per-row parquet columns: state and actions (float32[12], copied from the source episode), reward (progress 0->1 through the clip), act_pri (the primitive index, for stage 2), timestamp (restarting at 0 per clip, matching the clip's own video), frame_index, episode_index, index, and task_index = the primitive index, which is the training target.

Clips shorter than --min-clip-frames (default 8) are dropped. On the reviewed 9-episode dataset this yields 89 clips / 3638 frames (dummy 36, pick_grab 27, fold_flatten 26; median clip 30 frames), nothing dropped.

4. Verify it the way training will load it

python tools/check_act_pri_dataset.py --repo-id Nadeem-Shoukath115113/so101_tshirt_act_pri_clips

Checks the metadata, that task_index is constant within every clip and equals the clip's primitive, that seq_item["task"] (the actual training target) comes back as that primitive, that video decoding and DataLoader(num_workers>0) work, that the state normalizer loads, and that the frozen SigLIP encoder returns 768-d embeddings as act_pri_ws assumes.

5. Train the stage estimator

python train.py --config-name act_pri

config/act_pri.yaml is already pointed at the clip dataset. Note that Hydra changes the working directory, so checkpoints land in outputs/<date>/<time>/act_pri/so101_tshirt_3prim/.

Sizing for this dataset (80 train clips / 3234 samples, 9 val clips): batch 32 gives ~101 steps/epoch, so num_steps: 5_000 is ~50 epochs. total_steps must equal that budget or the cosine schedule never anneals, and num_epochs is only an outer bound - training stops at num_steps. Take the best val checkpoint rather than the last.

One caveat worth knowing: split_train_eval_episodes splits clips at random, so clips from the same source episode can land in both train and val. With 9 source episodes the val loss is an optimistic, noisy signal.


Output format (act_pri_labels/)

act_pri_labels/
├── vocab.json                     SARM2 22-class task_list, index map, MoE gates
├── groups/episode_XXXXXX.json     per-group predictions, labels, scores, image paths
├── images/episode_XXXXXX/*.jpg    the 3 camera images per group (used by the UI)
├── per_frame/episode_XXXXXX.npy   int64 act_pri index, one per recorded frame
├── act_pri.parquet                row-aligned act_pri column for training
└── summary.json                   label counts, modified counts, run-length segments

One group record:

{
  "group_index": 3, "start_frame": 45, "end_frame": 59, "num_frames": 15,
  "start_time": 3.0, "end_time": 4.0, "key_frame": 52, "key_time": 3.467,
  "pred": "fold_flatten", "pred_index": 2,
  "scores": {"pick_grab": 0.11, "fold_flatten": 0.87, "dummy": 0.02},
  "label": "pick_grab", "label_index": 7,
  "modified": true, "reviewed": true,
  "images": {"observation.images.left_top": "images/episode_000000/g0003_left_top.jpg", "...": "..."}
}

pred is what Qwen said and never changes; label is the label in force (what the UI edits); modified means the two differ.

act_pri.parquet has one row per recorded frame, in dataset row order:

column type meaning
index int64 global row index into the dataset's data parquet
episode_index int64
frame_index int64 frame within the episode
act_pri int64 SARM2 vocabulary index (7 / 2 / 21)
act_pri_name string readable name, for inspection only

Step 2 attaches act_pri to the LeRobot data parquet by joining on index.


Repository fixes these tools depend on

Three defects blocked python train.py --config-name act_pri before any of this work; all three are fixed in place:

  • lerobot/common/datasets/rm_act_pri_lerobot_dataset.py forwarded pre_decode_video_frames to LeRobotDataset.__init__, which does not accept it (TypeError on construction). It is now kept on the wrapper.
  • models/siglip_encoder.py assumed get_image_features / get_text_features return a tensor. transformers >= 5 returns a BaseModelOutputWithPooling, so the encoder now unwraps .pooler_output (both APIs supported).
  • torchcodec==0.4 (pinned in pyproject.toml) is built against an older torch and aborts the process on import (std::bad_alloc, uncatchable). Since LeRobot picks torchcodec as the default video backend when it is installed, every dataloader died. Upgraded to torchcodec==0.9.0, which imports and decodes correctly against torch 2.9.

Two environment packages were also repaired: torchaudio (see below) and setuptools, which had been upgraded to 84 - that release removes pkg_resources, which the pinned wandb==0.13.3 imports. Pinned back to setuptools<81 rather than breaking the wandb pin.

Notes on this machine

  • torchaudio was broken. The venv had torchaudio 2.11.0+cu130 against torch 2.9.0+cu128, which made every transformers import fail with Could not load this library: _torchaudio.abi3.so. Fixed with uv pip install --no-deps torchaudio==2.9.0 --index-url https://download.pytorch.org/whl/cu128.
  • Qwen/Qwen3-VL-32B-Thinking must be available locally or via the Hugging Face cache unless you omit --local-files-only. --local-files-only works offline.
  • Vision patch-embed fast path. Qwen3-VL embeds patches with a Conv3d whose kernel equals its stride, i.e. an ordinary matmul over flattened patches. cuDNN picks a pathological algorithm for that shape on this GPU: measured 14 s per forward pass versus 0.15 ms for the identical matmul. apply_fast_patch_embed() folds the conv weights into an nn.functional.linear call at load time - numerically identical up to bf16 accumulation order, and it takes the run from ~14 s/group to ~0.13 s/group. Pass --no-fast-patch-embed to use the original Conv3d.
  • The data/ directory shipped as git-LFS pointer files. The real blobs were present in the HF hub cache (datasets--Nadeem-Shoukath115113--cloth_folder) and have been copied into place, so data/ is a complete dataset now.

Dataset this was built against

layout LeRobot v3.0 (aggregated parquet + aggregated mp4 per camera)
fps 15
episodes / frames 9 / 3638
robot bi_so_follower (SO-101 bimanual), 12-D state and action
task "Fold the T-shirt on the table"
cameras observation.images.left_top, observation.images.left_left_wrist, observation.images.right_right_wrist
video codec AV1, 480x640, decoded with PyAV / dav1d

Both LeRobot v2.x and v3.0 layouts are handled, so --dataset-root can also point at the v2.0 datasets under ~/.cache/huggingface/lerobot/.


Files

file role
act_pri_common.py vocabulary, dataset introspection, grouping, per-frame expansion, parquet export
label_act_pri_qwen.py step 1 - the Qwen labeling CLI
annotate_act_pri_ui.py step 2 - the review web UI with boundary editing
build_act_pri_clips.py step 3 - reviewed labels -> the act_pri clip dataset
check_act_pri_dataset.py step 4 - verifies that dataset through the real training path
Total size
30.2 GB
Files
6,432
Last updated
Oct 15
Pre-warmed CDN
US EU US EU

Contributors