Buckets:
Action-primitive labeling with Qwen3-VL-32B-Thinking
Tools for producing SARM2 action-primitive (act_pri) labels for a LeRobot
dataset, plus a browser UI to review and correct them.
Built for the SO-101 bimanual T-shirt folding dataset, restricted to four primitives:
| label | meaning | SARM2 index | MoE gate |
|---|---|---|---|
pick_grab |
an arm approaches / reaches for / pinches / grabs the cloth | 7 | 0 |
fold_flatten |
an arm folds a sleeve onto the shirt, or folds / flattens the bottom half | 2 | 4 |
place |
an arm places or settles cloth in the target area | 8 | 1 |
dummy |
arms idle, parked, retreating, not manipulating the cloth | 21 | 7 |
The indices are positions in the 22-entry task_list of
config/act_pri.yaml, which is the vocabulary
FrameGapLeRobotDataset expects in the act_pri column
(lerobot/common/datasets/rm_act_pri_lerobot_dataset.py). prick_grab is
accepted everywhere as an alias for pick_grab.
How episodes are proposed
The annotator no longer pre-splits episodes into fixed 1-second chunks. Instead, it samples the complete episode into a temporal storyboard and asks Qwen3-VL-32B-Thinking to return only variable-length semantic segments. Qwen is explicitly told that the number of segments is unknown, that a typical episode may have only 3-5 segments, and that it must create a boundary only when the primitive changes.
The original Qwen proposal is stored separately as metadata, while the editable
human segments remain the source of truth for the final per-frame act_pri
export.
1. Label the dataset
cd /home/ubuntu/Workspace/Thesis_Code/opensarm
source .venv/bin/activate
python tools/label_act_pri_qwen.py \
--dataset-root data \
--out-dir act_pri_labels
The model is configured in one place inside tools/act_pri_common.py
via ANNOTATOR_MODEL_ID and is currently fixed to Qwen/Qwen3-VL-32B-Thinking.
No 8B fallback path is kept.
Useful flags
| flag | default | what it does |
|---|---|---|
--dataset-root |
data |
LeRobot dataset dir (v2.x and v3.0 layouts both supported) |
--out-dir |
act_pri_labels |
label store, shared with the review UI |
--episodes |
all |
all, 0-4, 0,3,7 |
--sample-frames |
48 |
how many frames from the complete episode are shown to Qwen via storyboard pages |
--save-storyboards |
off | save the storyboard pages under images/episode_XXXXXX/ for debugging |
--overwrite |
off | relabel episodes that already have labels |
--local-files-only |
off | never contact the network |
2. Review and correct the predictions
python tools/annotate_act_pri_ui.py --labels-dir act_pri_labels --dataset-root data --port 8000
Open http://127.0.0.1:8000.
- one large video player
- one full-episode semantic timeline with draggable boundaries
- one label dropdown for the selected segment
- double-click the timeline to split at the current frame
- every change is written to disk immediately
Keyboard: Space play/pause, Left and Right step one frame, A approve.
The UI reads and writes exactly the same files the CLI produces, so corrections feed straight into training.
3. Build the stage-1 (act_pri) training dataset
workspace/act_pri_ws.py does not read an act_pri column. It reads the
label from task_index mapped through cfg.model.task_list:
dataset : seq_item["task"] = task_list[task_index] rm_act_pri_lerobot_dataset.py:226
training: act_pri_idx_gt = task_list.index(task) act_pri_ws.py:214
One label per sampled window means every episode must be a single-primitive
clip, with task_index set to that primitive's index in the 22-entry SARM2
vocabulary. build_act_pri_clips.py run-length encodes the reviewed per-frame
labels into such clips and writes a LeRobot v2.0 dataset:
python tools/build_act_pri_clips.py \
--dataset-root data \
--labels-dir act_pri_labels \
--repo-id Nadeem-Shoukath115113/so101_tshirt_act_pri_clips
Add --dry-run to see the clip plan without writing, --overwrite to rebuild.
It writes to ~/.cache/huggingface/lerobot/<repo-id>/ (where get_valid_episodes
and LeRobotDataset resolve a repo id):
meta/info.json | episodes.jsonl | tasks.jsonl | stats.json
data/chunk-000/episode_XXXXXX.parquet one clip per file
videos/chunk-000/<camera>/episode_XXXXXX.mp4 h264, all-intra (gop=1)
state_norm.json cfg.general.state_norm_path
Per-row parquet columns: state and actions (float32[12], copied from the
source episode), reward (progress 0->1 through the clip), act_pri (the
primitive index, for stage 2), timestamp (restarting at 0 per clip, matching
the clip's own video), frame_index, episode_index, index, and
task_index = the primitive index, which is the training target.
Clips shorter than --min-clip-frames (default 8) are dropped. On the reviewed
9-episode dataset this yields 89 clips / 3638 frames (dummy 36, pick_grab 27,
fold_flatten 26; median clip 30 frames), nothing dropped.
4. Verify it the way training will load it
python tools/check_act_pri_dataset.py --repo-id Nadeem-Shoukath115113/so101_tshirt_act_pri_clips
Checks the metadata, that task_index is constant within every clip and equals
the clip's primitive, that seq_item["task"] (the actual training target) comes
back as that primitive, that video decoding and DataLoader(num_workers>0)
work, that the state normalizer loads, and that the frozen SigLIP encoder
returns 768-d embeddings as act_pri_ws assumes.
5. Train the stage estimator
python train.py --config-name act_pri
config/act_pri.yaml is already pointed at the clip dataset. Note that Hydra
changes the working directory, so checkpoints land in
outputs/<date>/<time>/act_pri/so101_tshirt_3prim/.
Sizing for this dataset (80 train clips / 3234 samples, 9 val clips): batch 32
gives ~101 steps/epoch, so num_steps: 5_000 is ~50 epochs. total_steps must
equal that budget or the cosine schedule never anneals, and num_epochs is only
an outer bound - training stops at num_steps. Take the best val checkpoint
rather than the last.
One caveat worth knowing: split_train_eval_episodes splits clips at random,
so clips from the same source episode can land in both train and val. With 9
source episodes the val loss is an optimistic, noisy signal.
Output format (act_pri_labels/)
act_pri_labels/
├── vocab.json SARM2 22-class task_list, index map, MoE gates
├── groups/episode_XXXXXX.json per-group predictions, labels, scores, image paths
├── images/episode_XXXXXX/*.jpg the 3 camera images per group (used by the UI)
├── per_frame/episode_XXXXXX.npy int64 act_pri index, one per recorded frame
├── act_pri.parquet row-aligned act_pri column for training
└── summary.json label counts, modified counts, run-length segments
One group record:
{
"group_index": 3, "start_frame": 45, "end_frame": 59, "num_frames": 15,
"start_time": 3.0, "end_time": 4.0, "key_frame": 52, "key_time": 3.467,
"pred": "fold_flatten", "pred_index": 2,
"scores": {"pick_grab": 0.11, "fold_flatten": 0.87, "dummy": 0.02},
"label": "pick_grab", "label_index": 7,
"modified": true, "reviewed": true,
"images": {"observation.images.left_top": "images/episode_000000/g0003_left_top.jpg", "...": "..."}
}
pred is what Qwen said and never changes; label is the label in force
(what the UI edits); modified means the two differ.
act_pri.parquet has one row per recorded frame, in dataset row order:
| column | type | meaning |
|---|---|---|
index |
int64 | global row index into the dataset's data parquet |
episode_index |
int64 | |
frame_index |
int64 | frame within the episode |
act_pri |
int64 | SARM2 vocabulary index (7 / 2 / 21) |
act_pri_name |
string | readable name, for inspection only |
Step 2 attaches act_pri to the LeRobot data parquet by joining on index.
Repository fixes these tools depend on
Three defects blocked python train.py --config-name act_pri before any of this
work; all three are fixed in place:
lerobot/common/datasets/rm_act_pri_lerobot_dataset.pyforwardedpre_decode_video_framestoLeRobotDataset.__init__, which does not accept it (TypeErroron construction). It is now kept on the wrapper.models/siglip_encoder.pyassumedget_image_features/get_text_featuresreturn a tensor. transformers >= 5 returns aBaseModelOutputWithPooling, so the encoder now unwraps.pooler_output(both APIs supported).torchcodec==0.4(pinned inpyproject.toml) is built against an older torch and aborts the process on import (std::bad_alloc, uncatchable). Since LeRobot picks torchcodec as the default video backend when it is installed, every dataloader died. Upgraded totorchcodec==0.9.0, which imports and decodes correctly against torch 2.9.
Two environment packages were also repaired: torchaudio (see below) and
setuptools, which had been upgraded to 84 - that release removes
pkg_resources, which the pinned wandb==0.13.3 imports. Pinned back to
setuptools<81 rather than breaking the wandb pin.
Notes on this machine
torchaudiowas broken. The venv hadtorchaudio 2.11.0+cu130againsttorch 2.9.0+cu128, which made everytransformersimport fail withCould not load this library: _torchaudio.abi3.so. Fixed withuv pip install --no-deps torchaudio==2.9.0 --index-url https://download.pytorch.org/whl/cu128.Qwen/Qwen3-VL-32B-Thinkingmust be available locally or via the Hugging Face cache unless you omit--local-files-only.--local-files-onlyworks offline.- Vision patch-embed fast path. Qwen3-VL embeds patches with a
Conv3dwhose kernel equals its stride, i.e. an ordinary matmul over flattened patches. cuDNN picks a pathological algorithm for that shape on this GPU: measured 14 s per forward pass versus 0.15 ms for the identical matmul.apply_fast_patch_embed()folds the conv weights into annn.functional.linearcall at load time - numerically identical up to bf16 accumulation order, and it takes the run from ~14 s/group to ~0.13 s/group. Pass--no-fast-patch-embedto use the originalConv3d. - The
data/directory shipped as git-LFS pointer files. The real blobs were present in the HF hub cache (datasets--Nadeem-Shoukath115113--cloth_folder) and have been copied into place, sodata/is a complete dataset now.
Dataset this was built against
| layout | LeRobot v3.0 (aggregated parquet + aggregated mp4 per camera) |
| fps | 15 |
| episodes / frames | 9 / 3638 |
| robot | bi_so_follower (SO-101 bimanual), 12-D state and action |
| task | "Fold the T-shirt on the table" |
| cameras | observation.images.left_top, observation.images.left_left_wrist, observation.images.right_right_wrist |
| video codec | AV1, 480x640, decoded with PyAV / dav1d |
Both LeRobot v2.x and v3.0 layouts are handled, so --dataset-root can also
point at the v2.0 datasets under ~/.cache/huggingface/lerobot/.
Files
| file | role |
|---|---|
act_pri_common.py |
vocabulary, dataset introspection, grouping, per-frame expansion, parquet export |
label_act_pri_qwen.py |
step 1 - the Qwen labeling CLI |
annotate_act_pri_ui.py |
step 2 - the review web UI with boundary editing |
build_act_pri_clips.py |
step 3 - reviewed labels -> the act_pri clip dataset |
check_act_pri_dataset.py |
step 4 - verifies that dataset through the real training path |
- Total size
- 30.2 GB
- Files
- 6,432
- Last updated
- Oct 15
- Pre-warmed CDN
- US EU US EU