Buckets:
| # Action-primitive labeling with Qwen3-VL-32B-Thinking | |
| Tools for producing SARM2 **action-primitive (`act_pri`) labels** for a LeRobot | |
| dataset, plus a browser UI to review and correct them. | |
| Built for the SO-101 bimanual T-shirt folding dataset, restricted to four | |
| primitives: | |
| | label | meaning | SARM2 index | MoE gate | | |
| |---|---|---|---| | |
| | `pick_grab` | an arm approaches / reaches for / pinches / grabs the cloth | 7 | 0 | | |
| | `fold_flatten` | an arm folds a sleeve onto the shirt, or folds / flattens the bottom half | 2 | 4 | | |
| | `place` | an arm places or settles cloth in the target area | 8 | 1 | | |
| | `dummy` | arms idle, parked, retreating, not manipulating the cloth | 21 | 7 | | |
| The indices are positions in the 22-entry `task_list` of | |
| [`config/act_pri.yaml`](../config/act_pri.yaml), which is the vocabulary | |
| `FrameGapLeRobotDataset` expects in the `act_pri` column | |
| (`lerobot/common/datasets/rm_act_pri_lerobot_dataset.py`). `prick_grab` is | |
| accepted everywhere as an alias for `pick_grab`. | |
| --- | |
| ## How episodes are proposed | |
| The annotator no longer pre-splits episodes into fixed 1-second chunks. | |
| Instead, it samples the **complete episode** into a temporal storyboard and asks | |
| **Qwen3-VL-32B-Thinking** to return only **variable-length semantic segments**. | |
| Qwen is explicitly told that the number of segments is unknown, that a typical | |
| episode may have only 3-5 segments, and that it must create a boundary only | |
| when the primitive changes. | |
| The original Qwen proposal is stored separately as metadata, while the editable | |
| human segments remain the source of truth for the final per-frame `act_pri` | |
| export. | |
| --- | |
| ## 1. Label the dataset | |
| ```bash | |
| cd /home/ubuntu/Workspace/Thesis_Code/opensarm | |
| source .venv/bin/activate | |
| python tools/label_act_pri_qwen.py \ | |
| --dataset-root data \ | |
| --out-dir act_pri_labels | |
| ``` | |
| The model is configured in one place inside [`tools/act_pri_common.py`](act_pri_common.py) | |
| via `ANNOTATOR_MODEL_ID` and is currently fixed to `Qwen/Qwen3-VL-32B-Thinking`. | |
| No 8B fallback path is kept. | |
| ### Useful flags | |
| | flag | default | what it does | | |
| |---|---|---| | |
| | `--dataset-root` | `data` | LeRobot dataset dir (v2.x and v3.0 layouts both supported) | | |
| | `--out-dir` | `act_pri_labels` | label store, shared with the review UI | | |
| | `--episodes` | `all` | `all`, `0-4`, `0,3,7` | | |
| | `--sample-frames` | `48` | how many frames from the complete episode are shown to Qwen via storyboard pages | | |
| | `--save-storyboards` | off | save the storyboard pages under `images/episode_XXXXXX/` for debugging | | |
| | `--overwrite` | off | relabel episodes that already have labels | | |
| | `--local-files-only` | off | never contact the network | | |
| --- | |
| ## 2. Review and correct the predictions | |
| ```bash | |
| python tools/annotate_act_pri_ui.py --labels-dir act_pri_labels --dataset-root data --port 8000 | |
| ``` | |
| Open <http://127.0.0.1:8000>. | |
| * one large video player | |
| * one full-episode semantic timeline with draggable boundaries | |
| * one label dropdown for the selected segment | |
| * double-click the timeline to split at the current frame | |
| * every change is written to disk immediately | |
| Keyboard: `Space` play/pause, `Left` and `Right` step one frame, `A` approve. | |
| The UI reads and writes exactly the same files the CLI produces, so corrections | |
| feed straight into training. | |
| --- | |
| ## 3. Build the stage-1 (act_pri) training dataset | |
| `workspace/act_pri_ws.py` does **not** read an `act_pri` column. It reads the | |
| label from `task_index` mapped through `cfg.model.task_list`: | |
| ``` | |
| dataset : seq_item["task"] = task_list[task_index] rm_act_pri_lerobot_dataset.py:226 | |
| training: act_pri_idx_gt = task_list.index(task) act_pri_ws.py:214 | |
| ``` | |
| One label per sampled window means **every episode must be a single-primitive | |
| clip**, with `task_index` set to that primitive's index in the 22-entry SARM2 | |
| vocabulary. `build_act_pri_clips.py` run-length encodes the reviewed per-frame | |
| labels into such clips and writes a LeRobot v2.0 dataset: | |
| ```bash | |
| python tools/build_act_pri_clips.py \ | |
| --dataset-root data \ | |
| --labels-dir act_pri_labels \ | |
| --repo-id Nadeem-Shoukath115113/so101_tshirt_act_pri_clips | |
| ``` | |
| Add `--dry-run` to see the clip plan without writing, `--overwrite` to rebuild. | |
| It writes to `~/.cache/huggingface/lerobot/<repo-id>/` (where `get_valid_episodes` | |
| and `LeRobotDataset` resolve a repo id): | |
| ``` | |
| meta/info.json | episodes.jsonl | tasks.jsonl | stats.json | |
| data/chunk-000/episode_XXXXXX.parquet one clip per file | |
| videos/chunk-000/<camera>/episode_XXXXXX.mp4 h264, all-intra (gop=1) | |
| state_norm.json cfg.general.state_norm_path | |
| ``` | |
| Per-row parquet columns: `state` and `actions` (float32[12], copied from the | |
| source episode), `reward` (progress 0->1 through the clip), `act_pri` (the | |
| primitive index, for stage 2), `timestamp` (restarting at 0 per clip, matching | |
| the clip's own video), `frame_index`, `episode_index`, `index`, and | |
| **`task_index` = the primitive index**, which is the training target. | |
| Clips shorter than `--min-clip-frames` (default 8) are dropped. On the reviewed | |
| 9-episode dataset this yields **89 clips / 3638 frames** (dummy 36, pick_grab 27, | |
| fold_flatten 26; median clip 30 frames), nothing dropped. | |
| ## 4. Verify it the way training will load it | |
| ```bash | |
| python tools/check_act_pri_dataset.py --repo-id Nadeem-Shoukath115113/so101_tshirt_act_pri_clips | |
| ``` | |
| Checks the metadata, that `task_index` is constant within every clip and equals | |
| the clip's primitive, that `seq_item["task"]` (the actual training target) comes | |
| back as that primitive, that video decoding and `DataLoader(num_workers>0)` | |
| work, that the state normalizer loads, and that the frozen SigLIP encoder | |
| returns 768-d embeddings as `act_pri_ws` assumes. | |
| ## 5. Train the stage estimator | |
| ```bash | |
| python train.py --config-name act_pri | |
| ``` | |
| `config/act_pri.yaml` is already pointed at the clip dataset. Note that Hydra | |
| changes the working directory, so checkpoints land in | |
| `outputs/<date>/<time>/act_pri/so101_tshirt_3prim/`. | |
| Sizing for this dataset (80 train clips / 3234 samples, 9 val clips): batch 32 | |
| gives ~101 steps/epoch, so `num_steps: 5_000` is ~50 epochs. `total_steps` must | |
| equal that budget or the cosine schedule never anneals, and `num_epochs` is only | |
| an outer bound - training stops at `num_steps`. Take the best val checkpoint | |
| rather than the last. | |
| One caveat worth knowing: `split_train_eval_episodes` splits *clips* at random, | |
| so clips from the same source episode can land in both train and val. With 9 | |
| source episodes the val loss is an optimistic, noisy signal. | |
| --- | |
| ## Output format (`act_pri_labels/`) | |
| ``` | |
| act_pri_labels/ | |
| ├── vocab.json SARM2 22-class task_list, index map, MoE gates | |
| ├── groups/episode_XXXXXX.json per-group predictions, labels, scores, image paths | |
| ├── images/episode_XXXXXX/*.jpg the 3 camera images per group (used by the UI) | |
| ├── per_frame/episode_XXXXXX.npy int64 act_pri index, one per recorded frame | |
| ├── act_pri.parquet row-aligned act_pri column for training | |
| └── summary.json label counts, modified counts, run-length segments | |
| ``` | |
| One group record: | |
| ```json | |
| { | |
| "group_index": 3, "start_frame": 45, "end_frame": 59, "num_frames": 15, | |
| "start_time": 3.0, "end_time": 4.0, "key_frame": 52, "key_time": 3.467, | |
| "pred": "fold_flatten", "pred_index": 2, | |
| "scores": {"pick_grab": 0.11, "fold_flatten": 0.87, "dummy": 0.02}, | |
| "label": "pick_grab", "label_index": 7, | |
| "modified": true, "reviewed": true, | |
| "images": {"observation.images.left_top": "images/episode_000000/g0003_left_top.jpg", "...": "..."} | |
| } | |
| ``` | |
| `pred` is what Qwen said and never changes; `label` is the label in force | |
| (what the UI edits); `modified` means the two differ. | |
| `act_pri.parquet` has one row per recorded frame, in dataset row order: | |
| | column | type | meaning | | |
| |---|---|---| | |
| | `index` | int64 | global row index into the dataset's data parquet | | |
| | `episode_index` | int64 | | | |
| | `frame_index` | int64 | frame within the episode | | |
| | `act_pri` | int64 | SARM2 vocabulary index (7 / 2 / 21) | | |
| | `act_pri_name` | string | readable name, for inspection only | | |
| Step 2 attaches `act_pri` to the LeRobot data parquet by joining on `index`. | |
| --- | |
| ## Repository fixes these tools depend on | |
| Three defects blocked `python train.py --config-name act_pri` before any of this | |
| work; all three are fixed in place: | |
| * `lerobot/common/datasets/rm_act_pri_lerobot_dataset.py` forwarded | |
| `pre_decode_video_frames` to `LeRobotDataset.__init__`, which does not accept | |
| it (`TypeError` on construction). It is now kept on the wrapper. | |
| * `models/siglip_encoder.py` assumed `get_image_features` / `get_text_features` | |
| return a tensor. transformers >= 5 returns a `BaseModelOutputWithPooling`, so | |
| the encoder now unwraps `.pooler_output` (both APIs supported). | |
| * `torchcodec==0.4` (pinned in `pyproject.toml`) is built against an older torch | |
| and **aborts the process** on import (`std::bad_alloc`, uncatchable). Since | |
| LeRobot picks torchcodec as the default video backend when it is installed, | |
| every dataloader died. Upgraded to `torchcodec==0.9.0`, which imports and | |
| decodes correctly against torch 2.9. | |
| Two environment packages were also repaired: `torchaudio` (see below) and | |
| `setuptools`, which had been upgraded to 84 - that release removes | |
| `pkg_resources`, which the pinned `wandb==0.13.3` imports. Pinned back to | |
| `setuptools<81` rather than breaking the wandb pin. | |
| ## Notes on this machine | |
| * **`torchaudio` was broken.** The venv had `torchaudio 2.11.0+cu130` against | |
| `torch 2.9.0+cu128`, which made every `transformers` import fail with | |
| `Could not load this library: _torchaudio.abi3.so`. Fixed with | |
| `uv pip install --no-deps torchaudio==2.9.0 --index-url https://download.pytorch.org/whl/cu128`. | |
| * **`Qwen/Qwen3-VL-32B-Thinking`** must be available locally or via the Hugging Face cache unless you omit `--local-files-only`. | |
| `--local-files-only` works offline. | |
| * **Vision patch-embed fast path.** Qwen3-VL embeds patches with a `Conv3d` | |
| whose kernel equals its stride, i.e. an ordinary matmul over flattened | |
| patches. cuDNN picks a pathological algorithm for that shape on this GPU: | |
| measured **14 s** per forward pass versus **0.15 ms** for the identical | |
| matmul. `apply_fast_patch_embed()` folds the conv weights into an | |
| `nn.functional.linear` call at load time - numerically identical up to bf16 | |
| accumulation order, and it takes the run from ~14 s/group to ~0.13 s/group. | |
| Pass `--no-fast-patch-embed` to use the original `Conv3d`. | |
| * **The `data/` directory shipped as git-LFS pointer files.** The real blobs | |
| were present in the HF hub cache | |
| (`datasets--Nadeem-Shoukath115113--cloth_folder`) and have been copied into | |
| place, so `data/` is a complete dataset now. | |
| ## Dataset this was built against | |
| | | | | |
| |---|---| | |
| | layout | LeRobot `v3.0` (aggregated parquet + aggregated mp4 per camera) | | |
| | fps | 15 | | |
| | episodes / frames | 9 / 3638 | | |
| | robot | `bi_so_follower` (SO-101 bimanual), 12-D state and action | | |
| | task | "Fold the T-shirt on the table" | | |
| | cameras | `observation.images.left_top`, `observation.images.left_left_wrist`, `observation.images.right_right_wrist` | | |
| | video codec | AV1, 480x640, decoded with PyAV / dav1d | | |
| Both LeRobot v2.x and v3.0 layouts are handled, so `--dataset-root` can also | |
| point at the v2.0 datasets under `~/.cache/huggingface/lerobot/`. | |
| --- | |
| ## Files | |
| | file | role | | |
| |---|---| | |
| | `act_pri_common.py` | vocabulary, dataset introspection, grouping, per-frame expansion, parquet export | | |
| | `label_act_pri_qwen.py` | step 1 - the Qwen labeling CLI | | |
| | `annotate_act_pri_ui.py` | step 2 - the review web UI with boundary editing | | |
| | `build_act_pri_clips.py` | step 3 - reviewed labels -> the act_pri clip dataset | | |
| | `check_act_pri_dataset.py` | step 4 - verifies that dataset through the real training path | | |
Xet Storage Details
- Size:
- 11.8 kB
- Xet hash:
- 5753240a65828e17fbec4563d21df8a6e55b8025ed2219ff24a0d09c7cfcc757
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.