Nadeem-Shoukath115113's picture
|
download
raw
11.8 kB
# Action-primitive labeling with Qwen3-VL-32B-Thinking
Tools for producing SARM2 **action-primitive (`act_pri`) labels** for a LeRobot
dataset, plus a browser UI to review and correct them.
Built for the SO-101 bimanual T-shirt folding dataset, restricted to four
primitives:
| label | meaning | SARM2 index | MoE gate |
|---|---|---|---|
| `pick_grab` | an arm approaches / reaches for / pinches / grabs the cloth | 7 | 0 |
| `fold_flatten` | an arm folds a sleeve onto the shirt, or folds / flattens the bottom half | 2 | 4 |
| `place` | an arm places or settles cloth in the target area | 8 | 1 |
| `dummy` | arms idle, parked, retreating, not manipulating the cloth | 21 | 7 |
The indices are positions in the 22-entry `task_list` of
[`config/act_pri.yaml`](../config/act_pri.yaml), which is the vocabulary
`FrameGapLeRobotDataset` expects in the `act_pri` column
(`lerobot/common/datasets/rm_act_pri_lerobot_dataset.py`). `prick_grab` is
accepted everywhere as an alias for `pick_grab`.
---
## How episodes are proposed
The annotator no longer pre-splits episodes into fixed 1-second chunks.
Instead, it samples the **complete episode** into a temporal storyboard and asks
**Qwen3-VL-32B-Thinking** to return only **variable-length semantic segments**.
Qwen is explicitly told that the number of segments is unknown, that a typical
episode may have only 3-5 segments, and that it must create a boundary only
when the primitive changes.
The original Qwen proposal is stored separately as metadata, while the editable
human segments remain the source of truth for the final per-frame `act_pri`
export.
---
## 1. Label the dataset
```bash
cd /home/ubuntu/Workspace/Thesis_Code/opensarm
source .venv/bin/activate
python tools/label_act_pri_qwen.py \
--dataset-root data \
--out-dir act_pri_labels
```
The model is configured in one place inside [`tools/act_pri_common.py`](act_pri_common.py)
via `ANNOTATOR_MODEL_ID` and is currently fixed to `Qwen/Qwen3-VL-32B-Thinking`.
No 8B fallback path is kept.
### Useful flags
| flag | default | what it does |
|---|---|---|
| `--dataset-root` | `data` | LeRobot dataset dir (v2.x and v3.0 layouts both supported) |
| `--out-dir` | `act_pri_labels` | label store, shared with the review UI |
| `--episodes` | `all` | `all`, `0-4`, `0,3,7` |
| `--sample-frames` | `48` | how many frames from the complete episode are shown to Qwen via storyboard pages |
| `--save-storyboards` | off | save the storyboard pages under `images/episode_XXXXXX/` for debugging |
| `--overwrite` | off | relabel episodes that already have labels |
| `--local-files-only` | off | never contact the network |
---
## 2. Review and correct the predictions
```bash
python tools/annotate_act_pri_ui.py --labels-dir act_pri_labels --dataset-root data --port 8000
```
Open <http://127.0.0.1:8000>.
* one large video player
* one full-episode semantic timeline with draggable boundaries
* one label dropdown for the selected segment
* double-click the timeline to split at the current frame
* every change is written to disk immediately
Keyboard: `Space` play/pause, `Left` and `Right` step one frame, `A` approve.
The UI reads and writes exactly the same files the CLI produces, so corrections
feed straight into training.
---
## 3. Build the stage-1 (act_pri) training dataset
`workspace/act_pri_ws.py` does **not** read an `act_pri` column. It reads the
label from `task_index` mapped through `cfg.model.task_list`:
```
dataset : seq_item["task"] = task_list[task_index] rm_act_pri_lerobot_dataset.py:226
training: act_pri_idx_gt = task_list.index(task) act_pri_ws.py:214
```
One label per sampled window means **every episode must be a single-primitive
clip**, with `task_index` set to that primitive's index in the 22-entry SARM2
vocabulary. `build_act_pri_clips.py` run-length encodes the reviewed per-frame
labels into such clips and writes a LeRobot v2.0 dataset:
```bash
python tools/build_act_pri_clips.py \
--dataset-root data \
--labels-dir act_pri_labels \
--repo-id Nadeem-Shoukath115113/so101_tshirt_act_pri_clips
```
Add `--dry-run` to see the clip plan without writing, `--overwrite` to rebuild.
It writes to `~/.cache/huggingface/lerobot/<repo-id>/` (where `get_valid_episodes`
and `LeRobotDataset` resolve a repo id):
```
meta/info.json | episodes.jsonl | tasks.jsonl | stats.json
data/chunk-000/episode_XXXXXX.parquet one clip per file
videos/chunk-000/<camera>/episode_XXXXXX.mp4 h264, all-intra (gop=1)
state_norm.json cfg.general.state_norm_path
```
Per-row parquet columns: `state` and `actions` (float32[12], copied from the
source episode), `reward` (progress 0->1 through the clip), `act_pri` (the
primitive index, for stage 2), `timestamp` (restarting at 0 per clip, matching
the clip's own video), `frame_index`, `episode_index`, `index`, and
**`task_index` = the primitive index**, which is the training target.
Clips shorter than `--min-clip-frames` (default 8) are dropped. On the reviewed
9-episode dataset this yields **89 clips / 3638 frames** (dummy 36, pick_grab 27,
fold_flatten 26; median clip 30 frames), nothing dropped.
## 4. Verify it the way training will load it
```bash
python tools/check_act_pri_dataset.py --repo-id Nadeem-Shoukath115113/so101_tshirt_act_pri_clips
```
Checks the metadata, that `task_index` is constant within every clip and equals
the clip's primitive, that `seq_item["task"]` (the actual training target) comes
back as that primitive, that video decoding and `DataLoader(num_workers>0)`
work, that the state normalizer loads, and that the frozen SigLIP encoder
returns 768-d embeddings as `act_pri_ws` assumes.
## 5. Train the stage estimator
```bash
python train.py --config-name act_pri
```
`config/act_pri.yaml` is already pointed at the clip dataset. Note that Hydra
changes the working directory, so checkpoints land in
`outputs/<date>/<time>/act_pri/so101_tshirt_3prim/`.
Sizing for this dataset (80 train clips / 3234 samples, 9 val clips): batch 32
gives ~101 steps/epoch, so `num_steps: 5_000` is ~50 epochs. `total_steps` must
equal that budget or the cosine schedule never anneals, and `num_epochs` is only
an outer bound - training stops at `num_steps`. Take the best val checkpoint
rather than the last.
One caveat worth knowing: `split_train_eval_episodes` splits *clips* at random,
so clips from the same source episode can land in both train and val. With 9
source episodes the val loss is an optimistic, noisy signal.
---
## Output format (`act_pri_labels/`)
```
act_pri_labels/
├── vocab.json SARM2 22-class task_list, index map, MoE gates
├── groups/episode_XXXXXX.json per-group predictions, labels, scores, image paths
├── images/episode_XXXXXX/*.jpg the 3 camera images per group (used by the UI)
├── per_frame/episode_XXXXXX.npy int64 act_pri index, one per recorded frame
├── act_pri.parquet row-aligned act_pri column for training
└── summary.json label counts, modified counts, run-length segments
```
One group record:
```json
{
"group_index": 3, "start_frame": 45, "end_frame": 59, "num_frames": 15,
"start_time": 3.0, "end_time": 4.0, "key_frame": 52, "key_time": 3.467,
"pred": "fold_flatten", "pred_index": 2,
"scores": {"pick_grab": 0.11, "fold_flatten": 0.87, "dummy": 0.02},
"label": "pick_grab", "label_index": 7,
"modified": true, "reviewed": true,
"images": {"observation.images.left_top": "images/episode_000000/g0003_left_top.jpg", "...": "..."}
}
```
`pred` is what Qwen said and never changes; `label` is the label in force
(what the UI edits); `modified` means the two differ.
`act_pri.parquet` has one row per recorded frame, in dataset row order:
| column | type | meaning |
|---|---|---|
| `index` | int64 | global row index into the dataset's data parquet |
| `episode_index` | int64 | |
| `frame_index` | int64 | frame within the episode |
| `act_pri` | int64 | SARM2 vocabulary index (7 / 2 / 21) |
| `act_pri_name` | string | readable name, for inspection only |
Step 2 attaches `act_pri` to the LeRobot data parquet by joining on `index`.
---
## Repository fixes these tools depend on
Three defects blocked `python train.py --config-name act_pri` before any of this
work; all three are fixed in place:
* `lerobot/common/datasets/rm_act_pri_lerobot_dataset.py` forwarded
`pre_decode_video_frames` to `LeRobotDataset.__init__`, which does not accept
it (`TypeError` on construction). It is now kept on the wrapper.
* `models/siglip_encoder.py` assumed `get_image_features` / `get_text_features`
return a tensor. transformers >= 5 returns a `BaseModelOutputWithPooling`, so
the encoder now unwraps `.pooler_output` (both APIs supported).
* `torchcodec==0.4` (pinned in `pyproject.toml`) is built against an older torch
and **aborts the process** on import (`std::bad_alloc`, uncatchable). Since
LeRobot picks torchcodec as the default video backend when it is installed,
every dataloader died. Upgraded to `torchcodec==0.9.0`, which imports and
decodes correctly against torch 2.9.
Two environment packages were also repaired: `torchaudio` (see below) and
`setuptools`, which had been upgraded to 84 - that release removes
`pkg_resources`, which the pinned `wandb==0.13.3` imports. Pinned back to
`setuptools<81` rather than breaking the wandb pin.
## Notes on this machine
* **`torchaudio` was broken.** The venv had `torchaudio 2.11.0+cu130` against
`torch 2.9.0+cu128`, which made every `transformers` import fail with
`Could not load this library: _torchaudio.abi3.so`. Fixed with
`uv pip install --no-deps torchaudio==2.9.0 --index-url https://download.pytorch.org/whl/cu128`.
* **`Qwen/Qwen3-VL-32B-Thinking`** must be available locally or via the Hugging Face cache unless you omit `--local-files-only`.
`--local-files-only` works offline.
* **Vision patch-embed fast path.** Qwen3-VL embeds patches with a `Conv3d`
whose kernel equals its stride, i.e. an ordinary matmul over flattened
patches. cuDNN picks a pathological algorithm for that shape on this GPU:
measured **14 s** per forward pass versus **0.15 ms** for the identical
matmul. `apply_fast_patch_embed()` folds the conv weights into an
`nn.functional.linear` call at load time - numerically identical up to bf16
accumulation order, and it takes the run from ~14 s/group to ~0.13 s/group.
Pass `--no-fast-patch-embed` to use the original `Conv3d`.
* **The `data/` directory shipped as git-LFS pointer files.** The real blobs
were present in the HF hub cache
(`datasets--Nadeem-Shoukath115113--cloth_folder`) and have been copied into
place, so `data/` is a complete dataset now.
## Dataset this was built against
| | |
|---|---|
| layout | LeRobot `v3.0` (aggregated parquet + aggregated mp4 per camera) |
| fps | 15 |
| episodes / frames | 9 / 3638 |
| robot | `bi_so_follower` (SO-101 bimanual), 12-D state and action |
| task | "Fold the T-shirt on the table" |
| cameras | `observation.images.left_top`, `observation.images.left_left_wrist`, `observation.images.right_right_wrist` |
| video codec | AV1, 480x640, decoded with PyAV / dav1d |
Both LeRobot v2.x and v3.0 layouts are handled, so `--dataset-root` can also
point at the v2.0 datasets under `~/.cache/huggingface/lerobot/`.
---
## Files
| file | role |
|---|---|
| `act_pri_common.py` | vocabulary, dataset introspection, grouping, per-frame expansion, parquet export |
| `label_act_pri_qwen.py` | step 1 - the Qwen labeling CLI |
| `annotate_act_pri_ui.py` | step 2 - the review web UI with boundary editing |
| `build_act_pri_clips.py` | step 3 - reviewed labels -> the act_pri clip dataset |
| `check_act_pri_dataset.py` | step 4 - verifies that dataset through the real training path |

Xet Storage Details

Size:
11.8 kB
·
Xet hash:
5753240a65828e17fbec4563d21df8a6e55b8025ed2219ff24a0d09c7cfcc757

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.