| --- |
| license: cc-by-nc-4.0 |
| tags: |
| - hand-object-interaction |
| - instance-segmentation |
| - sam3 |
| - hoist-former |
| library_name: pytorch |
| --- |
| |
| # SAM3-HOIST — Stage 1 held-object head (weight deltas) |
|
|
| Held-object identification and segmentation on **HOIST** (HOIST-Former, CVPR 2024), |
| built by fine-tuning **SAM3-I** with a new held-object interaction head. |
|
|
| > **These are deltas, not full checkpoints.** Each file carries only the 121 tensors that |
| > training changed (~14 MB). The frozen SAM3 trunk — 3,337 tensors, 7.3 GB, byte-identical |
| > to the base — is **not redistributed here**, because those weights are gated by Meta. |
| > Bring your own SAM3-I base checkpoint and run `merge_delta.py`; the result is |
| > bit-identical to the original 8.1 GB checkpoint. |
| |
| ## What is in this repo |
| |
| | Path | What it is | |
| |---|---| |
| | `stage1/final.delta.pt` | The 8-epoch Stage-1 fine-tune. This is **row C**, frame-level AP50 **46.74** | |
| | `stage1/epoch_1.delta.pt` … `epoch_8.delta.pt` | Per-epoch checkpoints of the same run | |
| | `merge_delta.py`, `extract_delta.py` | Reconstitute a full checkpoint from base + delta | |
| | `SHA256SUMS` | Checksums for the files in `stage1/` | |
| |
| `final.delta.pt` and `epoch_8.delta.pt` carry **identical weights** — the run finished at |
| epoch 8, so the "final" checkpoint is that epoch. Their SHA256 differ only because each |
| records which file it came from. Download either one. |
|
|
| The per-epoch files exist for a specific open question — see *The 22-point anomaly* below. |
|
|
| ## Usage |
|
|
| You need the SAM3-I base checkpoint first (gated; request access from the SAM3-I authors — |
| see [`docs/02-checkpoints.md`](https://github.com/ZaMinVo/SAM3-HOIST) in the code repo). |
|
|
| ```bash |
| pip install torch |
| huggingface-cli download ZaMinVo/SAM3-HOIST --local-dir sam3-hoist |
| |
| python sam3-hoist/merge_delta.py \ |
| --base /path/to/SAM3-I/checkpoint.pt \ |
| --delta sam3-hoist/stage1/final.delta.pt \ |
| --out ./hoist_held_object_final.pt |
| ``` |
|
|
| `merge_delta.py` refuses to run if the base does not match the fingerprint the delta was |
| built against — merging onto the wrong base produces a model that loads cleanly and scores |
| nonsense, which is the failure worth guarding. |
|
|
| Then evaluate with the code repo's frame dumper: |
|
|
| ```bash |
| python tools/eval_external/dump_sam3i_hoist_frames.py \ |
| --checkpoint_path ./hoist_held_object_final.pt \ |
| --stage interaction --resolution 1008 \ |
| --json /path/to/HOIST/test.json \ |
| --image_root /path/to/HOIST/test/JPEGImages \ |
| --out rowC.jsonl |
| python tools/eval_external/score_hoist_frames.py --dump rowC.jsonl --out rowC.json |
| ``` |
|
|
| ## Results |
|
|
| Frame-level, HOIST test: 321 videos / 5,973 frames / 6,784 GT instances, 1008 px, |
| `max_dets=100`. |
|
|
| | | AP50 | AP | AP75 | AR_1 | AR_10 | AR_100 | |
| |---|---:|---:|---:|---:|---:|---:| |
| | Stage 1 (this model) | **46.74** | 31.46 | 35.78 | 37.39 | 61.98 | 71.20 | |
| |
| Recall@k, IoU 0.5, threshold-free: |
| |
| | k | 1 | 5 | 10 | 20 | 50 | 100 | |
| |---|---:|---:|---:|---:|---:|---:| |
| | ceiling recall | 52.11 | 76.21 | 83.49 | 88.81 | 93.29 | 95.36 | |
| |
| **95.36% of held objects are inside the 100-candidate set, but only 52.11% are ranked |
| first** — a 43.25-point gap that a perfect re-ranker could close without touching a single |
| mask. A Stage-2 re-ranker was trained and is **a clean negative**: four different ranking |
| losses span 0.013 AP50, and the best moves ceiling@1 by +0.06. If you want to improve this |
| model, the evidence says give the ranker **different inputs, not a different objective**. |
| |
| ## Reading these numbers honestly |
| |
| - **`AP50`, never `AP`.** `AP` is COCO-averaged 0.50:0.95 and roughly half the value. |
| - **Frame-level, track identity ignored.** These are **not** the paper's spatio-temporal |
| AP50 and do not belong in a Table 2 column. For reference, HOIST-Former scores 72.93 |
| under this same frame-level metric, so this model is well behind it. |
| - **Resolution changes the result.** Everything here is at 1008 px; state the resolution |
| next to any number you quote. |
| - **The 22-point anomaly.** An ablation model trained on 8,000 images for 4 epochs at batch |
| 8 beat this run by **22.35 AP50** on identical frames, despite this run using 74,527 |
| images for 8 epochs at effective batch 64. Treat **46.74 as the floor of one recipe, not |
| the method's ceiling.** The per-epoch deltas in `stage1/` are published so this can be |
| diagnosed without retraining: score epochs 1–8 and look for an early peak, which would |
| point at the learning rate. |
| |
| ## Architecture |
| |
| The whole SAM3 vision trunk is frozen. The head cross-attends learned queries over the last |
| three FPN levels to produce one hand prompt and eight interaction prompts — matching HOIST's |
| one union hand mask plus up to eight contact regions — fused with the `adapter1` text |
| feature, then decoded by the frozen SAM3 decoder. |
| |
| Of the 649 tensors the trainer marks trainable, exactly **121 actually change** over 8 |
| epochs: 73 new head tensors (14.0 MB) and 48 adapter tensors (0.2 MB). The remaining 528 |
| adapter tensors come out byte-identical to where they started. |
| |
| ## Licence |
| |
| The deltas are released under CC-BY-NC-4.0. They are **not usable alone** — they require the |
| SAM3-I base checkpoint, which carries its own licence and access terms that you must satisfy |
| separately. Nothing in this repository redistributes Meta's SAM3 weights. |
| |
| ## Citation |
| |
| If you use this, please cite HOIST-Former (CVPR 2024) and SAM3, which this work builds on. |
| |
