README.md

---
license: cc-by-nc-4.0
tags:
  - hand-object-interaction
  - instance-segmentation
  - sam3
  - hoist-former
library_name: pytorch
---

# SAM3-HOIST — Stage 1 held-object head (weight deltas)

Held-object identification and segmentation on **HOIST** (HOIST-Former, CVPR 2024),
built by fine-tuning **SAM3-I** with a new held-object interaction head.

> **These are deltas, not full checkpoints.** Each file carries only the 121 tensors that
> training changed (~14 MB). The frozen SAM3 trunk — 3,337 tensors, 7.3 GB, byte-identical
> to the base — is **not redistributed here**, because those weights are gated by Meta.
> Bring your own SAM3-I base checkpoint and run `merge_delta.py`; the result is
> bit-identical to the original 8.1 GB checkpoint.

## What is in this repo

| Path | What it is |
|---|---|
| `stage1/final.delta.pt` | The 8-epoch Stage-1 fine-tune. This is **row C**, frame-level AP50 **46.74** |
| `stage1/epoch_1.delta.pt` … `epoch_8.delta.pt` | Per-epoch checkpoints of the same run |
| `merge_delta.py`, `extract_delta.py` | Reconstitute a full checkpoint from base + delta |
| `SHA256SUMS` | Checksums for the files in `stage1/` |

`final.delta.pt` and `epoch_8.delta.pt` carry **identical weights** — the run finished at
epoch 8, so the "final" checkpoint is that epoch. Their SHA256 differ only because each
records which file it came from. Download either one.

The per-epoch files exist for a specific open question — see *The 22-point anomaly* below.

## Usage

You need the SAM3-I base checkpoint first (gated; request access from the SAM3-I authors —
see [`docs/02-checkpoints.md`](https://github.com/ZaMinVo/SAM3-HOIST) in the code repo).

```bash
pip install torch
huggingface-cli download ZaMinVo/SAM3-HOIST --local-dir sam3-hoist

python sam3-hoist/merge_delta.py \
  --base  /path/to/SAM3-I/checkpoint.pt \
  --delta sam3-hoist/stage1/final.delta.pt \
  --out   ./hoist_held_object_final.pt
```

`merge_delta.py` refuses to run if the base does not match the fingerprint the delta was
built against — merging onto the wrong base produces a model that loads cleanly and scores
nonsense, which is the failure worth guarding.

Then evaluate with the code repo's frame dumper:

```bash
python tools/eval_external/dump_sam3i_hoist_frames.py \
  --checkpoint_path ./hoist_held_object_final.pt \
  --stage interaction --resolution 1008 \
  --json  /path/to/HOIST/test.json \
  --image_root /path/to/HOIST/test/JPEGImages \
  --out rowC.jsonl
python tools/eval_external/score_hoist_frames.py --dump rowC.jsonl --out rowC.json
```

## Results

Frame-level, HOIST test: 321 videos / 5,973 frames / 6,784 GT instances, 1008 px,
`max_dets=100`.

| | AP50 | AP | AP75 | AR_1 | AR_10 | AR_100 |
|---|---:|---:|---:|---:|---:|---:|
| Stage 1 (this model) | **46.74** | 31.46 | 35.78 | 37.39 | 61.98 | 71.20 |

Recall@k, IoU 0.5, threshold-free:

| k | 1 | 5 | 10 | 20 | 50 | 100 |
|---|---:|---:|---:|---:|---:|---:|
| ceiling recall | 52.11 | 76.21 | 83.49 | 88.81 | 93.29 | 95.36 |

**95.36% of held objects are inside the 100-candidate set, but only 52.11% are ranked
first** — a 43.25-point gap that a perfect re-ranker could close without touching a single
mask. A Stage-2 re-ranker was trained and is **a clean negative**: four different ranking
losses span 0.013 AP50, and the best moves ceiling@1 by +0.06. If you want to improve this
model, the evidence says give the ranker **different inputs, not a different objective**.

## Reading these numbers honestly

- **`AP50`, never `AP`.** `AP` is COCO-averaged 0.50:0.95 and roughly half the value.
- **Frame-level, track identity ignored.** These are **not** the paper's spatio-temporal
  AP50 and do not belong in a Table 2 column. For reference, HOIST-Former scores 72.93
  under this same frame-level metric, so this model is well behind it.
- **Resolution changes the result.** Everything here is at 1008 px; state the resolution
  next to any number you quote.
- **The 22-point anomaly.** An ablation model trained on 8,000 images for 4 epochs at batch
  8 beat this run by **22.35 AP50** on identical frames, despite this run using 74,527
  images for 8 epochs at effective batch 64. Treat **46.74 as the floor of one recipe, not
  the method's ceiling.** The per-epoch deltas in `stage1/` are published so this can be
  diagnosed without retraining: score epochs 1–8 and look for an early peak, which would
  point at the learning rate.

## Architecture

The whole SAM3 vision trunk is frozen. The head cross-attends learned queries over the last
three FPN levels to produce one hand prompt and eight interaction prompts — matching HOIST's
one union hand mask plus up to eight contact regions — fused with the `adapter1` text
feature, then decoded by the frozen SAM3 decoder.

Of the 649 tensors the trainer marks trainable, exactly **121 actually change** over 8
epochs: 73 new head tensors (14.0 MB) and 48 adapter tensors (0.2 MB). The remaining 528
adapter tensors come out byte-identical to where they started.

## Licence

The deltas are released under CC-BY-NC-4.0. They are **not usable alone** — they require the
SAM3-I base checkpoint, which carries its own licence and access terms that you must satisfy
separately. Nothing in this repository redistributes Meta's SAM3 weights.

## Citation

If you use this, please cite HOIST-Former (CVPR 2024) and SAM3, which this work builds on.