Add model card

README.md

@@ -0,0 +1,94 @@
+---
+license: apache-2.0
+pipeline_tag: mask-generation
+library_name: sam2
+tags:
+  - visual-query-segmentation
+  - video-object-segmentation
+  - sam2
+  - vqs
+datasets:
+  - bf0820/VQS-4K
+---
+
+# VQ-SAM
+
+Baseline checkpoint for **visual query segmentation (VQS)** from the NeurIPS 2026
+Datasets & Benchmarks paper [*Towards Visual Query Segmentation in the
+Wild*](https://arxiv.org/pdf/2603.08898).
+
+Given an untrimmed video and a **visual query** — one image plus a mask of the target, taken
+from **outside** that video — VQ-SAM segments the object in every frame where it appears.
+It extends SAM 2 with a spatio-temporal transformer block and target / distractor memory
+evolved through a multi-stage framework with an adaptive memory generation (AMG) module.
+
+- 📄 Paper: [arXiv:2603.08898](https://arxiv.org/pdf/2603.08898)
+- 💻 Code: [github.com/VAIL-Fan/VQS](https://github.com/VAIL-Fan/VQS)
+- 📦 Dataset: [bf0820/VQS-4K](https://huggingface.co/datasets/bf0820/VQS-4K)
+
+## Results on VQS-4K test (822 videos)
+
+| mSTIoU | mSTIoU₅₀ | mSTIoU₇₅ | mTIoU | mTIoU₅₀ | mTIoU₇₅ | Rec | Succ | FPS |
+|---|---|---|---|---|---|---|---|---|
+| 26.0 | 23.8 | 12.7 | 29.6 | 26.8 | 14.4 | 43.6 | 42.1 | 29 |
+
+Best published result on this benchmark; the strongest prior method (SAM2Long) reaches
+18.6 mSTIoU. These numbers are reproducible from this checkpoint with the evaluation code in
+the repository.
+
+## Files
+
+| File | Description |
+|---|---|
+| `vqsam_hiera_bplus.pt` | VQ-SAM, SAM 2.1 Hiera-B+ backbone, trained on the VQS-4K train split (1.0 GB) |
+
+## Usage
+
+```bash
+git clone https://github.com/VAIL-Fan/VQS && cd VQS
+pip install -r requirements.txt && pip install -e .
+
+python - <<'PY'
+from huggingface_hub import hf_hub_download
+hf_hub_download("bf0820/VQ-SAM", "vqsam_hiera_bplus.pt", local_dir="checkpoints")
+PY
+
+python tools/test_video_segmentation.py \
+  --sam2_cfg sam2/configs/sam2.1_training/sam2.1_hiera_b+_DAM_AdaptiveGate_Jan28.yaml \
+  --sam2_checkpoint checkpoints/vqsam_hiera_bplus.pt \
+  --base_video_dir ./data/VQS-4K/images \
+  --input_mask_dir ./data/VQS-4K/masks \
+  --video_list_file ./splits/test_t-IoU.txt \
+  --output_mask_dir ./outputs/predictions \
+  --num_frames_per_segment 8
+```
+
+The inference segment length must match the training clip length (8 for this checkpoint).
+
+## Training
+
+SAM 2.1 Hiera-B+ initialisation, 768x768, 8 frames per clip (1 query + 7 video), batch size 1
+per GPU, on 4x RTX A6000. Config
+`sam2.1_hiera_b+_DAM_AdaptiveGate_Jan28.yaml` in the repository, with two-stage memory,
+distractor memory, quality-weighted RAM and adaptive gating enabled.
+
+## License
+
+Code and weights: Apache 2.0, inherited from [SAM 2](https://github.com/facebookresearch/sam2).
+The VQS-4K dataset is separately licensed CC BY-NC 4.0, research use only.
+
+## Citation
+
+```bibtex
+@inproceedings{fan2026vqs,
+  title     = {Towards Visual Query Segmentation in the Wild},
+  author    = {Fan, Bing and Li, Minghao and Zhang, Hanzhi and Dong, Shaohua and
+               Mareedu, Naga Prudhvi and Liu, Xiaoqiong and Shi, Weishi and
+               Feng, Yunhe and Huang, Yan and Fan, Heng},
+  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)
+               Datasets and Benchmarks Track},
+  year      = {2026},
+  eprint    = {2603.08898},
+  archivePrefix = {arXiv}
+}
+```