---
license: cc-by-nc-4.0
base_model: Qwen/Qwen3-VL-4B-Instruct
tags:
  - gaze-following
  - gaze-target-estimation
  - social-gaze
  - vision-language-model
---

# OmniGF

Weights for **OmniGF: A Dual-Branch Vision-Language Framework for Unified Gaze
Following** (NeurIPS 2026 Spotlight, [paper](https://arxiv.org/abs/2605.26399)).
Code, training and evaluation instructions:
[github.com/cvlab-stonybrook/omnigf](https://github.com/cvlab-stonybrook/omnigf).

Each file holds only the weights OmniGF trains: the LoRA adapters (rank 16 on `q_proj` /
`v_proj`), the head encoder, the gaze heads and the added `<|gaze_pad|>` token
embedding. The Qwen3-VL-4B-Instruct backbone is downloaded from the Hub when the model is
built.

| File | Task | Paper |
|---|---|---|
| `omnigf_gazefollow.safetensors` | Gaze target estimation, GazeFollow | Table 1 |
| `omnigf_gazefollow_inout.safetensors` | GazeFollow with in/out loss; warm start for VAT, ChildPlay and VSGaze | |
| `omnigf_vat.safetensors` | Gaze target estimation, VideoAttentionTarget | Table 1 |
| `omnigf_childplay.safetensors` | Gaze target estimation, ChildPlay | Table 2 |
| `omnigf_semantic_gazefollow.safetensors` | Semantic gaze prediction, GazeFollow | Table 4 |
| `omnigf_social_vsgaze.safetensors` | Social gaze prediction, VSGaze | Table 6 |

`omnigf_social_vsgaze.safetensors` is a re-run of the Table 6 recipe, because the
original checkpoint was lost. On the VSGaze test set it gives Dist 0.099, AP_IO 0.992,
F1_LAH 0.816, F1_LAEO 0.756 and AP_SA 0.758, against 0.098 / 0.992 / 0.814 / 0.742 /
0.787 in the paper.

## Usage

From the root of the code repository:

```bash
hf download StonyBrook-CVLab/omnigf --local-dir checkpoints
hf download Idiap/resnet18-gaze360 gaze360_resnet18.pt --local-dir checkpoints
bash scripts/test_gazefollow.sh
```

The launchers look for these file names under `./checkpoints`. Keep
`transformers==5.2.0`; see the code repository's README.

## License

CC BY-NC 4.0. The head encoder is initialised from Idiap's
[ResNet-18 Gaze360 model](https://huggingface.co/Idiap/resnet18-gaze360), which is
licensed CC BY-NC 4.0. Qwen3-VL-4B-Instruct is licensed Apache-2.0.

## Citation

```bibtex
@inproceedings{miao2026omnigf,
  title     = {OmniGF: A Dual-Branch Vision-Language Framework for Unified Gaze Following},
  author    = {Miao, Qiaomu and Wu, Haoyu and Xu, Jingyi and Hoai, Minh and Samaras, Dimitris},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year      = {2026}
}
```