| --- |
| license: cc-by-nc-4.0 |
| base_model: Qwen/Qwen3-VL-4B-Instruct |
| tags: |
| - gaze-following |
| - gaze-target-estimation |
| - social-gaze |
| - vision-language-model |
| --- |
| |
| # OmniGF |
|
|
| Weights for **OmniGF: A Dual-Branch Vision-Language Framework for Unified Gaze |
| Following** (NeurIPS 2026 Spotlight, [paper](https://arxiv.org/abs/2605.26399)). |
| Code, training and evaluation instructions: |
| [github.com/cvlab-stonybrook/omnigf](https://github.com/cvlab-stonybrook/omnigf). |
|
|
| Each file holds only the weights OmniGF trains: the LoRA adapters (rank 16 on `q_proj` / |
| `v_proj`), the head encoder, the gaze heads and the added `<|gaze_pad|>` token |
| embedding. The Qwen3-VL-4B-Instruct backbone is downloaded from the Hub when the model is |
| built. |
|
|
| | File | Task | Paper | |
| |---|---|---| |
| | `omnigf_gazefollow.safetensors` | Gaze target estimation, GazeFollow | Table 1 | |
| | `omnigf_gazefollow_inout.safetensors` | GazeFollow with in/out loss; warm start for VAT, ChildPlay and VSGaze | | |
| | `omnigf_vat.safetensors` | Gaze target estimation, VideoAttentionTarget | Table 1 | |
| | `omnigf_childplay.safetensors` | Gaze target estimation, ChildPlay | Table 2 | |
| | `omnigf_semantic_gazefollow.safetensors` | Semantic gaze prediction, GazeFollow | Table 4 | |
| | `omnigf_social_vsgaze.safetensors` | Social gaze prediction, VSGaze | Table 6 | |
|
|
| `omnigf_social_vsgaze.safetensors` is a re-run of the Table 6 recipe, because the |
| original checkpoint was lost. On the VSGaze test set it gives Dist 0.099, AP_IO 0.992, |
| F1_LAH 0.816, F1_LAEO 0.756 and AP_SA 0.758, against 0.098 / 0.992 / 0.814 / 0.742 / |
| 0.787 in the paper. |
|
|
| ## Usage |
|
|
| From the root of the code repository: |
|
|
| ```bash |
| hf download StonyBrook-CVLab/omnigf --local-dir checkpoints |
| hf download Idiap/resnet18-gaze360 gaze360_resnet18.pt --local-dir checkpoints |
| bash scripts/test_gazefollow.sh |
| ``` |
|
|
| The launchers look for these file names under `./checkpoints`. Keep |
| `transformers==5.2.0`; see the code repository's README. |
|
|
| ## License |
|
|
| CC BY-NC 4.0. The head encoder is initialised from Idiap's |
| [ResNet-18 Gaze360 model](https://huggingface.co/Idiap/resnet18-gaze360), which is |
| licensed CC BY-NC 4.0. Qwen3-VL-4B-Instruct is licensed Apache-2.0. |
|
|
| ## Citation |
|
|
| ```bibtex |
| @inproceedings{miao2026omnigf, |
| title = {OmniGF: A Dual-Branch Vision-Language Framework for Unified Gaze Following}, |
| author = {Miao, Qiaomu and Wu, Haoyu and Xu, Jingyi and Hoai, Minh and Samaras, Dimitris}, |
| booktitle = {Advances in Neural Information Processing Systems (NeurIPS)}, |
| year = {2026} |
| } |
| ``` |
|
|
