license: cc-by-nc-4.0
base_model: Qwen/Qwen3-VL-4B-Instruct
tags:
- gaze-following
- gaze-target-estimation
- social-gaze
- vision-language-model
OmniGF
Weights for OmniGF: A Dual-Branch Vision-Language Framework for Unified Gaze Following (NeurIPS 2026 Spotlight, paper). Code, training and evaluation instructions: github.com/cvlab-stonybrook/omnigf.
Each file holds only the weights OmniGF trains: the LoRA adapters (rank 16 on q_proj /
v_proj), the head encoder, the gaze heads and the added <|gaze_pad|> token
embedding. The Qwen3-VL-4B-Instruct backbone is downloaded from the Hub when the model is
built.
| File | Task | Paper |
|---|---|---|
omnigf_gazefollow.safetensors |
Gaze target estimation, GazeFollow | Table 1 |
omnigf_gazefollow_inout.safetensors |
GazeFollow with in/out loss; warm start for VAT, ChildPlay and VSGaze | |
omnigf_vat.safetensors |
Gaze target estimation, VideoAttentionTarget | Table 1 |
omnigf_childplay.safetensors |
Gaze target estimation, ChildPlay | Table 2 |
omnigf_semantic_gazefollow.safetensors |
Semantic gaze prediction, GazeFollow | Table 4 |
omnigf_social_vsgaze.safetensors |
Social gaze prediction, VSGaze | Table 6 |
omnigf_social_vsgaze.safetensors is a re-run of the Table 6 recipe, because the
original checkpoint was lost. On the VSGaze test set it gives Dist 0.099, AP_IO 0.992,
F1_LAH 0.816, F1_LAEO 0.756 and AP_SA 0.758, against 0.098 / 0.992 / 0.814 / 0.742 /
0.787 in the paper.
Usage
From the root of the code repository:
hf download StonyBrook-CVLab/omnigf --local-dir checkpoints
hf download Idiap/resnet18-gaze360 gaze360_resnet18.pt --local-dir checkpoints
bash scripts/test_gazefollow.sh
The launchers look for these file names under ./checkpoints. Keep
transformers==5.2.0; see the code repository's README.
License
CC BY-NC 4.0. The head encoder is initialised from Idiap's ResNet-18 Gaze360 model, which is licensed CC BY-NC 4.0. Qwen3-VL-4B-Instruct is licensed Apache-2.0.
Citation
@inproceedings{miao2026omnigf,
title = {OmniGF: A Dual-Branch Vision-Language Framework for Unified Gaze Following},
author = {Miao, Qiaomu and Wu, Haoyu and Xu, Jingyi and Hoai, Minh and Samaras, Dimitris},
booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
year = {2026}
}