| --- |
| license: mit |
| library_name: pytorch |
| tags: |
| - gaze-prediction |
| - scanpath-prediction |
| - eye-tracking |
| - human-attention |
| - visual-search |
| - diffusion |
| --- |
| |
| # ST-DiffEye |
|
|
| Checkpoints for **ST-DiffEye: Diffusion-based Continuous Gaze Generation via Joint Scanpath-Trajectory Modeling**. |
|
|
| NeurIPS 2026 |
|
|
| [arXiv](https://arxiv.org/abs/2606.15486) · [Project Page](https://st-diffeye.github.io/) · [Code](https://github.com/briannlongzhao/ST-DiffEye) |
|
|
| TLDR: ST-DiffEye is a diffusion model that generates continuous eye trajectories and aligned scanpaths conditioned on a visual stimulus and an optional search target, using a per-point fixation index that ties each trajectory point to the human scanpath. |
|
|
| ## Checkpoints |
|
|
| | File | Setting | |
| |---|---| |
| | `COCOSearch18-TP.pt` | COCO-Search18, target present | |
| | `COCOSearch18-TA.pt` | COCO-Search18, target absent | |
| | `COCOFreeView.pt` | COCO-FreeView | |
| | `MIT1003.pt` | MIT1003 | |
|
|
| ## Usage |
|
|
| Set up the [code](https://github.com/briannlongzhao/ST-DiffEye) and download the checkpoints into `checkpoints/`: |
|
|
| ```bash |
| git clone https://github.com/briannlongzhao/ST-DiffEye.git |
| cd ST-DiffEye |
| conda env create -f environment.yml |
| conda activate stdiffeye |
| hf download briannlongzhao/ST-DiffEye --local-dir checkpoints |
| ``` |
|
|
| Generate trajectories and scanpaths for a single image. The demo needs `data/task_embeddings.npy` from the [data release](https://github.com/briannlongzhao/ST-DiffEye#data). |
|
|
| ```bash |
| # Visual search: target present / target absent |
| python src/demo.py --image path/to/image.jpg --task cocosearch18_tp --target clock |
| python src/demo.py --image path/to/image.jpg --task cocosearch18_ta --target "potted plant" |
| # Free viewing |
| python src/demo.py --image path/to/image.jpg --task cocofreeview |
| python src/demo.py --image path/to/image.jpg --task mit1003 |
| ``` |
|
|
| See the [code repository](https://github.com/briannlongzhao/ST-DiffEye) for data preparation, training and evaluation. |
|
|
| ## Citation |
|
|
| ```bibtex |
| @inproceedings{zhao2026stdiffeye, |
| title = {ST-DiffEye: Diffusion-based Continuous Gaze Generation via Joint Scanpath-Trajectory Modeling}, |
| author = {Zhao, Brian Nlong and Kara, Ozgur and Kim, Junho and Rehg, James M.}, |
| booktitle = {Advances in Neural Information Processing Systems (NeurIPS)}, |
| year = {2026} |
| } |
| ``` |
|
|
| ## Acknowledgments |
|
|
| This code builds on [ScanDiff](https://github.com/aimagelab/ScanDiff) and [DiffEye](https://github.com/rehg-lab/DiffEye). |
|
|
