metadata
license: mit
library_name: pytorch
tags:
- gaze-prediction
- scanpath-prediction
- eye-tracking
- human-attention
- visual-search
- diffusion
ST-DiffEye
Checkpoints for ST-DiffEye: Diffusion-based Continuous Gaze Generation via Joint Scanpath-Trajectory Modeling.
NeurIPS 2026
arXiv · Project Page · Code
TLDR: ST-DiffEye is a diffusion model that generates continuous eye trajectories and aligned scanpaths conditioned on a visual stimulus and an optional search target, using a per-point fixation index that ties each trajectory point to the human scanpath.
Checkpoints
| File | Setting |
|---|---|
COCOSearch18-TP.pt |
COCO-Search18, target present |
COCOSearch18-TA.pt |
COCO-Search18, target absent |
COCOFreeView.pt |
COCO-FreeView |
MIT1003.pt |
MIT1003 |
Usage
Set up the code and download the checkpoints into checkpoints/:
git clone https://github.com/briannlongzhao/ST-DiffEye.git
cd ST-DiffEye
conda env create -f environment.yml
conda activate stdiffeye
hf download briannlongzhao/ST-DiffEye --local-dir checkpoints
Generate trajectories and scanpaths for a single image. The demo needs data/task_embeddings.npy from the data release.
# Visual search: target present / target absent
python src/demo.py --image path/to/image.jpg --task cocosearch18_tp --target clock
python src/demo.py --image path/to/image.jpg --task cocosearch18_ta --target "potted plant"
# Free viewing
python src/demo.py --image path/to/image.jpg --task cocofreeview
python src/demo.py --image path/to/image.jpg --task mit1003
See the code repository for data preparation, training and evaluation.
Citation
@inproceedings{zhao2026stdiffeye,
title = {ST-DiffEye: Diffusion-based Continuous Gaze Generation via Joint Scanpath-Trajectory Modeling},
author = {Zhao, Brian Nlong and Kara, Ozgur and Kim, Junho and Rehg, James M.},
booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
year = {2026}
}