ST-DiffEye

Checkpoints for ST-DiffEye: Diffusion-based Continuous Gaze Generation via Joint Scanpath-Trajectory Modeling.

NeurIPS 2026

arXiv · Project Page · Code

TLDR: ST-DiffEye is a diffusion model that generates continuous eye trajectories and aligned scanpaths conditioned on a visual stimulus and an optional search target, using a per-point fixation index that ties each trajectory point to the human scanpath.

Checkpoints

File	Setting
COCOSearch18-TP.pt	COCO-Search18, target present
COCOSearch18-TA.pt	COCO-Search18, target absent
COCOFreeView.pt	COCO-FreeView
MIT1003.pt	MIT1003

Usage

Set up the code and download the checkpoints into checkpoints/:

git clone https://github.com/briannlongzhao/ST-DiffEye.git
cd ST-DiffEye
conda env create -f environment.yml
conda activate stdiffeye
hf download briannlongzhao/ST-DiffEye --local-dir checkpoints

Generate trajectories and scanpaths for a single image. The demo needs data/task_embeddings.npy from the data release.

# Visual search: target present / target absent
python src/demo.py --image path/to/image.jpg --task cocosearch18_tp --target clock
python src/demo.py --image path/to/image.jpg --task cocosearch18_ta --target "potted plant"
# Free viewing
python src/demo.py --image path/to/image.jpg --task cocofreeview
python src/demo.py --image path/to/image.jpg --task mit1003

See the code repository for data preparation, training and evaluation.

Citation

@inproceedings{zhao2026stdiffeye,
  title     = {ST-DiffEye: Diffusion-based Continuous Gaze Generation via Joint Scanpath-Trajectory Modeling},
  author    = {Zhao, Brian Nlong and Kara, Ozgur and Kim, Junho and Rehg, James M.},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year      = {2026}
}

Acknowledgments

This code builds on ScanDiff and DiffEye.