---
license: mit
library_name: pytorch
tags:
  - gaze-prediction
  - scanpath-prediction
  - eye-tracking
  - human-attention
  - visual-search
  - diffusion
---

# ST-DiffEye

Checkpoints for **ST-DiffEye: Diffusion-based Continuous Gaze Generation via Joint Scanpath-Trajectory Modeling**.

NeurIPS 2026

[arXiv](https://arxiv.org/abs/2606.15486) · [Project Page](https://st-diffeye.github.io/) · [Code](https://github.com/briannlongzhao/ST-DiffEye)

TLDR: ST-DiffEye is a diffusion model that generates continuous eye trajectories and aligned scanpaths conditioned on a visual stimulus and an optional search target, using a per-point fixation index that ties each trajectory point to the human scanpath.

## Checkpoints

| File | Setting |
|---|---|
| `COCOSearch18-TP.pt` | COCO-Search18, target present |
| `COCOSearch18-TA.pt` | COCO-Search18, target absent |
| `COCOFreeView.pt` | COCO-FreeView |
| `MIT1003.pt` | MIT1003 |

## Usage

Set up the [code](https://github.com/briannlongzhao/ST-DiffEye) and download the checkpoints into `checkpoints/`:

```bash
git clone https://github.com/briannlongzhao/ST-DiffEye.git
cd ST-DiffEye
conda env create -f environment.yml
conda activate stdiffeye
hf download briannlongzhao/ST-DiffEye --local-dir checkpoints
```

Generate trajectories and scanpaths for a single image. The demo needs `data/task_embeddings.npy` from the [data release](https://github.com/briannlongzhao/ST-DiffEye#data).

```bash
# Visual search: target present / target absent
python src/demo.py --image path/to/image.jpg --task cocosearch18_tp --target clock
python src/demo.py --image path/to/image.jpg --task cocosearch18_ta --target "potted plant"
# Free viewing
python src/demo.py --image path/to/image.jpg --task cocofreeview
python src/demo.py --image path/to/image.jpg --task mit1003
```

See the [code repository](https://github.com/briannlongzhao/ST-DiffEye) for data preparation, training and evaluation.

## Citation

```bibtex
@inproceedings{zhao2026stdiffeye,
  title     = {ST-DiffEye: Diffusion-based Continuous Gaze Generation via Joint Scanpath-Trajectory Modeling},
  author    = {Zhao, Brian Nlong and Kara, Ozgur and Kim, Junho and Rehg, James M.},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year      = {2026}
}
```

## Acknowledgments

This code builds on [ScanDiff](https://github.com/aimagelab/ScanDiff) and [DiffEye](https://github.com/rehg-lab/DiffEye).