Hugging Face's logo

DiLi-Lab
/
Eyettention

Model card Files
xet
 Community
1
Eyettention
File size: 7,060 Bytes
b100d33
 
 
 
 
c689a69
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
b100d33
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
---
license: mit
---


# Eyettention: An Attention-based Dual-Sequence Model for Predicting Human Scanpaths during Reading

In this paper, we develop Eyettention, the first dual-sequence model that simultaneously processes the sequence of words and the chronological sequence of fixations. The alignment of the two sequences is achieved by a cross-sequence attention mechanism. We show that Eyettention outperforms state-of-the-art models in predicting scanpaths. We provide an extensive within- and across-data set evaluation on different languages. An ablation study and qualitative analysis support an in-depth understanding of the model's behavior.

The overview above describes the [original authors' work](https://arxiv.org/abs/2304.10784). This project uses the [original Eyettention implementation](https://github.com/aeye-lab/Eyettention), reorganized as the `Eyettention` package, with raw-text inference, scanpath-prefix replay, an endpoint handler, and a Gradio interface.

## Setup

For this checkout, run setup and inference commands from the project root (the directory containing `Eyettention/`). Install dependencies:

```bash

python -m pip install -r Eyettention/requirements.txt

```

The requirements retain historical PyTorch and Transformers pins that may need adjustment for your Python version and platform. The local preprocessing code also requires `LAC` and a compatible PaddlePaddle installation; BSC Excel loading requires `openpyxl`. Install `gradio` if using the web interface. Use a separate environment from ScanDL2 because their dependency versions differ.

## Dataset

For the CELER dataset, follow the instructions at [the CELER repository](https://github.com/berzak/celer). In order to run experiments, place the downloaded dataset in `Eyettention/Data/celer/`. The local loaders expect `data_v2.0/sent_fix.tsv`, `data_v2.0/sent_ia.tsv`, and `metadata.tsv` beneath that directory.

The Chinese [Beijing Sentence Corpus (BSC)](https://osf.io/vr3k8/) belongs in `Eyettention/Data/beijing-sentence-corpus/`, with `BSC.Word.Info.v2.xlsx` and `BSC.EMD/BSC.EMD.txt`. See [utils.py](utils.py) for the dataset loaders.

Raw-text inference uses the trained checkpoints and normalization files already stored under `results/` and `Data/`; it does not require the full training corpora.

## Run Experiments

The experiment scripts use paths relative to `Eyettention/`. Starting at the project root:

```bash

cd Eyettention

mkdir -p results/BSC results/CELER

```

Keep the project root on `PYTHONPATH` when running the package modules.

### For Chinese BSC dataset

```bash

PYTHONPATH=.. python -m Eyettention.main_BSC --test_mode text --gpu 0

PYTHONPATH=.. python -m Eyettention.main_BSC --test_mode subject --gpu 0

PYTHONPATH=.. python -m Eyettention.main_BSC_NRS_setting --gpu 0

PYTHONPATH=.. python -m Eyettention.main_BSC_reader_identifier --gpu 0

```

### For English CELER dataset

```bash

PYTHONPATH=.. python -m Eyettention.main_celer --test_mode text --gpu 0

PYTHONPATH=.. python -m Eyettention.main_celer --test_mode subject --gpu 0

PYTHONPATH=.. python -m Eyettention.main_celer_NRS_setting --gpu 0

PYTHONPATH=.. python -m Eyettention.main_celer_reader_identifier --gpu 0

```

`text` evaluates new sentences; `subject` evaluates new readers. Select an available GPU with `--gpu`. Several historical experiment scripts call CUDA directly, so use a CUDA-enabled environment for these commands. Review output paths before running; `--save_data_folder` selects the results directory.

## Raw-text inference

Run the following from the project root:

```python

from Eyettention import EyettentionRawTextInference



model = EyettentionRawTextInference(

    checkpoint_path="Eyettention/results/CELER/Eyettention_english.pth",

    dataset="celer",

    device="cpu",

)

scanpath, density = model.generate_from_english_text(

    "The quick brown fox jumps.",

    max_pred_len=20,

    # Optional: replay observed fixation positions before sampling.

    # previous_scanpath=[0, 1, 2],

)

print(scanpath[0].tolist())

```

For Chinese, use `dataset="BSC"`, checkpoint `Eyettention/results/BSC/Eyettention_chinese.pth`, and `generate_from_chinese_text(...)`. Dataset names are case-sensitive. Use `device="cuda"` for GPU inference. BERT assets must be downloadable from Hugging Face or cached locally.

Keep `Data/feature_norm_celer.pickle` and `Data/feature_norm_BSC.pickle` alongside the package. The output contains fixation indices and per-step probability distributions, not fixation durations. Index `0` denotes CLS, positions `1..N` refer to words or Chinese characters, and `N+1` denotes SEP; stop at the first SEP when interpreting fixations. Use short sentences within the checkpoint's configured input limits. Prefix replay extends the generation loop, so `max_pred_len` is not a strict total-length cap when a prefix is provided.

## Endpoint handler

The added [handler.py](handler.py) wraps inference in a dictionary-based interface:

```python

from Eyettention.handler import EndpointHandler



handler = EndpointHandler()

result = handler({

    "inputs": "The quick brown fox jumps.",

    "parameters": {

        "dataset": "celer",

        "device": "cpu",

        "max_pred_len": 20,

    },

})

```

It returns `scanpath` and `density_steps`. Optional parameters include `checkpoint_path` and `previous_scanpath`. The handler defaults to BSC on CPU and does not itself start an HTTP server.

## Gradio interface

The added [app.py](app.py) provides a web interface. From the project root, point it to the local checkpoints and launch:

```bash

export EYETTENTION_BSC_CHECKPOINT="Eyettention/results/BSC/Eyettention_chinese.pth"

export EYETTENTION_CELER_CHECKPOINT="Eyettention/results/CELER/Eyettention_english.pth"

python -m Eyettention.app

```

Open `http://localhost:7860`, choose the dataset, enter text, and click **Run**. The interface also supports an observed scanpath prefix. It selects CUDA when available, otherwise CPU. Set `PORT` to change the default port; the app binds to `0.0.0.0`.

## Cite our work

If you use this code for your research, please consider citing the original authors' paper:

```bibtex

@article{deng2023eyettention,

  title={Eyettention: {A}n Attention-based Dual-Sequence Model for Predicting Human Scanpaths during Reading},

  author={Deng, Shuwen and Reich, David R and Prasse, Paul and Haller, Patrick and Scheffer, Tobias and J{\"a}ger, Lena A},

  journal={Proceedings of the {ACM} on Human-Computer Interaction},

  volume={7},

  number={ETRA},

  pages={1--24},

  year={2023},

  publisher={ACM New York, NY, USA}

}

```

## License

The original code is provided under the [MIT License](LICENSE.txt), copyright © 2023 AEye. Include the copyright and permission notice when redistributing copies or substantial portions of the software. See the license file for the full terms and warranty disclaimer. Datasets and dependencies retain their own licenses.