encoder-things-eeg2

These models predict the EEG response to an image as recorded in THINGS-EEG2: 63 channels from 0 to 0.79 s after image onset, sampled at 100 Hz, in microvolts. By default the prediction is the average over the dataset's ten participants, the best guess for someone who was not in the dataset; each participant's own prediction is available too. Four variants trade accuracy for speed. All were fitted on the 16,540 THINGS-EEG2 training images and scored on its 200 test images, whose concepts never appear in training.

Quick start

pip install torch transformers pillow

from PIL import Image
from transformers import AutoModel

model = AutoModel.from_pretrained(
    "epfl-neuroai/encoder-things-eeg2", revision="v1", trust_remote_code=True
).to("cuda")

images = [Image.open("dog.jpg"), Image.open("teapot.jpg")]
eeg = model(images)  # (2, 63, 80): average over participants
eeg_all = model(images, subjects="all")  # (2, 10, 63, 80)
eeg_one = model(images, subjects=["sub-03"])  # (2, 1, 63, 80)

model.subjects  # "sub-01" ... "sub-10"
model.channels  # channel names, in output order
model.times  # 0.0, 0.01, ..., 0.79 seconds after image onset

This loads the default variant, efficient. Add variant="tiny", "cnn" or "accurate" for another; only the selected variant's weights are downloaded. The model takes PIL images and returns a float32 tensor; batch_size (default 64) sets how many images go through the GPU at once. For efficient, tiny and cnn, torch.autocast("cuda", torch.float16) speeds up the GPU part 1.5 to 4 times and changes outputs by at most 0.3% of their standard deviation.

trust_remote_code=True runs this repository's Python files on your machine: modeling.py, the model in plain PyTorch, and two short transformers wrappers. Review them and pin revision. Without transformers (pip install torch pillow safetensors huggingface-hub numpy):

import importlib.util
from huggingface_hub import snapshot_download

code = snapshot_download("epfl-neuroai/encoder-things-eeg2", revision="v1", allow_patterns=["*.py"])
spec = importlib.util.spec_from_file_location("encoder_things_eeg2", f"{code}/modeling.py")
modeling = importlib.util.module_from_spec(spec)
spec.loader.exec_module(modeling)
model = modeling.load("epfl-neuroai/encoder-things-eeg2", revision="v1", device="cuda")

Variants

variant	image networks	test r: all channels	occipital and parietal	new participant	download	images/s	GPU memory
`efficient` (default)	ResNet-50 + ViT-S	0.241	0.345	0.168	1.3 GB	68	2.0 GiB
`tiny`	ResNet-18 + ViT-S	0.235	0.335	0.164	0.25 GB	73	0.7 GiB
`cnn`	ResNet-152	0.245	0.350	0.171	1.9 GB	63	2.7 GiB
`accurate`	ResNet-152 + Qwen3-Omni vision	0.261	0.373	0.184	7.5 GB + 4.0 GB from Qwen	15	9.0 GiB

efficient is close to the best scores and fast. tiny is a fifth of the download; accurate scores best at about a fifth of the speed and downloads the Qwen3-Omni vision weights from Qwen's repository at a pinned revision. Speed is end to end from PIL images on one NVIDIA A100 (batch 64, 16 for accurate), so the CPU resizing sets the pace for the smaller variants.

Scores. Test r is the Pearson correlation between predicted and recorded responses across the 200 test images, per channel and time point, averaged over the points whose noise ceiling exceeds 0.1 (mean ceiling 0.47, or 0.59 over occipital and parietal channels), then over participants. Each participant is scored with their own readout, or, for "new participant", with the average of the other nine. 95% intervals are about ±0.03. The leaderboard best of the THINGS-EEG2 benchmark is 0.220 (all channels) and 0.324 (occipital and parietal).

Why the score is r, not R²

Most of the EEG signal, recorded or predicted, is the same for every image. Every image evokes roughly the same event-related potential (ERP). With a new image every 200 ms, each 0.8 s epoch also holds the responses to its neighbours, a strong 5 Hz rhythm. And the signal drifts slowly within a run, so a slope remains in each epoch after baseline correction. None of this depends on the image, so the score asks a narrower question: at each channel and time point, do the predictions rise and fall across images with the recordings? Pearson r across images measures this and ignores offset and scale.

R² also needs the offset and scale to be right. Even with both perfect, r = 0.24 would explain only about 6% of the variance across images. And because the shared ERP is many times larger than the image-dependent part, a small mismatch in its amplitude or latency, typical between people, can be as large as the whole spread across images and push R² below zero. For efficient, R² is -0.005 with each participant's own readout and -1.27 for a new participant, while r is 0.241 and 0.168. A positive r with a negative R² does not mean the model is broken; the next section shows how to remove the shared part before comparing amplitudes.

Using the predictions with your own data

Matching your recordings

The output follows the THINGS-EEG2 benchmark of Gokce et al. (2026): the trial-averaged response to an image shown for 100 ms in a rapid sequence of five images per second, on the recording reference (Fz, which is not among the 63 outputs), in microvolts. To compare it with your recordings:

Cut epochs from -0.2 to 0.8 s around image onset, subtract the mean of the 200 ms before onset, resample to 100 Hz, keep 0 to 0.79 s (80 samples) and average each image's repetitions. If your amplifier records from DC, high-pass the continuous recording at 0.1 Hz first, as THINGS-EEG2 did; on the datasets below this changed scores by less than 0.003.
Keep the channels you share with model.channels, in the same order, and re-reference both the predictions and your data to their average. To include Fz, append a channel of zeros to the predictions first.
Check for a constant delay. Consumer headsets and other presentation setups often add tens of milliseconds between the trigger and the image.

import numpy as np

predicted = model(images).cpu().numpy()  # (images, 63, 80)
# recorded: your (images, channels, 80) trial averages; names: their channel names
shared = [name for name in model.channels if name in names]
p = predicted[:, [model.channels.index(name) for name in shared]]
q = recorded[:, [names.index(name) for name in shared]]
p, q = p - p.mean(1, keepdims=True), q - q.mean(1, keepdims=True)  # average reference
p, q = p - p.mean(0), q - q.mean(0)
r = (p * q).sum(0) / np.sqrt((p**2).sum(0) * (q**2).sum(0))  # (channels, 80), r across images

Comparing amplitudes

For R², mean squared error or overlaid waveforms, remove the leftover slope from the predictions and from your trial averages in the same way: detrend each 0 to 0.79 s epoch (pip install scipy).

from scipy.signal import detrend

predicted = detrend(model(images).cpu().numpy(), axis=-1)
recorded = detrend(recorded, axis=-1)  # your (images, 63, 80) trial averages

For efficient on the test images, with the same processing applied to both sides:

processing	own readout: r	own readout: R²	new participant: r	new participant: R²
none	0.241	-0.005	0.168	-1.27
mean over the test images subtracted	0.241	0.085	0.168	0.031
linear detrend of each epoch	0.293	0.052	0.195	-0.78
1 Hz high-pass of each epoch	0.272	0.034	0.178	-0.91
1 Hz high-pass, then detrend	0.281	0.033	0.182	-0.97

Subtracting each side's mean over the test images gives the best possible offset, but it uses the images being scored. Detrending needs no such information and makes R² positive for 9 of the 10 participants' own readouts. It does not do so for a new participant, whose ERP amplitude and timing differ from the average. The step has to be the same on both sides. If your recordings were high-passed at 1 Hz or detrended before epoching, processing the 0.8 s predictions cannot reproduce that. When we filtered THINGS-EEG2 this way, r was about 0.25, but R² stayed between -0.13 and -0.06 with processed predictions (own readout) and fell to about -4 with unprocessed ones. For calibrated amplitudes, fit an offset and gain per channel and time point on images you do not evaluate on (with few images, one gain per channel).

Other datasets

We applied efficient without refitting to three other image-EEG datasets, following the steps above: shared channels re-referenced to their average, and one constant delay, chosen on half of the images, with scores from the other half (and then the halves swapped). Scores are Pearson r across images for the default output, against each participant's trial averages and against their mean over participants, over channel-time points whose noise ceiling in that dataset exceeds 0.1.

dataset	participants	shared channels	images	repetitions	presentation
Alljoined-1.6M (Xu et al., 2025)	20	32 (consumer headset)	the 200 THINGS-EEG2 test images	about 80	100 ms on, 100 ms off
THINGS-EEG1 (Grootswagers et al., 2022)	48	61	200 THINGS images not in THINGS-EEG2	12	50 ms on, 50 ms off
Alljoined1 (Xu et al., 2024)	8	59	960 natural scenes from NSD	7 or 8	300 ms on, 300-350 ms off

dataset	recordings lag the markers by	r, each participant	r, mean over participants
Alljoined-1.6M	40 ms	0.098	0.272
THINGS-EEG1	0-10 ms	0.037	0.047
Alljoined1	60-70 ms	0.040	0.034

Where the two halves chose different delays, both are given. On Alljoined-1.6M, which showed the same images, the THINGS-EEG2 recordings themselves (averaged over its ten participants) reach r = 0.298 against the Alljoined-1.6M mean, so the model gets about 90% of the way to that cross-dataset ceiling. THINGS-EEG1 presents images twice as fast and Alljoined1 shows scenes instead of single objects, with slower presentation and a different system; both keep much less of the image-specific signal.

How the encoders were built

EEG data: THINGS-EEG2 (10 participants; 16,540 training images shown 4 times, 200 test images shown 80 times), band-passed 0.1-100 Hz during recording and preprocessed as in step 1 above.
Image networks: adversarially trained ResNets from Gokce & Schrimpf (2025) at an intermediate stage, ViT-S/16 from timm (patch mean after block 7), and the Qwen3-Omni vision encoder (blocks 11-18). Their last blocks were fine-tuned to the training images' EEG with low-rank updates (LoRA).
Readout: features are standardized (ResNets compressed with PCA), combined with fixed weights, and mapped to the 5,040 channel-time values by a ridge regression per participant, with penalties chosen by leave-one-concept-out cross-validation.
Preprocessing: ResNets and ViT-S see a 224 x 224 center crop after resizing the shorter side to 256 pixels; Qwen sees a square center crop at 512 x 512.
Export check: in float32 with TF32 off, the released files reproduce the original test predictions to within 1e-5 of their standard deviation (3e-5 for accurate).

Limitations

Fitted on ten adults from one lab and one 5 Hz presentation paradigm. New participants, devices and paradigms lower accuracy; the average readout is the best starting point for them.
The predicted waveform includes the overlap from neighbouring images; for slower presentation it will not match after about 200 ms.
Validated only for THINGS-style object images, not for scenes, text or close-up faces.
The ResNets and ViT-S were trained on ImageNet, licensed for non-commercial research; check this for commercial use.

License

Apache 2.0 for the weights and code in this repository. THINGS-EEG2 is CC-BY 4.0; the THINGS images are not redistributed. ViT-S (timm) and Qwen3-Omni are Apache 2.0; the Qwen weights are downloaded from Qwen's repository, not redistributed here.

Citation

Please cite the benchmark and the ResNet checkpoints:

@inproceedings{gokce2026multimodal_brain_scaling,
  title     = {Multimodal Scaling Laws for Task \& Data-Optimized Models of Visual Cortex},
  author    = {Abdulkadir Gokce and Yingtian Tang and Martin Schrimpf},
  booktitle = {Forty-third International Conference on Machine Learning},
  year      = {2026},
  url       = {https://openreview.net/forum?id=OQ6jQHJPTT}
}

@inproceedings{gokce2025scalingprimatevvs,
  title     = {Scaling Laws for Task-Optimized Models of the Primate Visual Ventral Stream},
  author    = {Abdulkadir Gokce and Martin Schrimpf},
  booktitle = {Forty-second International Conference on Machine Learning},
  year      = {2025},
  url       = {https://openreview.net/forum?id=WxY61MmHYo}
}

Also cite THINGS-EEG2 (Gifford et al., 2022, NeuroImage 264, 119754), THINGS (Hebart et al., 2019, PLOS ONE 14, e0223792), ViT-S for efficient and tiny (Steiner et al., 2022, TMLR), and Qwen3-Omni for accurate (Qwen Team, 2025, arXiv:2509.17765). Transfer datasets: Alljoined-1.6M (Xu et al., 2025, arXiv:2508.18571), THINGS-EEG1 (Grootswagers et al., 2022, Scientific Data 9, 3) and Alljoined1 (Xu et al., 2024, arXiv:2404.05553).