File size: 2,374 Bytes
e0b9fcd
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
---
license: mit
library_name: pytorch
tags: [fmri, neuroscience, brain-decoding, natural-scenes-dataset]
datasets: [teooi/cortexdecoder-nsd-flatmaps]
---

# CortexDecoder

Bridge weights for [CortexDecoder](https://www.biorxiv.org/content/10.64898/2026.10.09.757766v1), trained on NSD Subject 1. Code is on [GitHub](https://github.com/teooi/CortexDecoder) and the flatmaps and
reconstructions are in [teooi/cortexdecoder-nsd-flatmaps](https://huggingface.co/datasets/teooi/cortexdecoder-nsd-flatmaps).

A frozen CortexMAE encoder turns visual-cortex flatmap patches into 200 brain tokens of dimension 768. Two cross-attention
bridges map those tokens to image features.

| File | Bridge | Output | Parameters |
|---|---|---|---|
| `vgg_bridge.safetensors` | low-level | 7190 VGG16-BN tokens of dimension 512, inverted to an image with a Deep Image Prior | 43.7M |
| `clip_bridge.safetensors` | semantic | 256 OpenCLIP ViT-bigG/14 tokens of dimension 1664, used to condition SDXL-unCLIP | 39.3M |

`checkpoints/` has the original training checkpoints, which also contain the optimizer state. `config.json` lists the
bridge settings.

## Loading

```python
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from cortexdecoder.core.config import Config                     # from the GitHub repo, code/v2
from cortexdecoder.models.bridge import FlatmapSingleHeadBridge

bridge = FlatmapSingleHeadBridge(Config(), brain_dim=768, num_patches=200, target="clip")
bridge.load_state_dict(load_file(hf_hub_download("teooi/cortexdecoder", "clip_bridge.safetensors")))
clip_tokens = bridge(brain_tokens)                               # [B, 200, 768] -> [B, 256, 1664]
```

The brain tokens come from `code/v2/scripts/generate_roi_tokens.py`, which runs CortexMAE with only the visual-cortex
patches visible.

## Results

NSD Subject 1, 998 test images, one diffusion sample per image.

| PixCorr | SSIM | Alex(2) | Alex(5) | Incep | CLIP | Eff | SwAV |
|---|---|---|---|---|---|---|---|
| 0.366 | 0.413 | 98.1% | 99.1% | 94.9% | 93.5% | 0.628 | 0.360 |

## Citation

```bibtex
@article{imototar2026cortexdecoder,
  title   = {CortexDecoder: fMRI-to-image Reconstruction from Cortical Flatmaps},
  author  = {Imoto-Tar, Teo and Fung, Maxwell and Gurha, Arnav},
  journal = {bioRxiv},
  year    = {2026},
  doi     = {10.64898/2026.10.09.757766}
}
```