cortexdecoder / README.md
teooi's picture
Add 6 files
e0b9fcd verified
|
Raw History Blame Contribute Delete
2.37 kB
---
license: mit
library_name: pytorch
tags: [fmri, neuroscience, brain-decoding, natural-scenes-dataset]
datasets: [teooi/cortexdecoder-nsd-flatmaps]
---
# CortexDecoder
Bridge weights for [CortexDecoder](https://www.biorxiv.org/content/10.64898/2026.10.09.757766v1), trained on NSD Subject 1. Code is on [GitHub](https://github.com/teooi/CortexDecoder) and the flatmaps and
reconstructions are in [teooi/cortexdecoder-nsd-flatmaps](https://huggingface.co/datasets/teooi/cortexdecoder-nsd-flatmaps).
A frozen CortexMAE encoder turns visual-cortex flatmap patches into 200 brain tokens of dimension 768. Two cross-attention
bridges map those tokens to image features.
| File | Bridge | Output | Parameters |
|---|---|---|---|
| `vgg_bridge.safetensors` | low-level | 7190 VGG16-BN tokens of dimension 512, inverted to an image with a Deep Image Prior | 43.7M |
| `clip_bridge.safetensors` | semantic | 256 OpenCLIP ViT-bigG/14 tokens of dimension 1664, used to condition SDXL-unCLIP | 39.3M |
`checkpoints/` has the original training checkpoints, which also contain the optimizer state. `config.json` lists the
bridge settings.
## Loading
```python
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from cortexdecoder.core.config import Config # from the GitHub repo, code/v2
from cortexdecoder.models.bridge import FlatmapSingleHeadBridge
bridge = FlatmapSingleHeadBridge(Config(), brain_dim=768, num_patches=200, target="clip")
bridge.load_state_dict(load_file(hf_hub_download("teooi/cortexdecoder", "clip_bridge.safetensors")))
clip_tokens = bridge(brain_tokens) # [B, 200, 768] -> [B, 256, 1664]
```
The brain tokens come from `code/v2/scripts/generate_roi_tokens.py`, which runs CortexMAE with only the visual-cortex
patches visible.
## Results
NSD Subject 1, 998 test images, one diffusion sample per image.
| PixCorr | SSIM | Alex(2) | Alex(5) | Incep | CLIP | Eff | SwAV |
|---|---|---|---|---|---|---|---|
| 0.366 | 0.413 | 98.1% | 99.1% | 94.9% | 93.5% | 0.628 | 0.360 |
## Citation
```bibtex
@article{imototar2026cortexdecoder,
title = {CortexDecoder: fMRI-to-image Reconstruction from Cortical Flatmaps},
author = {Imoto-Tar, Teo and Fung, Maxwell and Gurha, Arnav},
journal = {bioRxiv},
year = {2026},
doi = {10.64898/2026.10.09.757766}
}
```