cortexdecoder / README.md
teooi's picture
Add 6 files
e0b9fcd verified
|
Raw History Blame Contribute Delete
2.37 kB
metadata
license: mit
library_name: pytorch
tags:
  - fmri
  - neuroscience
  - brain-decoding
  - natural-scenes-dataset
datasets:
  - teooi/cortexdecoder-nsd-flatmaps

CortexDecoder

Bridge weights for CortexDecoder, trained on NSD Subject 1. Code is on GitHub and the flatmaps and reconstructions are in teooi/cortexdecoder-nsd-flatmaps.

A frozen CortexMAE encoder turns visual-cortex flatmap patches into 200 brain tokens of dimension 768. Two cross-attention bridges map those tokens to image features.

File Bridge Output Parameters
vgg_bridge.safetensors low-level 7190 VGG16-BN tokens of dimension 512, inverted to an image with a Deep Image Prior 43.7M
clip_bridge.safetensors semantic 256 OpenCLIP ViT-bigG/14 tokens of dimension 1664, used to condition SDXL-unCLIP 39.3M

checkpoints/ has the original training checkpoints, which also contain the optimizer state. config.json lists the bridge settings.

Loading

from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from cortexdecoder.core.config import Config                     # from the GitHub repo, code/v2
from cortexdecoder.models.bridge import FlatmapSingleHeadBridge

bridge = FlatmapSingleHeadBridge(Config(), brain_dim=768, num_patches=200, target="clip")
bridge.load_state_dict(load_file(hf_hub_download("teooi/cortexdecoder", "clip_bridge.safetensors")))
clip_tokens = bridge(brain_tokens)                               # [B, 200, 768] -> [B, 256, 1664]

The brain tokens come from code/v2/scripts/generate_roi_tokens.py, which runs CortexMAE with only the visual-cortex patches visible.

Results

NSD Subject 1, 998 test images, one diffusion sample per image.

PixCorr SSIM Alex(2) Alex(5) Incep CLIP Eff SwAV
0.366 0.413 98.1% 99.1% 94.9% 93.5% 0.628 0.360

Citation

@article{imototar2026cortexdecoder,
  title   = {CortexDecoder: fMRI-to-image Reconstruction from Cortical Flatmaps},
  author  = {Imoto-Tar, Teo and Fung, Maxwell and Gurha, Arnav},
  journal = {bioRxiv},
  year    = {2026},
  doi     = {10.64898/2026.10.09.757766}
}