--- license: mit library_name: pytorch tags: [fmri, neuroscience, brain-decoding, natural-scenes-dataset] datasets: [teooi/cortexdecoder-nsd-flatmaps] --- # CortexDecoder Bridge weights for [CortexDecoder](https://www.biorxiv.org/content/10.64898/2026.10.09.757766v1), trained on NSD Subject 1. Code is on [GitHub](https://github.com/teooi/CortexDecoder) and the flatmaps and reconstructions are in [teooi/cortexdecoder-nsd-flatmaps](https://huggingface.co/datasets/teooi/cortexdecoder-nsd-flatmaps). A frozen CortexMAE encoder turns visual-cortex flatmap patches into 200 brain tokens of dimension 768. Two cross-attention bridges map those tokens to image features. | File | Bridge | Output | Parameters | |---|---|---|---| | `vgg_bridge.safetensors` | low-level | 7190 VGG16-BN tokens of dimension 512, inverted to an image with a Deep Image Prior | 43.7M | | `clip_bridge.safetensors` | semantic | 256 OpenCLIP ViT-bigG/14 tokens of dimension 1664, used to condition SDXL-unCLIP | 39.3M | `checkpoints/` has the original training checkpoints, which also contain the optimizer state. `config.json` lists the bridge settings. ## Loading ```python from huggingface_hub import hf_hub_download from safetensors.torch import load_file from cortexdecoder.core.config import Config # from the GitHub repo, code/v2 from cortexdecoder.models.bridge import FlatmapSingleHeadBridge bridge = FlatmapSingleHeadBridge(Config(), brain_dim=768, num_patches=200, target="clip") bridge.load_state_dict(load_file(hf_hub_download("teooi/cortexdecoder", "clip_bridge.safetensors"))) clip_tokens = bridge(brain_tokens) # [B, 200, 768] -> [B, 256, 1664] ``` The brain tokens come from `code/v2/scripts/generate_roi_tokens.py`, which runs CortexMAE with only the visual-cortex patches visible. ## Results NSD Subject 1, 998 test images, one diffusion sample per image. | PixCorr | SSIM | Alex(2) | Alex(5) | Incep | CLIP | Eff | SwAV | |---|---|---|---|---|---|---|---| | 0.366 | 0.413 | 98.1% | 99.1% | 94.9% | 93.5% | 0.628 | 0.360 | ## Citation ```bibtex @article{imototar2026cortexdecoder, title = {CortexDecoder: fMRI-to-image Reconstruction from Cortical Flatmaps}, author = {Imoto-Tar, Teo and Fung, Maxwell and Gurha, Arnav}, journal = {bioRxiv}, year = {2026}, doi = {10.64898/2026.10.09.757766} } ```