|
Download README.md from teooi/cortexdecoder: direct link, hf CLI and curl.
- Browser
- Download file 2.37 kB
-
https://huggingface.co/teooi/cortexdecoder/resolve/main/README.md
- Command line
-
hf download hf://teooi/cortexdecoder/README.md
-
curl -L -o README.md https://huggingface.co/teooi/cortexdecoder/resolve/main/README.md
2.37 kB
| license: mit | |
| library_name: pytorch | |
| tags: [fmri, neuroscience, brain-decoding, natural-scenes-dataset] | |
| datasets: [teooi/cortexdecoder-nsd-flatmaps] | |
| # CortexDecoder | |
| Bridge weights for [CortexDecoder](https://www.biorxiv.org/content/10.64898/2026.10.09.757766v1), trained on NSD Subject 1. Code is on [GitHub](https://github.com/teooi/CortexDecoder) and the flatmaps and | |
| reconstructions are in [teooi/cortexdecoder-nsd-flatmaps](https://huggingface.co/datasets/teooi/cortexdecoder-nsd-flatmaps). | |
| A frozen CortexMAE encoder turns visual-cortex flatmap patches into 200 brain tokens of dimension 768. Two cross-attention | |
| bridges map those tokens to image features. | |
| | File | Bridge | Output | Parameters | | |
| |---|---|---|---| | |
| | `vgg_bridge.safetensors` | low-level | 7190 VGG16-BN tokens of dimension 512, inverted to an image with a Deep Image Prior | 43.7M | | |
| | `clip_bridge.safetensors` | semantic | 256 OpenCLIP ViT-bigG/14 tokens of dimension 1664, used to condition SDXL-unCLIP | 39.3M | | |
| `checkpoints/` has the original training checkpoints, which also contain the optimizer state. `config.json` lists the | |
| bridge settings. | |
| ## Loading | |
| ```python | |
| from huggingface_hub import hf_hub_download | |
| from safetensors.torch import load_file | |
| from cortexdecoder.core.config import Config # from the GitHub repo, code/v2 | |
| from cortexdecoder.models.bridge import FlatmapSingleHeadBridge | |
| bridge = FlatmapSingleHeadBridge(Config(), brain_dim=768, num_patches=200, target="clip") | |
| bridge.load_state_dict(load_file(hf_hub_download("teooi/cortexdecoder", "clip_bridge.safetensors"))) | |
| clip_tokens = bridge(brain_tokens) # [B, 200, 768] -> [B, 256, 1664] | |
| ``` | |
| The brain tokens come from `code/v2/scripts/generate_roi_tokens.py`, which runs CortexMAE with only the visual-cortex | |
| patches visible. | |
| ## Results | |
| NSD Subject 1, 998 test images, one diffusion sample per image. | |
| | PixCorr | SSIM | Alex(2) | Alex(5) | Incep | CLIP | Eff | SwAV | | |
| |---|---|---|---|---|---|---|---| | |
| | 0.366 | 0.413 | 98.1% | 99.1% | 94.9% | 93.5% | 0.628 | 0.360 | | |
| ## Citation | |
| ```bibtex | |
| @article{imototar2026cortexdecoder, | |
| title = {CortexDecoder: fMRI-to-image Reconstruction from Cortical Flatmaps}, | |
| author = {Imoto-Tar, Teo and Fung, Maxwell and Gurha, Arnav}, | |
| journal = {bioRxiv}, | |
| year = {2026}, | |
| doi = {10.64898/2026.10.09.757766} | |
| } | |
| ``` | |