teooi commited on
Commit
e0b9fcd
·
verified ·
1 Parent(s): 8858ebb

Add 6 files

Browse files
README.md ADDED
@@ -0,0 +1,58 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: mit
3
+ library_name: pytorch
4
+ tags: [fmri, neuroscience, brain-decoding, natural-scenes-dataset]
5
+ datasets: [teooi/cortexdecoder-nsd-flatmaps]
6
+ ---
7
+
8
+ # CortexDecoder
9
+
10
+ Bridge weights for [CortexDecoder](https://www.biorxiv.org/content/10.64898/2026.10.09.757766v1), trained on NSD Subject 1. Code is on [GitHub](https://github.com/teooi/CortexDecoder) and the flatmaps and
11
+ reconstructions are in [teooi/cortexdecoder-nsd-flatmaps](https://huggingface.co/datasets/teooi/cortexdecoder-nsd-flatmaps).
12
+
13
+ A frozen CortexMAE encoder turns visual-cortex flatmap patches into 200 brain tokens of dimension 768. Two cross-attention
14
+ bridges map those tokens to image features.
15
+
16
+ | File | Bridge | Output | Parameters |
17
+ |---|---|---|---|
18
+ | `vgg_bridge.safetensors` | low-level | 7190 VGG16-BN tokens of dimension 512, inverted to an image with a Deep Image Prior | 43.7M |
19
+ | `clip_bridge.safetensors` | semantic | 256 OpenCLIP ViT-bigG/14 tokens of dimension 1664, used to condition SDXL-unCLIP | 39.3M |
20
+
21
+ `checkpoints/` has the original training checkpoints, which also contain the optimizer state. `config.json` lists the
22
+ bridge settings.
23
+
24
+ ## Loading
25
+
26
+ ```python
27
+ from huggingface_hub import hf_hub_download
28
+ from safetensors.torch import load_file
29
+ from cortexdecoder.core.config import Config # from the GitHub repo, code/v2
30
+ from cortexdecoder.models.bridge import FlatmapSingleHeadBridge
31
+
32
+ bridge = FlatmapSingleHeadBridge(Config(), brain_dim=768, num_patches=200, target="clip")
33
+ bridge.load_state_dict(load_file(hf_hub_download("teooi/cortexdecoder", "clip_bridge.safetensors")))
34
+ clip_tokens = bridge(brain_tokens) # [B, 200, 768] -> [B, 256, 1664]
35
+ ```
36
+
37
+ The brain tokens come from `code/v2/scripts/generate_roi_tokens.py`, which runs CortexMAE with only the visual-cortex
38
+ patches visible.
39
+
40
+ ## Results
41
+
42
+ NSD Subject 1, 998 test images, one diffusion sample per image.
43
+
44
+ | PixCorr | SSIM | Alex(2) | Alex(5) | Incep | CLIP | Eff | SwAV |
45
+ |---|---|---|---|---|---|---|---|
46
+ | 0.366 | 0.413 | 98.1% | 99.1% | 94.9% | 93.5% | 0.628 | 0.360 |
47
+
48
+ ## Citation
49
+
50
+ ```bibtex
51
+ @article{imototar2026cortexdecoder,
52
+ title = {CortexDecoder: fMRI-to-image Reconstruction from Cortical Flatmaps},
53
+ author = {Imoto-Tar, Teo and Fung, Maxwell and Gurha, Arnav},
54
+ journal = {bioRxiv},
55
+ year = {2026},
56
+ doi = {10.64898/2026.10.09.757766}
57
+ }
58
+ ```
checkpoints/clip_best.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7968b43bf70d2c4009547b2024c0b3a79a8b469790f58c89ec39db6717e43a3f
3
+ size 471515019
checkpoints/vgg_best.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1ed302c774be3deed12f738de4a898cdb726cc738ae2a90a8b719e8d330b2343
3
+ size 524787601
clip_bridge.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2259b8f9773fe3a41435215f8df5691d20faac804048532cf82a51c2dce263b3
3
+ size 157148016
config.json ADDED
@@ -0,0 +1,28 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "vgg_bridge": {
3
+ "d_model": 768,
4
+ "n_layers": 4,
5
+ "n_heads": 8,
6
+ "ffn_dim": 3072,
7
+ "dropout": 0.1,
8
+ "brain_dim": 768,
9
+ "num_brain_tokens": 200,
10
+ "brain_tokens": "flatmap_tokens_roi-nsdgeneral_fp16",
11
+ "best_epoch": 21,
12
+ "parameters_millions": 43.7
13
+ },
14
+ "clip_bridge": {
15
+ "d_model": 768,
16
+ "n_layers": 4,
17
+ "n_heads": 8,
18
+ "ffn_dim": 3072,
19
+ "dropout": 0.1,
20
+ "brain_dim": 768,
21
+ "num_brain_tokens": 200,
22
+ "brain_tokens": "flatmap_tokens_roi-nsdgeneral_fp16",
23
+ "best_epoch": 5,
24
+ "parameters_millions": 39.3,
25
+ "clip_loss": "mse",
26
+ "decoder_loss_weight": 16.0
27
+ }
28
+ }
vgg_bridge.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:184aa5606c83a8fb6f9c093aabc4d8a3f19604b710b3fea8a7a6b3fa338d6720
3
+ size 174905712