| --- |
| license: mit |
| tags: |
| - brain-decoding |
| - eeg |
| - fmri |
| - image-reconstruction |
| - language-decoding |
| - neuroscience |
| library_name: pytorch |
| --- |
| |
| # Eigenbrain: frozen brain-decoding inference bundle |
|
|
| This repository contains frozen inference assets for three independent |
| brain-decoding demonstrations: EEG classification, neural language decoding, |
| and NSD Subject-1 fMRI-to-image reconstruction. It is an inference/replay |
| bundle; it does not train a model and does not fit preprocessing transforms. |
|
|
| ## Contents |
|
|
| | Task | Input | Frozen route | Main output | |
| |---|---|---|---| |
| | EEG classification | SEED-V eigenmode EEG | temporal/mode-attention encoder + 5-class head | predicted class | |
| | Language decoding | Alice BCI observation and prior | PoE + subject layer + QFormer + BGE tokens + Phi-4 LoRA | decoded English text | |
| | Image reconstruction | NSD Subject-1 CIFTI-4096 and modes-2000 | anchored PoE + MindEye2 BrainNetwork + diffusion prior + SDXL-unCLIP | reconstructed PNG | |
|
|
| The language route uses the public base models `microsoft/Phi-4-mini-instruct` |
| and `BAAI/bge-m3`; the files in this repository are the project-specific |
| alignment, projector, and LoRA weights. The image route uses a large frozen |
| unCLIP checkpoint and requires a CUDA GPU with substantial storage and memory. |
|
|
| ## Download |
|
|
| Download this repository together with the companion inference code. The |
| project code expects the following local layout: |
|
|
| ```text |
| assets/ |
| weights/ |
| weights/alice_lora/ |
| weights/nsd_sub01/ |
| models/Phi-4-mini-instruct/ |
| models/bge-m3/ |
| ``` |
|
|
| From the companion GitHub repository, run `cd demo` first. |
| The companion `download_assets.py` restores the local layout above and downloads the project files from this |
| repository and the two public base models: |
|
|
| ```bash |
| export EIGEN_HF_REPO=SSp1ash/Eigenbrain |
| python download_assets.py |
| ``` |
|
|
| For classification and language only, omit the large NSD image files with |
| `python download_assets.py --skip-nsd`. |
|
|
| ## Run |
|
|
| ```bash |
| python run_inference.py --task classification --device cuda |
| |
| python run_inference.py --task language --device cuda \ |
| --phi-path models/Phi-4-mini-instruct \ |
| --bge-path models/bge-m3 |
| |
| python run_inference.py --task image --device cuda |
| ``` |
|
|
| The image command evaluates the default ten held-out NSD examples and writes |
| the selected qualitative reconstructions and image-level metrics under |
| `outputs/nsd_sub01/`. |
|
|
| ## Files |
|
|
| Demo data are stored under `assets/`; checkpoints are organized by task under |
| `demo/checkpoints/`. The largest file is |
| `demo/checkpoints/visual_decoding/unclip.ckpt` (about 18 GB). |
| Public base models are intentionally not duplicated here. |
|
|
| ## License and limitations |
|
|
| The code and frozen assets are provided for research demonstration and |
| reproducibility. Check the licenses and terms of the public Phi-4, BGE-M3, |
| MindEye2/SDXL-unCLIP components and the NSD data before redistribution or |
| commercial use. The NSD route is within-subject reconstruction for Subject 1, |
| not a cross-subject general-purpose image decoder. |
|
|
| ## Checkpoint naming |
|
|
| `demo/checkpoint_manifest.json` maps the task-based Hub names to the local filenames expected by the inference code and records file checksums. Renaming does not modify checkpoint bytes, training history, or evaluation status. Standard PEFT adapter filenames are retained for library compatibility. |
|
|
