| --- |
| license: cc-by-nc-4.0 |
| tags: |
| - pytorch |
| - brain-of |
| - neuroimaging |
| - fmri |
| - eeg |
| - meg |
| - omnifunctional modality |
| - brain foundation model |
| - joint pretraining |
| - arxiv:2602.23410 |
| --- |
| |
| <p align="center"> |
| <img src="assets/brain-of-logo.png" width="80" alt="Brain-OF logo"> |
| </p> |
|
|
| <h1 align="center">Brain-OF Base</h1> |
|
|
| <p align="center"> |
| <strong>Brain-OF: An Omnifunctional Foundation Model for fMRI, EEG and MEG</strong><br> |
| NeurIPS 2026 |
| </p> |
|
|
| <p align="center"> |
| <a href="https://hanningguo.github.io/BrainOF/"><img src="https://img.shields.io/badge/Project-Website-3564C0" alt="Project website"></a> |
| <a href="https://arxiv.org/abs/2602.23410"><img src="https://img.shields.io/badge/arXiv-2602.23410-B31B1B" alt="Paper on arXiv"></a> |
| <a href="https://github.com/HanningGuo/BrainOF"><img src="https://img.shields.io/badge/GitHub-Code-181717" alt="Code on GitHub"></a> |
| </p> |
|
|
| <!-- Replace YOUR_NAMESPACE throughout this card before publishing. --> |
| |
| ## Introduction |
| |
| <p align="justify"> |
| Brain-OF Base is a foundation model jointly pretrained on <strong>fMRI, EEG, and MEG</strong>, aiming to learn richer representations through cross-modal synergy. |
| <strong>ARNESS</strong> maps signals with varying channel/ROI counts and temporal resolutions to a shared latent space. <strong>Sparse MoE</strong> captures shared and modality-specific patterns, while <strong>DINT attention</strong> filters irrelevant context and integrates local and global dependencies. <strong>Masked Temporal-Frequency Modeling</strong> pretrains the model to reconstruct masked signals in the time and frequency domains. |
| Pretrained on <strong>37 public datasets</strong> with <strong>5.87 million samples</strong> from <strong>32,278 participants</strong>, the model supports unimodal and paired multimodal downstream tasks. |
| </p> |
| |
| ## Model specifications |
| |
| | Property | Value | |
| | --- | --- | |
| | Total parameters | 47.5M | |
| | Active parameters per token | 21.5M | |
| | Backbone layers | 12 | |
| | Patch size | 64 time points | |
| | Checkpoint format | PyTorch `.pth`; DDP loading path | |
| |
| ## Usage |
| |
| After [installing the environment](https://github.com/HanningGuo/BrainOF#installation), replace the checkpoint path below and run the example from the BrainOF repository root. It extracts a **256-dimensional embedding** from a randomly generated signal: |
| |
| ```python |
| import numpy as np |
| import torch |
| from datasets import DataTransform |
| from models.transformer import brainof_base_50M_A22M |
|
|
| model = brainof_base_50M_A22M(num_classes=0, layer_scale_type="per_channel") |
| checkpoint = torch.load( |
| "/path/to/brainof_base_50M_A22M.pth", map_location="cpu", weights_only=False |
| ) |
| weights = { |
| k.removeprefix("encoder."): v |
| for k, v in checkpoint["model"].items() if k.startswith("encoder.") |
| } |
| model.load_state_dict(weights) |
| model.eval() |
| |
| # Generate a random signal: 19 channels, 2,000 time points |
| signal = np.random.default_rng(0).standard_normal((19, 2000)).astype(np.float32) |
| transform = DataTransform(signal_size=signal.shape, patch_size=64, max_seq_length=2048) |
| patches, mask, _ = transform(signal) |
| x = torch.tensor(patches, dtype=torch.float32).unsqueeze(0) |
| attn_mask = torch.tensor(mask, dtype=torch.bool).unsqueeze(0) |
| |
| with torch.inference_mode(): |
| embedding = model(x, attn_mask) |
| print(embedding.shape) # torch.Size([1, 256]) |
| ``` |
| |
| For real data, replace the random signal with a [preprocessed](https://github.com/HanningGuo/BrainOF#data-preparation) NumPy array of shape `(channels_or_ROIs, time_points)`. `DataTransform` normalizes, patches, and pads it. We recommend using at most **2,048 patch tokens** per sample and the token count must not exceed `max_seq_length` (see [Input Size and Tokenization](https://github.com/HanningGuo/BrainOF#input-size-and-tokenization)). |
|
|
| ## Model family |
|
|
| | Model | Total parameters | Active / token | Layers | Embedding dim. | |
| | :--- | ---: | ---: | ---: | ---: | |
| | <a href="https://huggingface.co/han-ning/Brain-OF-Base" title="Current model"><img src="https://img.shields.io/badge/Brain--OF-Base-FFE28A" alt="Brain-OF Base — current model" style="outline: 2px solid #A87900; outline-offset: -2px; border-radius: 3px;"></a> | **47.5M** | **21.5M** | **12** | **256** | |
| | [](https://huggingface.co/han-ning/Brain-OF-Large) | 331M | 150M | 24 | 512 | |
| | [](https://huggingface.co/han-ning/Brain-OF-Huge) | 1.7B | 500M | 36 | 768 | |
| | [](https://huggingface.co/han-ning/Brain-OF-Giant) | 9B | 1.2B | 48 | 1024 | |
|
|
|
|
| ## License |
|
|
| This repository and the model weights are licensed under the [CC BY-NC 4.0](https://huggingface.co/han-ning/Brain-OF-Base/blob/main/LICENSE). |
|
|
| ## Citation |
|
|
| ```bibtex |
| @inproceedings{guo2026brainof, |
| title = {{Brain-OF}: An Omnifunctional Foundation Model for {fMRI}, {EEG} and {MEG}}, |
| author = {Guo, Hanning and Bi, Hanwen and Abdellatif, Farah and Galbenus, Andrei and Shah, Jon. N. and Morrison, Abigail and Dammers, Jürgen}, |
| booktitle = {Advances in Neural Information Processing Systems}, |
| year = {2026}, |
| url = {https://arxiv.org/abs/2602.23410} |
| } |
| ``` |
|
|
