| --- |
| license: cc-by-nc-4.0 |
| tags: |
| - pytorch |
| - brain-of |
| - neuroimaging |
| - fmri |
| - eeg |
| - meg |
| - omnifunctional modality |
| - brain foundation model |
| - joint pretraining |
| - arxiv:2602.23410 |
| --- |
| |
| <p align="center"> |
| <img src="assets/brain-of-logo.png" width="80" alt="Brain-OF logo"> |
| </p> |
|
|
| <h1 align="center">Brain-OF Giant</h1> |
|
|
| <p align="center"> |
| <strong>Brain-OF: An Omnifunctional Foundation Model for fMRI, EEG and MEG</strong><br> |
| NeurIPS 2026 |
| </p> |
|
|
| <p align="center"> |
| <a href="https://hanningguo.github.io/BrainOF/"><img src="https://img.shields.io/badge/Project-Website-3564C0" alt="Project website"></a> |
| <a href="https://arxiv.org/abs/2602.23410"><img src="https://img.shields.io/badge/arXiv-2602.23410-B31B1B" alt="Paper on arXiv"></a> |
| <a href="https://github.com/HanningGuo/BrainOF"><img src="https://img.shields.io/badge/GitHub-Code-181717" alt="Code on GitHub"></a> |
| </p> |
|
|
| <!-- Replace YOUR_NAMESPACE throughout this card before publishing. --> |
| |
| ## Introduction |
| |
| <p align="justify"> |
| Brain-OF Giant is the largest scale of the Brain-OF family, designed to learn richer representations from <strong>fMRI, EEG, and MEG</strong> through cross-modal synergy. |
| <strong>ARNESS</strong> maps signals with varying channel/ROI counts and temporal resolutions to a shared latent space. <strong>Sparse MoE</strong> captures shared and modality-specific patterns, while <strong>DINT attention</strong> filters irrelevant context and integrates local and global dependencies. <strong>Masked Temporal-Frequency Modeling</strong> pretrains the model to reconstruct masked signals in the time and frequency domains. |
| The published Base, Large, Huge, and Giant models were pretrained on <strong>37 public datasets</strong> with <strong>5.87 million samples</strong> from <strong>32,278 participants</strong>. |
| </p> |
| |
| ## Model specifications |
| |
| | Property | Value | |
| | --- | --- | |
| | Total parameters | 9B | |
| | Active parameters per token | 1.2B | |
| | Backbone layers | 48 | |
| | Patch size | 64 time points | |
| | Checkpoint format | FSDP distributed checkpoint directory | |
| |
| ## Usage |
| |
| After [installing the environment](https://github.com/HanningGuo/BrainOF#installation), replace the checkpoint directory below and run the example from the BrainOF repository root. It extracts a **1,024-dimensional embedding** from a randomly generated signal: |
| |
| Keep `.metadata` and all checkpoint shards in that directory. This example uses PyTorch Distributed Checkpoint to load the full encoder into CPU memory, so sufficient RAM is required. |
| |
| ```python |
| import numpy as np |
| import torch |
| import torch.distributed.checkpoint as dcp |
| from datasets import DataTransform |
| from models.transformer import brainof_giant_9B_A1_2B |
| |
| model = brainof_giant_9B_A1_2B(num_classes=0, layer_scale_type="per_channel") |
| checkpoint = {"model": {"encoder": model.state_dict()}} |
| dcp.load( |
| checkpoint, |
| checkpoint_id="/path/to/brainof_giant_9B_A1_2B", |
| no_dist=True, |
| ) |
| model.load_state_dict(checkpoint["model"]["encoder"]) |
| model.eval() |
| |
| # Generate a random signal: 19 channels, 2,000 time points |
| signal = np.random.default_rng(0).standard_normal((19, 2000)).astype(np.float32) |
| transform = DataTransform(signal_size=signal.shape, patch_size=64, max_seq_length=2048) |
| patches, mask, _ = transform(signal) |
| x = torch.tensor(patches, dtype=torch.float32).unsqueeze(0) |
| attn_mask = torch.tensor(mask, dtype=torch.bool).unsqueeze(0) |
| |
| with torch.inference_mode(): |
| embedding = model(x, attn_mask) |
| print(embedding.shape) # torch.Size([1, 1024]) |
| ``` |
| |
| For real data, replace the random signal with a [preprocessed](https://github.com/HanningGuo/BrainOF#data-preparation) NumPy array of shape `(channels_or_ROIs, time_points)`. `DataTransform` normalizes, patches, and pads it. We recommend using at most **2,048 patch tokens** per sample and the token count must not exceed `max_seq_length` (see [Input Size and Tokenization](https://github.com/HanningGuo/BrainOF#input-size-and-tokenization)). |
|
|
| ## Model family |
|
|
| | Model | Total parameters | Active / token | Layers | Embedding dim. | |
| | :--- | ---: | ---: | ---: | ---: | |
| | [](https://huggingface.co/han-ning/Brain-OF-Base) | 47.5M | 21.5M | 12 | 256 | |
| | [](https://huggingface.co/han-ning/Brain-OF-Large) | 331M | 150M | 24 | 512 | |
| | [](https://huggingface.co/han-ning/Brain-OF-Huge) | 1.7B | 500M | 36 | 768 | |
| | <a href="https://huggingface.co/han-ning/Brain-OF-Giant" title="Current model"><img src="https://img.shields.io/badge/Brain--OF-Giant-D99A00" alt="Brain-OF Giant — current model" style="outline: 2px solid #A87900; outline-offset: -2px; border-radius: 3px;"></a> | **9B** | **1.2B** | **48** | **1024** | |
|
|
| ## License |
|
|
| This repository and the model weights are licensed under the [CC BY-NC 4.0](https://huggingface.co/han-ning/Brain-OF-Giant/blob/main/LICENSE). |
|
|
| ## Contact |
|
|
| Brain-OF Giant is currently available through limited access only. A public release is tentatively scheduled for the **first half of 2027**. |
|
|
| To request access before the public release, please email [Hanning Guo](mailto:hanning.guo@rwth-aachen.de) from your **official institutional email address** (e.g., `.edu` or an equivalent university or research institute domain), including a brief description of your **research objectives**. |
|
|
| ## Citation |
|
|
| ```bibtex |
| @inproceedings{guo2026brainof, |
| title = {{Brain-OF}: An Omnifunctional Foundation Model for {fMRI}, {EEG} and {MEG}}, |
| author = {Guo, Hanning and Bi, Hanwen and Abdellatif, Farah and Galbenus, Andrei and Shah, Jon. N. and Morrison, Abigail and Dammers, Jürgen}, |
| booktitle = {Advances in Neural Information Processing Systems}, |
| year = {2026}, |
| url = {https://arxiv.org/abs/2602.23410} |
| } |
| ``` |
|
|
