license: cc-by-nc-4.0
tags:
- pytorch
- brain-of
- neuroimaging
- fmri
- eeg
- meg
- omnifunctional modality
- brain foundation model
- joint pretraining
- arxiv:2602.23410
Brain-OF Giant
Brain-OF: An Omnifunctional Foundation Model for fMRI, EEG and MEG
NeurIPS 2026
Introduction
Brain-OF Giant is the largest scale of the Brain-OF family, designed to learn richer representations from fMRI, EEG, and MEG through cross-modal synergy. ARNESS maps signals with varying channel/ROI counts and temporal resolutions to a shared latent space. Sparse MoE captures shared and modality-specific patterns, while DINT attention filters irrelevant context and integrates local and global dependencies. Masked Temporal-Frequency Modeling pretrains the model to reconstruct masked signals in the time and frequency domains. The published Base, Large, Huge, and Giant models were pretrained on 37 public datasets with 5.87 million samples from 32,278 participants.
Model specifications
| Property | Value |
|---|---|
| Total parameters | 9B |
| Active parameters per token | 1.2B |
| Backbone layers | 48 |
| Patch size | 64 time points |
| Checkpoint format | FSDP distributed checkpoint directory |
Usage
After installing the environment, replace the checkpoint directory below and run the example from the BrainOF repository root. It extracts a 1,024-dimensional embedding from a randomly generated signal:
Keep .metadata and all checkpoint shards in that directory. This example uses PyTorch Distributed Checkpoint to load the full encoder into CPU memory, so sufficient RAM is required.
import numpy as np
import torch
import torch.distributed.checkpoint as dcp
from datasets import DataTransform
from models.transformer import brainof_giant_9B_A1_2B
model = brainof_giant_9B_A1_2B(num_classes=0, layer_scale_type="per_channel")
checkpoint = {"model": {"encoder": model.state_dict()}}
dcp.load(
checkpoint,
checkpoint_id="/path/to/brainof_giant_9B_A1_2B",
no_dist=True,
)
model.load_state_dict(checkpoint["model"]["encoder"])
model.eval()
# Generate a random signal: 19 channels, 2,000 time points
signal = np.random.default_rng(0).standard_normal((19, 2000)).astype(np.float32)
transform = DataTransform(signal_size=signal.shape, patch_size=64, max_seq_length=2048)
patches, mask, _ = transform(signal)
x = torch.tensor(patches, dtype=torch.float32).unsqueeze(0)
attn_mask = torch.tensor(mask, dtype=torch.bool).unsqueeze(0)
with torch.inference_mode():
embedding = model(x, attn_mask)
print(embedding.shape) # torch.Size([1, 1024])
For real data, replace the random signal with a preprocessed NumPy array of shape (channels_or_ROIs, time_points). DataTransform normalizes, patches, and pads it. We recommend using at most 2,048 patch tokens per sample and the token count must not exceed max_seq_length (see Input Size and Tokenization).
Model family
| Model | Total parameters | Active / token | Layers | Embedding dim. |
|---|---|---|---|---|
 |
47.5M | 21.5M | 12 | 256 |
 |
331M | 150M | 24 | 512 |
 |
1.7B | 500M | 36 | 768 |
 |
9B | 1.2B | 48 | 1024 |
License
This repository and the model weights are licensed under the CC BY-NC 4.0.
Contact
Brain-OF Giant is currently available through limited access only. A public release is tentatively scheduled for the first half of 2027.
To request access before the public release, please email Hanning Guo from your official institutional email address (e.g., .edu or an equivalent university or research institute domain), including a brief description of your research objectives.
Citation
@inproceedings{guo2026brainof,
title = {{Brain-OF}: An Omnifunctional Foundation Model for {fMRI}, {EEG} and {MEG}},
author = {Guo, Hanning and Bi, Hanwen and Abdellatif, Farah and Galbenus, Andrei and Shah, Jon. N. and Morrison, Abigail and Dammers, Jürgen},
booktitle = {Advances in Neural Information Processing Systems},
year = {2026},
url = {https://arxiv.org/abs/2602.23410}
}