---
license: cc-by-nc-4.0
tags:
  - pytorch
  - brain-of
  - neuroimaging
  - fmri
  - eeg
  - meg
  - omnifunctional modality
  - brain foundation model
  - joint pretraining
  - arxiv:2602.23410
---

<p align="center">
  <img src="assets/brain-of-logo.png" width="80" alt="Brain-OF logo">
</p>

<h1 align="center">Brain-OF Base</h1>

<p align="center">
  <strong>Brain-OF: An Omnifunctional Foundation Model for fMRI, EEG and MEG</strong><br>
  NeurIPS 2026
</p>

<p align="center">
  <a href="https://hanningguo.github.io/BrainOF/"><img src="https://img.shields.io/badge/Project-Website-3564C0" alt="Project website"></a>
  <a href="https://arxiv.org/abs/2602.23410"><img src="https://img.shields.io/badge/arXiv-2602.23410-B31B1B" alt="Paper on arXiv"></a>
  <a href="https://github.com/HanningGuo/BrainOF"><img src="https://img.shields.io/badge/GitHub-Code-181717" alt="Code on GitHub"></a>
</p>

<!-- Replace YOUR_NAMESPACE throughout this card before publishing. -->

## Introduction

<p align="justify">
Brain-OF Base is a foundation model jointly pretrained on <strong>fMRI, EEG, and MEG</strong>, aiming to learn richer representations through cross-modal synergy.
<strong>ARNESS</strong> maps signals with varying channel/ROI counts and temporal resolutions to a shared latent space. <strong>Sparse MoE</strong> captures shared and modality-specific patterns, while <strong>DINT attention</strong> filters irrelevant context and integrates local and global dependencies. <strong>Masked Temporal-Frequency Modeling</strong> pretrains the model to reconstruct masked signals in the time and frequency domains.
Pretrained on <strong>37 public datasets</strong> with <strong>5.87 million samples</strong> from <strong>32,278 participants</strong>, the model supports unimodal and paired multimodal downstream tasks.
</p>

## Model specifications

| Property | Value |
| --- | --- |
| Total parameters | 47.5M |
| Active parameters per token | 21.5M |
| Backbone layers | 12 |
| Patch size | 64 time points |
| Checkpoint format | PyTorch `.pth`; DDP loading path |

## Usage

After [installing the environment](https://github.com/HanningGuo/BrainOF#installation), replace the checkpoint path below and run the example from the BrainOF repository root. It extracts a **256-dimensional embedding** from a randomly generated signal:

```python
import numpy as np
import torch
from datasets import DataTransform
from models.transformer import brainof_base_50M_A22M

model = brainof_base_50M_A22M(num_classes=0, layer_scale_type="per_channel")
checkpoint = torch.load(
    "/path/to/brainof_base_50M_A22M.pth", map_location="cpu", weights_only=False
)
weights = {
    k.removeprefix("encoder."): v
    for k, v in checkpoint["model"].items() if k.startswith("encoder.")
}
model.load_state_dict(weights)
model.eval()

# Generate a random signal: 19 channels, 2,000 time points
signal = np.random.default_rng(0).standard_normal((19, 2000)).astype(np.float32)
transform = DataTransform(signal_size=signal.shape, patch_size=64, max_seq_length=2048)
patches, mask, _ = transform(signal)
x = torch.tensor(patches, dtype=torch.float32).unsqueeze(0)
attn_mask = torch.tensor(mask, dtype=torch.bool).unsqueeze(0)

with torch.inference_mode():
    embedding = model(x, attn_mask)
print(embedding.shape)  # torch.Size([1, 256])
```

For real data, replace the random signal with a [preprocessed](https://github.com/HanningGuo/BrainOF#data-preparation) NumPy array of shape `(channels_or_ROIs, time_points)`. `DataTransform` normalizes, patches, and pads it. We recommend using at most **2,048 patch tokens** per sample and the token count must not exceed `max_seq_length` (see [Input Size and Tokenization](https://github.com/HanningGuo/BrainOF#input-size-and-tokenization)).

## Model family

| Model | Total parameters | Active / token | Layers | Embedding dim. |
| :--- | ---: | ---: | ---: | ---: |
| <a href="https://huggingface.co/han-ning/Brain-OF-Base" title="Current model"><img src="https://img.shields.io/badge/Brain--OF-Base-FFE28A" alt="Brain-OF Base — current model" style="outline: 2px solid #A87900; outline-offset: -2px; border-radius: 3px;"></a> | **47.5M** | **21.5M** | **12** | **256** |
| [![Brain-OF Large](https://img.shields.io/badge/Brain--OF-Large-FFD21E)](https://huggingface.co/han-ning/Brain-OF-Large) | 331M | 150M | 24 | 512 |
| [![Brain-OF Huge](https://img.shields.io/badge/Brain--OF-Huge-F4B400)](https://huggingface.co/han-ning/Brain-OF-Huge) | 1.7B | 500M | 36 | 768 |
| [![Brain-OF Giant](https://img.shields.io/badge/Brain--OF-Giant-D99A00)](https://huggingface.co/han-ning/Brain-OF-Giant) | 9B | 1.2B | 48 | 1024 |


## License

This repository and the model weights are licensed under the [CC BY-NC 4.0](https://huggingface.co/han-ning/Brain-OF-Base/blob/main/LICENSE).

## Citation

```bibtex
@inproceedings{guo2026brainof,
  title     = {{Brain-OF}: An Omnifunctional Foundation Model for {fMRI}, {EEG} and {MEG}},
  author    = {Guo, Hanning and Bi, Hanwen and Abdellatif, Farah and Galbenus, Andrei and Shah, Jon. N. and Morrison, Abigail and Dammers, Jürgen},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2026},
  url       = {https://arxiv.org/abs/2602.23410}
}
```