README.md

---
library_name: braindecode
license: unknown
tags:
- BrainBERT
- braindecode
- ieeg
- seeg
- foundation-model
- model_hub_mixin
- pytorch_model_hub_mixin
---

# BrainBERT — Foundation Model for Intracranial Neural Signals

Pretrained weights for [`braindecode.models.BrainBERT`](https://braindecode.org/stable/generated/braindecode.models.BrainBERT.html),
a faithful braindecode port of **BrainBERT** (Wang et al., ICLR 2023), a
self-supervised foundation model for intracranial (sEEG/iEEG) recordings.

- Paper: [BrainBERT: Self-supervised representation learning for intracranial recordings](https://arxiv.org/abs/2302.14367) (ICLR 2023)
- Original code & weights: [czlwang/BrainBERT](https://github.com/czlwang/BrainBERT)
- braindecode docs: https://braindecode.org

## Provenance & license

These weights are the **official pretrained "large" checkpoint** (`stft`
variant) released by the original authors. The Transformer encoder and input
encoding are mapped **1:1** into the braindecode `BrainBERT` module (the masked
spectrogram-reconstruction head, used only for the self-supervised pretraining
objective, is kept for weight parity; the classification head is a
braindecode-native addition, randomly initialized).

The upstream repository ships **no explicit license file**, so this repository is
labelled `unknown`: the original authors retain all rights, and these weights are
re-hosted for convenience to make `from_pretrained` work out of the box. If you
use them, please **cite the original BrainBERT paper** and refer to the authors'
repository for terms of use.

## Model configuration

This checkpoint uses the released "large" configuration (~43M parameters):

| | |
|---|---|
| `hidden_dim` | 768 |
| `ffn_dim` | 3072 |
| `n_layers` | 6 |
| `n_heads` | 12 |
| `freq_cutoff` (input_dim) | 40 |
| `nperseg` | 400 |
| `noverlap` | 350 |
| `activation` | GELU |
| `sfreq` | 2048 Hz |

The signal is expected at **2048 Hz** (Laplacian-re-referenced, as in the paper).
The short-time Fourier transform front-end is computed **inside** the model, so it
consumes raw `(batch, n_chans, n_times)` signal directly (upstream fed a
pre-computed spectrogram). Frames are pooled over time and channels, so you may
freely change `n_chans` and `n_outputs` (the classification head is task-specific
and randomly initialized — fine-tune it on your downstream task). `n_times` only
needs to be long enough to yield at least one spectrogram frame.

## Usage

```python
from braindecode.models import BrainBERT

# The encoder loads the pretrained weights; the classification head is
# (re)initialized for your task via n_outputs.
model = BrainBERT.from_pretrained("braindecode/brainbert-pretrained", n_outputs=2)
```

## Fidelity

The ported encoder reproduces the upstream reference
(`MaskedTFModel`, `intermediate_rep=True`) to within **~6e-5** max absolute
difference on the authors' demo signal — the residual is cross-device float32
rounding in the fixed sinusoidal positional table (computed on GPU at training
time, regenerated on CPU here), which grows only at positions far beyond any
realistic sequence length.

## Citation

```bibtex
@inproceedings{wang2023brainbert,
  title     = {{BrainBERT}: Self-supervised representation learning for intracranial recordings},
  author    = {Wang, Christopher and Subramaniam, Vishwaas and Yaari, Adam Uri and
               Kreiman, Gabriel and Katz, Boris and Cases, Ignacio and Barbu, Andrei},
  booktitle = {International Conference on Learning Representations (ICLR)},
  year      = {2023}
}
```