File size: 3,556 Bytes
670f9de | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 | ---
library_name: braindecode
license: unknown
tags:
- BrainBERT
- braindecode
- ieeg
- seeg
- foundation-model
- model_hub_mixin
- pytorch_model_hub_mixin
---
# BrainBERT — Foundation Model for Intracranial Neural Signals
Pretrained weights for [`braindecode.models.BrainBERT`](https://braindecode.org/stable/generated/braindecode.models.BrainBERT.html),
a faithful braindecode port of **BrainBERT** (Wang et al., ICLR 2023), a
self-supervised foundation model for intracranial (sEEG/iEEG) recordings.
- Paper: [BrainBERT: Self-supervised representation learning for intracranial recordings](https://arxiv.org/abs/2302.14367) (ICLR 2023)
- Original code & weights: [czlwang/BrainBERT](https://github.com/czlwang/BrainBERT)
- braindecode docs: https://braindecode.org
## Provenance & license
These weights are the **official pretrained "large" checkpoint** (`stft`
variant) released by the original authors. The Transformer encoder and input
encoding are mapped **1:1** into the braindecode `BrainBERT` module (the masked
spectrogram-reconstruction head, used only for the self-supervised pretraining
objective, is kept for weight parity; the classification head is a
braindecode-native addition, randomly initialized).
The upstream repository ships **no explicit license file**, so this repository is
labelled `unknown`: the original authors retain all rights, and these weights are
re-hosted for convenience to make `from_pretrained` work out of the box. If you
use them, please **cite the original BrainBERT paper** and refer to the authors'
repository for terms of use.
## Model configuration
This checkpoint uses the released "large" configuration (~43M parameters):
| | |
|---|---|
| `hidden_dim` | 768 |
| `ffn_dim` | 3072 |
| `n_layers` | 6 |
| `n_heads` | 12 |
| `freq_cutoff` (input_dim) | 40 |
| `nperseg` | 400 |
| `noverlap` | 350 |
| `activation` | GELU |
| `sfreq` | 2048 Hz |
The signal is expected at **2048 Hz** (Laplacian-re-referenced, as in the paper).
The short-time Fourier transform front-end is computed **inside** the model, so it
consumes raw `(batch, n_chans, n_times)` signal directly (upstream fed a
pre-computed spectrogram). Frames are pooled over time and channels, so you may
freely change `n_chans` and `n_outputs` (the classification head is task-specific
and randomly initialized — fine-tune it on your downstream task). `n_times` only
needs to be long enough to yield at least one spectrogram frame.
## Usage
```python
from braindecode.models import BrainBERT
# The encoder loads the pretrained weights; the classification head is
# (re)initialized for your task via n_outputs.
model = BrainBERT.from_pretrained("braindecode/brainbert-pretrained", n_outputs=2)
```
## Fidelity
The ported encoder reproduces the upstream reference
(`MaskedTFModel`, `intermediate_rep=True`) to within **~6e-5** max absolute
difference on the authors' demo signal — the residual is cross-device float32
rounding in the fixed sinusoidal positional table (computed on GPU at training
time, regenerated on CPU here), which grows only at positions far beyond any
realistic sequence length.
## Citation
```bibtex
@inproceedings{wang2023brainbert,
title = {{BrainBERT}: Self-supervised representation learning for intracranial recordings},
author = {Wang, Christopher and Subramaniam, Vishwaas and Yaari, Adam Uri and
Kreiman, Gabriel and Katz, Boris and Cases, Ignacio and Barbu, Andrei},
booktitle = {International Conference on Learning Representations (ICLR)},
year = {2023}
}
```
|