adammnrr commited on
Commit
670f9de
·
verified ·
1 Parent(s): 6c53255

Add BrainBERT 'large' pretrained weights (ported from czlwang/BrainBERT)

Browse files
Files changed (4) hide show
  1. README.md +92 -0
  2. config.json +19 -0
  3. model.safetensors +3 -0
  4. pytorch_model.bin +3 -0
README.md ADDED
@@ -0,0 +1,92 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: braindecode
3
+ license: unknown
4
+ tags:
5
+ - BrainBERT
6
+ - braindecode
7
+ - ieeg
8
+ - seeg
9
+ - foundation-model
10
+ - model_hub_mixin
11
+ - pytorch_model_hub_mixin
12
+ ---
13
+
14
+ # BrainBERT — Foundation Model for Intracranial Neural Signals
15
+
16
+ Pretrained weights for [`braindecode.models.BrainBERT`](https://braindecode.org/stable/generated/braindecode.models.BrainBERT.html),
17
+ a faithful braindecode port of **BrainBERT** (Wang et al., ICLR 2023), a
18
+ self-supervised foundation model for intracranial (sEEG/iEEG) recordings.
19
+
20
+ - Paper: [BrainBERT: Self-supervised representation learning for intracranial recordings](https://arxiv.org/abs/2302.14367) (ICLR 2023)
21
+ - Original code & weights: [czlwang/BrainBERT](https://github.com/czlwang/BrainBERT)
22
+ - braindecode docs: https://braindecode.org
23
+
24
+ ## Provenance & license
25
+
26
+ These weights are the **official pretrained "large" checkpoint** (`stft`
27
+ variant) released by the original authors. The Transformer encoder and input
28
+ encoding are mapped **1:1** into the braindecode `BrainBERT` module (the masked
29
+ spectrogram-reconstruction head, used only for the self-supervised pretraining
30
+ objective, is kept for weight parity; the classification head is a
31
+ braindecode-native addition, randomly initialized).
32
+
33
+ The upstream repository ships **no explicit license file**, so this repository is
34
+ labelled `unknown`: the original authors retain all rights, and these weights are
35
+ re-hosted for convenience to make `from_pretrained` work out of the box. If you
36
+ use them, please **cite the original BrainBERT paper** and refer to the authors'
37
+ repository for terms of use.
38
+
39
+ ## Model configuration
40
+
41
+ This checkpoint uses the released "large" configuration (~43M parameters):
42
+
43
+ | | |
44
+ |---|---|
45
+ | `hidden_dim` | 768 |
46
+ | `ffn_dim` | 3072 |
47
+ | `n_layers` | 6 |
48
+ | `n_heads` | 12 |
49
+ | `freq_cutoff` (input_dim) | 40 |
50
+ | `nperseg` | 400 |
51
+ | `noverlap` | 350 |
52
+ | `activation` | GELU |
53
+ | `sfreq` | 2048 Hz |
54
+
55
+ The signal is expected at **2048 Hz** (Laplacian-re-referenced, as in the paper).
56
+ The short-time Fourier transform front-end is computed **inside** the model, so it
57
+ consumes raw `(batch, n_chans, n_times)` signal directly (upstream fed a
58
+ pre-computed spectrogram). Frames are pooled over time and channels, so you may
59
+ freely change `n_chans` and `n_outputs` (the classification head is task-specific
60
+ and randomly initialized — fine-tune it on your downstream task). `n_times` only
61
+ needs to be long enough to yield at least one spectrogram frame.
62
+
63
+ ## Usage
64
+
65
+ ```python
66
+ from braindecode.models import BrainBERT
67
+
68
+ # The encoder loads the pretrained weights; the classification head is
69
+ # (re)initialized for your task via n_outputs.
70
+ model = BrainBERT.from_pretrained("braindecode/brainbert-pretrained", n_outputs=2)
71
+ ```
72
+
73
+ ## Fidelity
74
+
75
+ The ported encoder reproduces the upstream reference
76
+ (`MaskedTFModel`, `intermediate_rep=True`) to within **~6e-5** max absolute
77
+ difference on the authors' demo signal — the residual is cross-device float32
78
+ rounding in the fixed sinusoidal positional table (computed on GPU at training
79
+ time, regenerated on CPU here), which grows only at positions far beyond any
80
+ realistic sequence length.
81
+
82
+ ## Citation
83
+
84
+ ```bibtex
85
+ @inproceedings{wang2023brainbert,
86
+ title = {{BrainBERT}: Self-supervised representation learning for intracranial recordings},
87
+ author = {Wang, Christopher and Subramaniam, Vishwaas and Yaari, Adam Uri and
88
+ Kreiman, Gabriel and Katz, Boris and Cases, Ignacio and Barbu, Andrei},
89
+ booktitle = {International Conference on Learning Representations (ICLR)},
90
+ year = {2023}
91
+ }
92
+ ```
config.json ADDED
@@ -0,0 +1,19 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "hidden_dim": 768,
3
+ "ffn_dim": 3072,
4
+ "n_layers": 6,
5
+ "n_heads": 12,
6
+ "nperseg": 400,
7
+ "noverlap": 350,
8
+ "freq_cutoff": 40,
9
+ "stft_clip": 5,
10
+ "activation": "torch.nn.modules.activation.GELU",
11
+ "drop_prob": 0.1,
12
+ "n_outputs": 2,
13
+ "n_chans": 1,
14
+ "chs_info": null,
15
+ "n_times": 2048,
16
+ "input_window_seconds": null,
17
+ "sfreq": 2048.0,
18
+ "braindecode_version": "1.7.0dev0"
19
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a20e81cfb90192b58dd3002c5dac960a0193c727f026e6fd27bb872363edb23c
3
+ size 172753928
pytorch_model.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d953e90c8ab27afb3f44a9bce1ecd52705012dc948702227cdfeca6ac4c4da60
3
+ size 172776905