File size: 6,451 Bytes
811b858 89a1369 811b858 89a1369 811b858 89a1369 811b858 e07cbe3 811b858 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 | ---
license: apache-2.0
tags:
- physiowave
- physiowave-t
- multimodal-fusion
- multimodal
- eeg
- ecg
- emg
- seizure-detection
- sensor-fusion
- biosignal
library_name: pytorch
---
# PhysioWave-T-RALF: multimodal seizure detection on SeizeIT2 (EEG + ECG + EMG)
The **PhysioWave-T** multimodal model, built on the three released encoders ([EEG](https://huggingface.co/ForeverBlue/EEG), [ECG](https://huggingface.co/ForeverBlue/ECG), [sEMG](https://huggingface.co/ForeverBlue/EMG)). Code: <https://github.com/ForeverBlue816/PhysioWave>.
Three independently pretrained C1 encoders -- EEG, ECG and sEMG -- fused by
**RALF (Reliability-Aware Latent Fusion)** for
seizure detection on SeizeIT2 (behind-the-ear EEG, ECG and deltoid EMG).
Each encoder hands the fusion a few summary tokens; RALF predicts a
reliability per modality and sample, gates the cross-attention with it, masks
absent modalities out, and adds a residual of reliability-weighted unimodal
logits.
Encoders **frozen** (the pretrained cores fixed; montage adapters, pools,
fusion core and heads trained), 69.4 M parameters in all.
Run `ralf_frozen_szo`, selected on validation auprc at epoch 5
counted from 0 (validation auprc 0.380).
Classes: ['background', 'seizure'].
## Test results
SeizeIT2's official split: patients sub-097 to sub-125 are the test set and
were never seen before the final evaluation; 12% of the training patients
were held out for checkpoint selection.
Continuous test recordings (778 recordings, 2,912 h, 181 seizures), the SeizeIT2 paper's scoring (any-overlap, means over recordings):
| metric | value |
|---|---|
| sensitivity at <= 2 false alarms/h | 32.0 % |
| sensitivity at <= 6.22 false alarms/h | 58.1 % |
| sensitivity at <= 11 false alarms/h | 74.4 % |
| AUROC (the dataset paper's epoch-based area) | 0.741 |
| AUPR (the dataset paper's epoch-based area) | 0.0121 |
Window-level test set (4 s windows; background subsampled to 30 per hour):
| modalities present | AUPRC | AUROC |
|---|---|---|
| EEG + ECG + EMG | 0.399 | 0.773 |
| EEG + ECG | 0.394 | 0.768 |
| EEG + EMG | 0.265 | 0.691 |
| ECG + EMG | 0.337 | 0.746 |
| EEG | 0.252 | 0.692 |
| ECG | 0.313 | 0.746 |
| EMG | 0.192 | 0.615 |
Under corruption of one modality at severity 0.7,
on top of the complete set, with the reliability the model assigned to each
modality:
| condition | AUPRC | reliability EEG | ECG | EMG |
|---|---|---|---|---|
| clean | 0.399 | 0.91 | 0.85 | 0.91 |
| EEG noise | 0.368 | 0.63 | 0.85 | 0.91 |
| EEG bad channel | 0.358 | 0.29 | 0.85 | 0.91 |
| EEG drift | 0.332 | 0.32 | 0.85 | 0.91 |
| ECG saturation | 0.367 | 0.91 | 0.33 | 0.91 |
| ECG noise | 0.395 | 0.91 | 0.65 | 0.91 |
| EMG dropout | 0.403 | 0.91 | 0.85 | 0.29 |
| EMG noise | 0.398 | 0.91 | 0.85 | 0.24 |
A corrupted modality's reliability drops and the other two stay where they
were; that is the mechanism the fusion is built around.
## Training curves
*(a) Validation AUPRC per epoch for the released run (bold) and for the five frozen-encoder RALF variants of the paper's schedule sweep; the dot on each curve is its selected epoch. The released checkpoint is epoch 5 of the original configuration, selected on validation AUPRC. (b) The released run's loss terms. The reliability loss holds at 0.04 through epoch 7, where the six-epoch warm-up reaches its peak learning rate; after that the reliability head drifts and the task loss follows, which is why selection stopped at epoch 5 and why the paper reports RALF as the band over all six runs rather than one curve. The variants at lr 1e-4 train stably for their full 15 epochs and select at epochs 2 to 5 with validation AUPRC 0.376 to 0.401.*
## Files
| file | what it is |
|---|---|
| `ralf_seizeit2_best.pth` | the model: encoders, adapters, fusion core and heads (epoch 5) |
| `results.json` | the test matrix: every modality subset and corruption |
| `events_cross.json` | event-level scoring on the continuous test recordings |
| `history.json` | training and validation per epoch |
| `encoders.json` | which pretrained keys each encoder took |
| `config_resolved.yaml` | the run's config; encoders point at the released files |
## The encoders it was built on
| modality | file | repository |
|---|---|---|
| EEG | `pretrain_eeg_c1_moe_best.pth` | ForeverBlue/EEG |
| ECG | `pretrain_ecg_c1_moe_best.pth` | ForeverBlue/ECG |
| EMG | `pretrain_emg_c1_moe_epoch38.pth` | ForeverBlue/EMG |
The weights here already contain all three encoders as they were loaded, so
they are not needed to RUN the model. They are needed to BUILD it:
`build_model` constructs each encoder from its pretraining checkpoint before
the fusion weights are loaded on top. Download the three files into one
directory and point `encoders.<m>` at them (the released config already
names them).
## Loading
The model's montage adapters and heads are sized by the task's data, so it is
rebuilt from the prepared windows (`Fusion/seizeit2_prep.py` in the code
repository) and then loaded strictly:
```python
import torch
from physiowave.config import load_config
from physiowave.fusion.train import build_model, load_tasks
cfg = load_config("config_resolved.yaml")
cfg["data"]["tasks"]["seizeit2"]["dir"] = "path/to/seizeit2" # train.h5 / val.h5 / test.h5
cfg["encoders"] = {m: f"path/to/encoders/{f}" for m, f in cfg["encoders"].items()
if m in ("eeg", "ecg", "emg") and f}
tasks = load_tasks(cfg, list(cfg["data"]["modalities"]))
model, _ = build_model(cfg, tasks, out_dir="encoders_built")
ck = torch.load("ralf_seizeit2_best.pth", map_location="cpu", weights_only=False)
model.load_state_dict(ck["model"]) # strict: every key must match
model.eval()
```
Only `train.h5` is read to size the model; nothing is trained.
## Training
```
fusion ralf, width 256, 8 fusion tokens, depth 2, heads 8
summary 4 tokens per modality
encoders frozen
lr 0.0003, batch 32, epochs 60 (patience 10), bf16
loss task CE + 0.2 x unimodal CE + 1.0 x reliability MSE + 1.0 x subset consistency
augment each modality corrupted with p 0.5; modalities dropped with p 0.3
seed 42
```
One run, one seed. The numbers above are that run's, not a mean.
Code: <https://github.com/ForeverBlue816/PhysioWave> (`docs/fusion.md`)
---
Prepared by `scripts/prepare_hf_fusion_release.py` from `ralf_frozen_szo` on
2026-10-29.
|