--- license: apache-2.0 tags: - physiowave - physiowave-t - multimodal-fusion - multimodal - eeg - ecg - emg - seizure-detection - sensor-fusion - biosignal library_name: pytorch --- # PhysioWave-T-RALF: multimodal seizure detection on SeizeIT2 (EEG + ECG + EMG) The **PhysioWave-T** multimodal model, built on the three released encoders ([EEG](https://huggingface.co/ForeverBlue/EEG), [ECG](https://huggingface.co/ForeverBlue/ECG), [sEMG](https://huggingface.co/ForeverBlue/EMG)). Code: . Three independently pretrained C1 encoders -- EEG, ECG and sEMG -- fused by **RALF (Reliability-Aware Latent Fusion)** for seizure detection on SeizeIT2 (behind-the-ear EEG, ECG and deltoid EMG). Each encoder hands the fusion a few summary tokens; RALF predicts a reliability per modality and sample, gates the cross-attention with it, masks absent modalities out, and adds a residual of reliability-weighted unimodal logits. Encoders **frozen** (the pretrained cores fixed; montage adapters, pools, fusion core and heads trained), 69.4 M parameters in all. Run `ralf_frozen_szo`, selected on validation auprc at epoch 5 counted from 0 (validation auprc 0.380). Classes: ['background', 'seizure']. ## Test results SeizeIT2's official split: patients sub-097 to sub-125 are the test set and were never seen before the final evaluation; 12% of the training patients were held out for checkpoint selection. Continuous test recordings (778 recordings, 2,912 h, 181 seizures), the SeizeIT2 paper's scoring (any-overlap, means over recordings): | metric | value | |---|---| | sensitivity at <= 2 false alarms/h | 32.0 % | | sensitivity at <= 6.22 false alarms/h | 58.1 % | | sensitivity at <= 11 false alarms/h | 74.4 % | | AUROC (the dataset paper's epoch-based area) | 0.741 | | AUPR (the dataset paper's epoch-based area) | 0.0121 | Window-level test set (4 s windows; background subsampled to 30 per hour): | modalities present | AUPRC | AUROC | |---|---|---| | EEG + ECG + EMG | 0.399 | 0.773 | | EEG + ECG | 0.394 | 0.768 | | EEG + EMG | 0.265 | 0.691 | | ECG + EMG | 0.337 | 0.746 | | EEG | 0.252 | 0.692 | | ECG | 0.313 | 0.746 | | EMG | 0.192 | 0.615 | Under corruption of one modality at severity 0.7, on top of the complete set, with the reliability the model assigned to each modality: | condition | AUPRC | reliability EEG | ECG | EMG | |---|---|---|---|---| | clean | 0.399 | 0.91 | 0.85 | 0.91 | | EEG noise | 0.368 | 0.63 | 0.85 | 0.91 | | EEG bad channel | 0.358 | 0.29 | 0.85 | 0.91 | | EEG drift | 0.332 | 0.32 | 0.85 | 0.91 | | ECG saturation | 0.367 | 0.91 | 0.33 | 0.91 | | ECG noise | 0.395 | 0.91 | 0.65 | 0.91 | | EMG dropout | 0.403 | 0.91 | 0.85 | 0.29 | | EMG noise | 0.398 | 0.91 | 0.85 | 0.24 | A corrupted modality's reliability drops and the other two stay where they were; that is the mechanism the fusion is built around. ## Training curves ![Checkpoint selection and loss terms](training_curves.png) *(a) Validation AUPRC per epoch for the released run (bold) and for the five frozen-encoder RALF variants of the paper's schedule sweep; the dot on each curve is its selected epoch. The released checkpoint is epoch 5 of the original configuration, selected on validation AUPRC. (b) The released run's loss terms. The reliability loss holds at 0.04 through epoch 7, where the six-epoch warm-up reaches its peak learning rate; after that the reliability head drifts and the task loss follows, which is why selection stopped at epoch 5 and why the paper reports RALF as the band over all six runs rather than one curve. The variants at lr 1e-4 train stably for their full 15 epochs and select at epochs 2 to 5 with validation AUPRC 0.376 to 0.401.* ## Files | file | what it is | |---|---| | `ralf_seizeit2_best.pth` | the model: encoders, adapters, fusion core and heads (epoch 5) | | `results.json` | the test matrix: every modality subset and corruption | | `events_cross.json` | event-level scoring on the continuous test recordings | | `history.json` | training and validation per epoch | | `encoders.json` | which pretrained keys each encoder took | | `config_resolved.yaml` | the run's config; encoders point at the released files | ## The encoders it was built on | modality | file | repository | |---|---|---| | EEG | `pretrain_eeg_c1_moe_best.pth` | ForeverBlue/EEG | | ECG | `pretrain_ecg_c1_moe_best.pth` | ForeverBlue/ECG | | EMG | `pretrain_emg_c1_moe_epoch38.pth` | ForeverBlue/EMG | The weights here already contain all three encoders as they were loaded, so they are not needed to RUN the model. They are needed to BUILD it: `build_model` constructs each encoder from its pretraining checkpoint before the fusion weights are loaded on top. Download the three files into one directory and point `encoders.` at them (the released config already names them). ## Loading The model's montage adapters and heads are sized by the task's data, so it is rebuilt from the prepared windows (`Fusion/seizeit2_prep.py` in the code repository) and then loaded strictly: ```python import torch from physiowave.config import load_config from physiowave.fusion.train import build_model, load_tasks cfg = load_config("config_resolved.yaml") cfg["data"]["tasks"]["seizeit2"]["dir"] = "path/to/seizeit2" # train.h5 / val.h5 / test.h5 cfg["encoders"] = {m: f"path/to/encoders/{f}" for m, f in cfg["encoders"].items() if m in ("eeg", "ecg", "emg") and f} tasks = load_tasks(cfg, list(cfg["data"]["modalities"])) model, _ = build_model(cfg, tasks, out_dir="encoders_built") ck = torch.load("ralf_seizeit2_best.pth", map_location="cpu", weights_only=False) model.load_state_dict(ck["model"]) # strict: every key must match model.eval() ``` Only `train.h5` is read to size the model; nothing is trained. ## Training ``` fusion ralf, width 256, 8 fusion tokens, depth 2, heads 8 summary 4 tokens per modality encoders frozen lr 0.0003, batch 32, epochs 60 (patience 10), bf16 loss task CE + 0.2 x unimodal CE + 1.0 x reliability MSE + 1.0 x subset consistency augment each modality corrupted with p 0.5; modalities dropped with p 0.3 seed 42 ``` One run, one seed. The numbers above are that run's, not a mean. Code: (`docs/fusion.md`) --- Prepared by `scripts/prepare_hf_fusion_release.py` from `ralf_frozen_szo` on 2026-10-29.