ECG / README.md

Model card: PhysioWave-T name, training curves, links to the series

a193638 verified 2 days ago

9.93 kB

---
license: apache-2.0
tags:
  - physiowave
  - physiowave-t
  - foundation-model
  - ecg
  - electrocardiogram
  - biosignal
  - self-supervised
  - masked-autoencoder
  - pretrained
library_name: pytorch
---
# PhysioWave-T-ECG: multi-route pretrained ECG encoder

One of the three **PhysioWave-T** encoders ([EEG](https://huggingface.co/ForeverBlue/EEG), ECG, [sEMG](https://huggingface.co/ForeverBlue/EMG)); the [RALF fusion model](https://huggingface.co/ForeverBlue/RALF-SeizeIT2) combines them. Code: <https://github.com/ForeverBlue816/PhysioWave>.

Masked dual-reconstruction pretraining on 12-lead and single-lead ECG. One shared transformer serves two lead configurations through route-specific wavelet frontends, and lead identity enters as a learned embedding of the lead NAME rather than of its index.

Trained for 57 epochs of 60 configured
(57000 optimizer updates) on a mixture of public
ECG corpora, 30.1 M parameters. The files
here are built around `pretrain_ecg_c1_moe_best.pth`, epoch 33.

## Objective

```
0.5*L_spec + 0.5*L_raw + 0.001*L_foldKL
```

Two heads at equal weight. One predicts folded wavelet patches, the other
predicts the preprocessed ECG waveform of the same masked patches.
Masking is applied BEFORE the wavelet frontend (True),
at ratio 0.7, and the spec target is per-patch
normalised (True).

## Training curves

Pretraining ran for 60 epochs (60,000 optimizer updates) on 16 GPUs over the seven corpora. The validation loss reaches its plateau by about 20,000 steps and stays there while training continues to improve; the released checkpoint (epoch 33, the lowest validation total loss) lies in that plateau, and the four per-criterion checkpoints published beside it are within validation noise of one another.

![Training and validation per epoch](training_curves.png)

*Train and validation per epoch: the total loss, the two reconstruction terms (folded wavelet patch, raw waveform) and the masked spectral correlation.*

## Validation, at the released checkpoint (epoch 33)

| metric | value | what 1.0 would mean |
|---|---|---|
| total loss | 0.1827 | — |
| spec MSE | 0.2562 | — |
| spec NMSE | 0.256 | predicting zero |
| spec Pearson r | 0.851 | perfect |
| raw SmoothL1 | 0.1092 | — |
| raw NMSE | 0.255 | predicting zero |
| raw Pearson r | 0.845 | perfect |
| route-macro total | 0.2041 | — |

NMSE is dimensionless: MSE(prediction, target) / MSE(0, target), so 1.0 is
what predicting zero scores on that head's own target. The two losses are NOT
comparable to each other -- one is an MSE on normalised wavelet coefficients,
the other a SmoothL1 on the window-normalised waveform. The columns that compare the
heads are the correlations and the NMSEs.

The route-macro row weights the two routes equally. The plain rows weight by
sample count, and validation is dominated by the largest corpora, so a model
specialising toward the smaller ones moves the two rows in opposite
directions.

## Files

### Which file do you want

| your situation | download |
|---|---|
| fine-tune on any montage | `ecg_c1_encoder.pth` |
| your montage IS one of the two routes below | `pretrain_ecg_c1_moe_best.pth`, then cut the route out (one command, below) |
| continue pretraining, or take the model apart yourself | `pretrain_ecg_c1_moe_best.pth` |

**`pretrain_ecg_c1_moe_best.pth` is the complete model** -- all two wavelet frontends, the
patchers, both reconstruction decoders. Nothing is held back; only the
optimiser state was stripped, which no downstream use needs. Take it if you
want to keep or strip pieces yourself.

`ecg_c1_encoder.pth` is the same weights minus everything route-bound and
pretraining-only. It exists because it cannot be loaded wrong.

| file | what it is |
|---|---|
| `ecg_c1_encoder.pth` | encoder alone: channel embedding, gate, transformer |
| `pretrain_ecg_c1_moe_best.pth` | complete model, lowest validation **total** loss (epoch 33) |
| `pretrain_ecg_c1_moe_final.pth` | complete model, last epoch (epoch 57) |
| `pretrain_ecg_c1_moe_best_spec.pth` | complete model, lowest spec MSE (epoch 19) |
| `pretrain_ecg_c1_moe_best_raw.pth` | complete model, lowest raw SmoothL1 (epoch 37) |
| `pretrain_ecg_c1_moe_best_macro.pth` | complete model, lowest route-macro total (epoch 28) |
| `metrics_epoch.jsonl` | the full training curve |
| `config_resolved.yaml` | every resolved hyperparameter |

### Why several checkpoints

Each is the minimum of a different validation metric, and they are published
together so a downstream comparison can ask which criterion transfers instead
of assuming the one that names `best.pth`. Reconstruction loss cannot answer
that question; a downstream task can.

`best.pth` is selected on the TOTAL loss -- the loss that was trained (epoch
32 counted from 0). It is not the best spec loss, and not
the best raw loss.

### What is in the complete model

| component | parameters | share | |
|---|---|---|---|
| shared transformer | 28.38 M | 94.3% | encoder, always transfers |
| channel-name embedding | 0.00 M | 0.0% | always transfers |
| channel projection | 0.03 M | 0.1% | always transfers |
| wavelet frontends (all two routes) | 0.03 M | 0.1% | route-bound; downstream usually builds its own |
| patchers (both rates) | 0.20 M | 0.6% | route-bound |
| spec decoder | 0.72 M | 2.4% | pretraining only |
| raw decoder | 0.72 M | 2.4% | pretraining only |
| **total** | **30.08 M** | 100% | |

The shared transformer is 94% of the model and the two frontends together are 0%, which is why there is no per-route encoder file here: two of them would duplicate that transformer two times to deliver frontends that are a small fraction of it. Cut the one you want out of the complete checkpoint instead.

### What transfers, and what does not

`downstream.py` splits a checkpoint into two sets. The channel embedding, its
gate and the shared transformer **always** load. The wavelet frontend and the
patcher load **only when your montage is the route they were trained for** --
same channel count, same patch length -- and are skipped otherwise rather
than reshaped.

A 12-lead recording at 500 Hz in 10 s windows IS the `L12_500` route, so on the most common ECG input the pretrained frontend and patcher transfer as well: cut that route out (below) and pass `route_id="L12_500"`. Any other lead set, rate or window length builds its own.

The two routes the shared transformer was pretrained across:

| route | channels | rate |
|---|---|---|
| `L12_500` | 12 | 500 Hz |
| `L1_250` | 1 | 250 Hz |

To fine-tune ON one of them exactly, take that route's frontend and patcher out
of the complete checkpoint:

```bash
python scripts/export_eeg_pretrained_encoder.py \
    --checkpoint pretrain_ecg_c1_moe_best.pth --route L12_500 \
    --output encoder_L12_500.pth
```

The exporter reads the modality from the checkpoint; its name is historical.
Omit `--route` and you get `ecg_c1_encoder.pth` again. Both pretraining decoders are
dropped either way: fine-tuning against the pretext objective by accident is
the failure that prevents.

The released checkpoint and the last epoch differ by 2.6% in validation total loss (0.1827 against 0.1877). Which transfers better is a downstream question, not a reconstruction-loss question -- measure both.

## The channel vocabulary travels with the weights

An embedding row means whatever channel held that id when the row was learned.
Every checkpoint here carries the vocabulary and its hash. Resolving
lead names against a different vocabulary silently trains on relabelled
channels, which is why the hash is checked rather than assumed.

## Architecture

```
embed_dim   512      depth 9      heads 8
norm        rmsnorm     ffn swiglu     qk_norm True
wavelet     max_level 3, kernel 16, bank db4, db6, sym4, sym8, coif2
channel     id embedding, injected at the token site
mask        frequency_guided, ratio 0.7
```

## Loading

`ECGC1Downstream` builds the frontend and patcher for your montage and loads this
file into everything else. Nothing here is route-bound, so the montage, the
window length and the patch length are yours to choose.

```python
import torch
from physiowave.ecg_c1.downstream import ECGC1Downstream
ENC = "ecg_c1_encoder.pth"
ck = torch.load(ENC, map_location="cpu", weights_only=False)
mc = ck["model_config"]                  # the width the weights were trained at
CH = ["I", "II", "III", "aVR", "aVL", "aVF",
      "V1", "V2", "V3", "V4", "V5", "V6"]   # your leads
model = ECGC1Downstream(
    in_channels=len(CH), window_samples=5000, sampling_rate=500.0,
    patch_samples=250, num_classes=2, channel_names=CH,
    embed_dim=mc["embed_dim"], depth=mc["depth"], num_heads=mc["num_heads"],
    channel_embed_dim=mc.get("channel_embed_dim", 64),
    channel_vocab_size=ck["channel_vocab_size"])
report = model.load_pretrained(ENC)
print(model.describe_transfer(report))
# loaded: transformer N, channel encoder N, gate yes, frontend 0, patcher 0
class Meta:                               # or your dataset's ChannelMeta
    channel_names = CH
    channel_mask = None
out = model(torch.randn(2, len(CH), 5000), Meta())   # {"logits": [2, 2]}
```

`frontend 0, patcher 0` is the expected line, not a warning: those are the two
things your montage supplies.

**The frontend you build is not randomly initialised.** Its filters are seeded
from real wavelet families -- `db4, db6, sym4, sym8, coif2` -- and learned from there during
fine-tuning. What a fresh frontend costs you is the adaptation those filters
underwent during pretraining, not the wavelet prior itself.

Names, not indices, are what this encoder and your data agree on. Pass real
lead names and check `channel_vocab_sha256` -- `load_pretrained` refuses a
mismatch rather than silently training on relabelled channels.

Code: <https://github.com/ForeverBlue816/PhysioWave>

---
Prepared by `scripts/prepare_hf_release.py` from `pretrain_ecg_c1_moe`
on 2026-10-29.

	---
	license: apache-2.0
	tags:
	- physiowave
	- physiowave-t
	- foundation-model
	- ecg
	- electrocardiogram
	- biosignal
	- self-supervised
	- masked-autoencoder
	- pretrained
	library_name: pytorch
	---

	# PhysioWave-T-ECG: multi-route pretrained ECG encoder

	One of the three PhysioWave-T encoders ([EEG](https://huggingface.co/ForeverBlue/EEG), ECG, [sEMG](https://huggingface.co/ForeverBlue/EMG)); the [RALF fusion model](https://huggingface.co/ForeverBlue/RALF-SeizeIT2) combines them. Code: <https://github.com/ForeverBlue816/PhysioWave>.

	Masked dual-reconstruction pretraining on 12-lead and single-lead ECG. One shared transformer serves two lead configurations through route-specific wavelet frontends, and lead identity enters as a learned embedding of the lead NAME rather than of its index.

	Trained for 57 epochs of 60 configured
	(57000 optimizer updates) on a mixture of public
	ECG corpora, 30.1 M parameters. The files
	here are built around `pretrain_ecg_c1_moe_best.pth`, epoch 33.

	## Objective

	```
	0.5L_spec + 0.5L_raw + 0.001*L_foldKL
	```

	Two heads at equal weight. One predicts folded wavelet patches, the other
	predicts the preprocessed ECG waveform of the same masked patches.
	Masking is applied BEFORE the wavelet frontend (True),
	at ratio 0.7, and the spec target is per-patch
	normalised (True).

	## Training curves

	Pretraining ran for 60 epochs (60,000 optimizer updates) on 16 GPUs over the seven corpora. The validation loss reaches its plateau by about 20,000 steps and stays there while training continues to improve; the released checkpoint (epoch 33, the lowest validation total loss) lies in that plateau, and the four per-criterion checkpoints published beside it are within validation noise of one another.

	![Training and validation per epoch](training_curves.png)

	Train and validation per epoch: the total loss, the two reconstruction terms (folded wavelet patch, raw waveform) and the masked spectral correlation.

	## Validation, at the released checkpoint (epoch 33)

	\| metric \| value \| what 1.0 would mean \|
	\|---\|---\|---\|
	\| total loss \| 0.1827 \| — \|
	\| spec MSE \| 0.2562 \| — \|
	\| spec NMSE \| 0.256 \| predicting zero \|
	\| spec Pearson r \| 0.851 \| perfect \|
	\| raw SmoothL1 \| 0.1092 \| — \|
	\| raw NMSE \| 0.255 \| predicting zero \|
	\| raw Pearson r \| 0.845 \| perfect \|
	\| route-macro total \| 0.2041 \| — \|

	NMSE is dimensionless: MSE(prediction, target) / MSE(0, target), so 1.0 is
	what predicting zero scores on that head's own target. The two losses are NOT
	comparable to each other -- one is an MSE on normalised wavelet coefficients,
	the other a SmoothL1 on the window-normalised waveform. The columns that compare the
	heads are the correlations and the NMSEs.

	The route-macro row weights the two routes equally. The plain rows weight by
	sample count, and validation is dominated by the largest corpora, so a model
	specialising toward the smaller ones moves the two rows in opposite
	directions.

	## Files

	### Which file do you want

	\| your situation \| download \|
	\|---\|---\|
	\| fine-tune on any montage \| `ecg_c1_encoder.pth` \|
	\| your montage IS one of the two routes below \| `pretrain_ecg_c1_moe_best.pth`, then cut the route out (one command, below) \|
	\| continue pretraining, or take the model apart yourself \| `pretrain_ecg_c1_moe_best.pth` \|

	`pretrain_ecg_c1_moe_best.pth` is the complete model -- all two wavelet frontends, the
	patchers, both reconstruction decoders. Nothing is held back; only the
	optimiser state was stripped, which no downstream use needs. Take it if you
	want to keep or strip pieces yourself.

	`ecg_c1_encoder.pth` is the same weights minus everything route-bound and
	pretraining-only. It exists because it cannot be loaded wrong.

	\| file \| what it is \|
	\|---\|---\|
	\| `ecg_c1_encoder.pth` \| encoder alone: channel embedding, gate, transformer \|
	\| `pretrain_ecg_c1_moe_best.pth` \| complete model, lowest validation total loss (epoch 33) \|
	\| `pretrain_ecg_c1_moe_final.pth` \| complete model, last epoch (epoch 57) \|
	\| `pretrain_ecg_c1_moe_best_spec.pth` \| complete model, lowest spec MSE (epoch 19) \|
	\| `pretrain_ecg_c1_moe_best_raw.pth` \| complete model, lowest raw SmoothL1 (epoch 37) \|
	\| `pretrain_ecg_c1_moe_best_macro.pth` \| complete model, lowest route-macro total (epoch 28) \|
	\| `metrics_epoch.jsonl` \| the full training curve \|
	\| `config_resolved.yaml` \| every resolved hyperparameter \|

	### Why several checkpoints

	Each is the minimum of a different validation metric, and they are published
	together so a downstream comparison can ask which criterion transfers instead
	of assuming the one that names `best.pth`. Reconstruction loss cannot answer
	that question; a downstream task can.

	`best.pth` is selected on the TOTAL loss -- the loss that was trained (epoch
	32 counted from 0). It is not the best spec loss, and not
	the best raw loss.

	### What is in the complete model

	\| component \| parameters \| share \| \|
	\|---\|---\|---\|---\|
	\| shared transformer \| 28.38 M \| 94.3% \| encoder, always transfers \|
	\| channel-name embedding \| 0.00 M \| 0.0% \| always transfers \|
	\| channel projection \| 0.03 M \| 0.1% \| always transfers \|
	\| wavelet frontends (all two routes) \| 0.03 M \| 0.1% \| route-bound; downstream usually builds its own \|
	\| patchers (both rates) \| 0.20 M \| 0.6% \| route-bound \|
	\| spec decoder \| 0.72 M \| 2.4% \| pretraining only \|
	\| raw decoder \| 0.72 M \| 2.4% \| pretraining only \|
	\| total \| 30.08 M \| 100% \| \|

	The shared transformer is 94% of the model and the two frontends together are 0%, which is why there is no per-route encoder file here: two of them would duplicate that transformer two times to deliver frontends that are a small fraction of it. Cut the one you want out of the complete checkpoint instead.

	### What transfers, and what does not

	`downstream.py` splits a checkpoint into two sets. The channel embedding, its
	gate and the shared transformer always load. The wavelet frontend and the
	patcher load only when your montage is the route they were trained for --
	same channel count, same patch length -- and are skipped otherwise rather
	than reshaped.

	A 12-lead recording at 500 Hz in 10 s windows IS the `L12_500` route, so on the most common ECG input the pretrained frontend and patcher transfer as well: cut that route out (below) and pass `route_id="L12_500"`. Any other lead set, rate or window length builds its own.

	The two routes the shared transformer was pretrained across:

	\| route \| channels \| rate \|
	\|---\|---\|---\|
	\| `L12_500` \| 12 \| 500 Hz \|
	\| `L1_250` \| 1 \| 250 Hz \|

	To fine-tune ON one of them exactly, take that route's frontend and patcher out
	of the complete checkpoint:

	```bash
	python scripts/export_eeg_pretrained_encoder.py \
	--checkpoint pretrain_ecg_c1_moe_best.pth --route L12_500 \
	--output encoder_L12_500.pth
	```

	The exporter reads the modality from the checkpoint; its name is historical.
	Omit `--route` and you get `ecg_c1_encoder.pth` again. Both pretraining decoders are
	dropped either way: fine-tuning against the pretext objective by accident is
	the failure that prevents.

	The released checkpoint and the last epoch differ by 2.6% in validation total loss (0.1827 against 0.1877). Which transfers better is a downstream question, not a reconstruction-loss question -- measure both.

	## The channel vocabulary travels with the weights

	An embedding row means whatever channel held that id when the row was learned.
	Every checkpoint here carries the vocabulary and its hash. Resolving
	lead names against a different vocabulary silently trains on relabelled
	channels, which is why the hash is checked rather than assumed.

	## Architecture

	```
	embed_dim 512 depth 9 heads 8
	norm rmsnorm ffn swiglu qk_norm True
	wavelet max_level 3, kernel 16, bank db4, db6, sym4, sym8, coif2
	channel id embedding, injected at the token site
	mask frequency_guided, ratio 0.7
	```

	## Loading

	`ECGC1Downstream` builds the frontend and patcher for your montage and loads this
	file into everything else. Nothing here is route-bound, so the montage, the
	window length and the patch length are yours to choose.

	```python
	import torch
	from physiowave.ecg_c1.downstream import ECGC1Downstream

	ENC = "ecg_c1_encoder.pth"
	ck = torch.load(ENC, map_location="cpu", weights_only=False)
	mc = ck["model_config"] # the width the weights were trained at
	CH = ["I", "II", "III", "aVR", "aVL", "aVF",
	"V1", "V2", "V3", "V4", "V5", "V6"] # your leads

	model = ECGC1Downstream(
	in_channels=len(CH), window_samples=5000, sampling_rate=500.0,
	patch_samples=250, num_classes=2, channel_names=CH,
	embed_dim=mc["embed_dim"], depth=mc["depth"], num_heads=mc["num_heads"],
	channel_embed_dim=mc.get("channel_embed_dim", 64),
	channel_vocab_size=ck["channel_vocab_size"])

	report = model.load_pretrained(ENC)
	print(model.describe_transfer(report))
	# loaded: transformer N, channel encoder N, gate yes, frontend 0, patcher 0

	class Meta: # or your dataset's ChannelMeta
	channel_names = CH
	channel_mask = None

	out = model(torch.randn(2, len(CH), 5000), Meta()) # {"logits": [2, 2]}
	```

	`frontend 0, patcher 0` is the expected line, not a warning: those are the two
	things your montage supplies.

	The frontend you build is not randomly initialised. Its filters are seeded
	from real wavelet families -- `db4, db6, sym4, sym8, coif2` -- and learned from there during
	fine-tuning. What a fresh frontend costs you is the adaptation those filters
	underwent during pretraining, not the wavelet prior itself.

	Names, not indices, are what this encoder and your data agree on. Pass real
	lead names and check `channel_vocab_sha256` -- `load_pretrained` refuses a
	mismatch rather than silently training on relabelled channels.

	Code: <https://github.com/ForeverBlue816/PhysioWave>

	---
	Prepared by `scripts/prepare_hf_release.py` from `pretrain_ecg_c1_moe`
	on 2026-10-29.