TamAko783/anomalies-diffusion-dae-baseline

DAE baseline for unsupervised brain-MRI anomaly localisation

Research only. Not validated for clinical use. Not a diffusion model.

Experimental Machines. Coarse-noise denoising autoencoder (U-Net, uadcal.detectors.dae.UNet) trained on healthy IXI T2 (319 subjects, 20,000 steps). It is the cheap baseline against which the diffusion detector (anomalies-diffusion-andi) is compared. Weights: best_model.safetensors (7.7 MB; step 5000).

Selection and held-out results

• Checkpoint chosen on 10 MSLUB validation subjects by mean per-scan voxel average precision (AP): 0.043 (step 5000), 0.030 (10000), 0.019 (15000), 0.026 (20000). Selection is noisy at n = 10.
• Held-out MSLUB test (20 scans, mean [95% bootstrap CI over subjects]):

Metric	DAE	Intensity baseline
Voxel AP	0.079 [0.036, 0.132]	0.022
Dice (validation-tuned threshold)	0.107	0.048
Scan AUROC (domain + lesion)	0.600	0.655
Lesion-component recall	8.4%	42.7%
Seconds per volume (L4)	0.23	0.02

IXI (training) and MSLUB (test) differ in scanner and population, so scan-level separation is confounded by dataset shift. report.pdf and summary.json hold the full benchmark.

Healthy-only calibration audit

See anomalies-diffusion-calibration: the conformal false-positive guarantee holds empirically (realised FPR 0.035 at alpha 0.05 and 0.10 on 86 healthy scans), but healthy-calibrated thresholds leave the detector almost silent on MSLUB (Dice 0.032) and no group-bias claim is supported at this sample size.

Limitations

Single training site mix (IXI Guys/HH/IOP), 20 held-out lesion scans, one lesion population (multiple sclerosis), research checkpoint selection on 10 scans. Weights are labelled CC BY-SA 3.0 out of caution because IXI is CC BY-SA 3.0.

Project results, time and cost (all experiments)

All runs use Runpod: one Pod at a time, one GPU per Pod (an NVIDIA L4, or an equivalent fallback GPU when the L4 was unavailable), controlled from a laptop. "Billed" amounts are from Runpod's billing records for the specific Pods; "est." amounts are wall-clock time x list price where the Pod could not be itemised.

Time and cost

#	Stage	Date	Pod time	Cost (USD)	Basis
1	DAE baseline: preprocess, train 20k steps, benchmark, upload	Nov 1	~42 min	~0.41	est.
2	ANDi 20k steps: train, benchmark, paired comparison with DAE	Nov 1	~1 h 50 min	~1.08	est.
3	Healthy-only calibration audit (DAE, ANDi-20k)	Nov 1	~1 h 11 min	0.68	billed
4	ANDi continued to 60k steps, benchmark	Nov 1-2	~3 h 16 min	1.94	billed
5	qmr-v1 study, first Pod (hit its 4.5 h runtime cap during evaluation)	Nov 2	4 h 30 min	3.26	billed
6	qmr-v1 two resume attempts that exited at once (stale finished-job marker)	Nov 2	~6 min	0.15	billed
7	qmr-v1 resume under a fresh marker (completed)	Nov 2	~51 min	0.57	billed
8	qmr-v2: float32 audit, training replay, pAUC-vs-BCE fix study	Nov 3	~2 h 58 min	1.61	billed
	Total itemised (rows 1-8)		~15.4 h	~9.70	8.21 billed + 1.49 est.

Not itemised and therefore not in the total: several short code-sync Pods (a few cents each), earlier development Pods run before the one-Pod-at-a-time rule was added (some were duplicates), and storage on the shared network volume (Runpod bills the whole volume, which is shared with other uses; its share for this project was not measured). Cost of a single scan at inference on an L4 at list price: ANDi about 4.9 s per volume ($0.0008); DAE 0.23 s ($0.00004).

Results

All lesion results use 20 held-out MS scans (MSLUB, a different scanner vendor from the IXI training data) and are exploratory after the first look.

Experiment	Headline result	Strength
DAE baseline (not a diffusion model)	Voxel AP 0.079, Dice 0.107, scan AUROC 0.600, 0.23 s per volume	benchmark
ANDi diffusion model, 20k steps	AP 0.084, Dice 0.120, AUROC 0.701, 4.84 s per volume	benchmark
ANDi, 60k steps	AP 0.090, Dice 0.125, AUROC 0.707; +0.0056 AP vs 20k (CI +0.002 to +0.010); vs DAE AP +0.011 (CI -0.036 to +0.056, not significant)	small gain; ~21x the DAE's cost
Healthy-only conformal calibration (guarantee)	False-positive rate inside the expected band on 86 healthy test scans for every detector tested	solid (healthy data only)
Group bias audit (site, age, sex)	No confirmed group disparity; one uncorrected nominal signal (ANDi, age 40-59 vs <40)	underpowered
Healthy-calibrated threshold on MS lesions	Dice 0.012 (ANDi-20k) and 0.032 (DAE) in the audit; ~0.02 for raw scores of every detector in qmr-v1	consistent across runs; likely scanner shift (untested)
qmr-v1: calibration-aligned objective (QMR)	Failed: Dice 0.000 at the healthy-calibrated threshold; worse than its BCE ablation (-0.0007)	clear
qmr-v1: per-voxel normalisation (CSS)	DAE Dice 0.023 -> 0.052 (+0.029, CI +0.013 to +0.048); no detectable effect for ANDi	one clear positive, n = 20
qmr-v1: synthetic lesions for model selection	Not a substitute for real labels (Dice 0.03-0.05 lower); selection signal nearly flat	exploratory
qmr-v2: float32 audit	Float16 sigmoid ceiling explains the exact zero (thresholds became finite) but not the poor detection (Dice ~0.002)	clear
qmr-v2: training replay	Learned threshold trailed the healthy tail while both saturated (one seed)	supports the failure mechanism in that run
qmr-v2: pre-specified fix (partial-AUC)	Failed: collapsed to a constant score; MSLUB Dice 0.0001 vs 0.021 for BCE on logits	clear
qmr-v2: BCE on float32 logits	Dice 0.029 and 0.014 (mean 0.021; DAE 0.023); original seed re-scored gave 0.0015	exploratory; seed spread as large as the effect