TamAko783/anomalies-diffusion-andi

ANDi-style normative diffusion model for unsupervised brain-MRI anomaly localisation

Research only. Not validated for clinical use.

Experimental Machines. Aggregated normative diffusion model (independent re-implementation of the idea in arXiv:2312.01904; uadcal.detectors.andi.DenoiseUNet, EMA weights) trained on healthy IXI T2 (319 subjects). The upstream repository has no licence, so no upstream code was copied. Differences from the published setup (so numbers are not comparable to published ANDi results): smaller U-Net (6.43 M parameters), 128x128 slices, plain Gaussian noise, 20,000 steps only, scoring stride 10, geometric-mean aggregation of per-timestep errors over timesteps 125 to 26. Weights: best_model.safetensors (25.8 MB, step 20000). A three-times-longer (60k-step) continuation is in anomalies-diffusion-andi-60k (test AP 0.090 vs 0.084 here, paired gain +0.0056, CI excludes zero).

Selection and held-out results

• Checkpoint chosen on 10 MSLUB validation subjects by mean per-scan voxel AP: 0.055 / 0.070 / 0.075 / 0.081 at steps 5k / 10k / 15k / 20k. Still rising at 20k steps, so this model is under-trained.
• Held-out MSLUB test (20 scans, mean [95% bootstrap CI over subjects]):

Metric	ANDi	DAE	Intensity baseline
Voxel AP	0.084 [0.057, 0.114]	0.079 [0.036, 0.132]	0.022
Dice (validation-tuned threshold)	0.120 [0.084, 0.157]	0.107	0.048
Scan AUROC (domain + lesion)	0.701	0.600	0.655
Lesion-component recall	22.4%	8.4%	42.7%
Seconds per volume (L4)	4.84	0.23	0.02

Paired ANDi minus DAE: AP +0.006 [-0.045, 0.050] and Dice +0.013 [-0.041, 0.064] (not distinguishable from zero with 20 subjects); AUROC +0.100 [0.060, 0.140] (ANDi better on 15 of 20 scans). So at 21x the inference cost, ANDi did not demonstrably beat the DAE on localisation in this setting. IXI and MSLUB differ in scanner and population; scan-level separation is confounded by dataset shift.

Healthy-only calibration audit

See anomalies-diffusion-calibration: realised FPR 0.047 (alpha 0.05) and 0.128 (alpha 0.10) on 86 healthy scans, both inside the 99% validity band; healthy-calibrated thresholds give MSLUB Dice 0.012 (alpha 0.05); group-bias claims are not supported at this sample size (one exploratory nominal association, ANDi age 40-59 vs <40 at alpha 0.10, uncorrected).

Limitations

One training site mix, 20 held-out lesion scans, one lesion population (multiple sclerosis), under-trained model, noisy checkpoint selection. Weights labelled CC BY-SA 3.0 out of caution because IXI is CC BY-SA 3.0.

Project results, time and cost (all experiments)

All runs use Runpod: one Pod at a time, one GPU per Pod (an NVIDIA L4, or an equivalent fallback GPU when the L4 was unavailable), controlled from a laptop. "Billed" amounts are from Runpod's billing records for the specific Pods; "est." amounts are wall-clock time x list price where the Pod could not be itemised.

Time and cost

#	Stage	Date	Pod time	Cost (USD)	Basis
1	DAE baseline: preprocess, train 20k steps, benchmark, upload	Nov 1	~42 min	~0.41	est.
2	ANDi 20k steps: train, benchmark, paired comparison with DAE	Nov 1	~1 h 50 min	~1.08	est.
3	Healthy-only calibration audit (DAE, ANDi-20k)	Nov 1	~1 h 11 min	0.68	billed
4	ANDi continued to 60k steps, benchmark	Nov 1-2	~3 h 16 min	1.94	billed
5	qmr-v1 study, first Pod (hit its 4.5 h runtime cap during evaluation)	Nov 2	4 h 30 min	3.26	billed
6	qmr-v1 two resume attempts that exited at once (stale finished-job marker)	Nov 2	~6 min	0.15	billed
7	qmr-v1 resume under a fresh marker (completed)	Nov 2	~51 min	0.57	billed
8	qmr-v2: float32 audit, training replay, pAUC-vs-BCE fix study	Nov 3	~2 h 58 min	1.61	billed
	Total itemised (rows 1-8)		~15.4 h	~9.70	8.21 billed + 1.49 est.

Not itemised and therefore not in the total: several short code-sync Pods (a few cents each), earlier development Pods run before the one-Pod-at-a-time rule was added (some were duplicates), and storage on the shared network volume (Runpod bills the whole volume, which is shared with other uses; its share for this project was not measured). Cost of a single scan at inference on an L4 at list price: ANDi about 4.9 s per volume ($0.0008); DAE 0.23 s ($0.00004).

Results

All lesion results use 20 held-out MS scans (MSLUB, a different scanner vendor from the IXI training data) and are exploratory after the first look.

Experiment	Headline result	Strength
DAE baseline (not a diffusion model)	Voxel AP 0.079, Dice 0.107, scan AUROC 0.600, 0.23 s per volume	benchmark
ANDi diffusion model, 20k steps	AP 0.084, Dice 0.120, AUROC 0.701, 4.84 s per volume	benchmark
ANDi, 60k steps	AP 0.090, Dice 0.125, AUROC 0.707; +0.0056 AP vs 20k (CI +0.002 to +0.010); vs DAE AP +0.011 (CI -0.036 to +0.056, not significant)	small gain; ~21x the DAE's cost
Healthy-only conformal calibration (guarantee)	False-positive rate inside the expected band on 86 healthy test scans for every detector tested	solid (healthy data only)
Group bias audit (site, age, sex)	No confirmed group disparity; one uncorrected nominal signal (ANDi, age 40-59 vs <40)	underpowered
Healthy-calibrated threshold on MS lesions	Dice 0.012 (ANDi-20k) and 0.032 (DAE) in the audit; ~0.02 for raw scores of every detector in qmr-v1	consistent across runs; likely scanner shift (untested)
qmr-v1: calibration-aligned objective (QMR)	Failed: Dice 0.000 at the healthy-calibrated threshold; worse than its BCE ablation (-0.0007)	clear
qmr-v1: per-voxel normalisation (CSS)	DAE Dice 0.023 -> 0.052 (+0.029, CI +0.013 to +0.048); no detectable effect for ANDi	one clear positive, n = 20
qmr-v1: synthetic lesions for model selection	Not a substitute for real labels (Dice 0.03-0.05 lower); selection signal nearly flat	exploratory
qmr-v2: float32 audit	Float16 sigmoid ceiling explains the exact zero (thresholds became finite) but not the poor detection (Dice ~0.002)	clear
qmr-v2: training replay	Learned threshold trailed the healthy tail while both saturated (one seed)	supports the failure mechanism in that run
qmr-v2: pre-specified fix (partial-AUC)	Failed: collapsed to a constant score; MSLUB Dice 0.0001 vs 0.021 for BCE on logits	clear
qmr-v2: BCE on float32 logits	Dice 0.029 and 0.014 (mean 0.021; DAE 0.023); original seed re-scored gave 0.0015	exploratory; seed spread as large as the effect