TamAko783/anomalies-diffusion-calibration

Healthy-only conformal calibration and bias audit of unsupervised brain-MRI anomaly detectors

Research only. Not validated for clinical use. No MRI data is stored in this repository (aggregates only).

Experimental Machines. This repository holds the audit of two unsupervised detectors (anomalies-diffusion-dae-baseline, anomalies-diffusion-andi) using split-conformal calibration on healthy scans only.

Question

Can an anomaly detector's alarm threshold be set from healthy scans alone with a finite-sample guarantee on the false-positive rate, and does that guarantee hold equally across scanner site, age band and sex?

Method

• Per-scan score from the detector; conformal p-values from a held-out healthy calibration set (randomized and conservative variants; Bates et al., Ann. Stat. 2023), Mondrian (per-group) variants, BH / Storey-BH for batches, conformal risk control for a false-lesion-volume budget (Angelopoulos et al., ICLR 2024).
• G0 validity check: the realised false-positive rate on an independent healthy test set must fall inside the 99% binomial band around alpha.
• Threshold protocols: P2 = Dice-optimal on lesion-validation scans; P3 = healthy 99.5th voxel percentile; P4 = conformal-risk-control false-lesion guarantee at alpha 0.05 / 0.10.
• Group FPRs by site, age band and sex, plus a site/age/sex-adjusted logistic model of the per-scan flag (audit.json).

Data

Role	Cohort	Scans
Detector training (earlier runs)	IXI T2, healthy	319
Conformal calibration	IXI T2, healthy (3 sites: Guys, HH, IOP)	173
Healthy test	IXI T2, healthy	86
Lesion validation / test	MSLUB T2 (multiple sclerosis, Ljubljana)	10 / 20

Splits are subject-level and hashed; the pipeline asserts no training subject appears in calibration or test. IXI is CC BY-SA 3.0; MSLUB is CC-BY. Derived artifacts are labelled CC BY-SA 3.0 out of caution (whether trained weights are adaptations of the data is unsettled).

Results (healthy IXI test set, n = 86; checkpoints chosen earlier on MSLUB validation)

Check	DAE (step 5000)	ANDi (step 20000)
Realised FPR at alpha 0.05 (99% band 0 to 0.140)	0.035 (inside)	0.047 (inside)
Realised FPR at alpha 0.10 (99% band 0.023 to 0.209)	0.035 (inside)	0.128 (inside)
Site FPR at alpha 0.10, pooled threshold: Guys / HH / IOP (n = 61 / 18 / 7)	0.033 / 0.056 / 0.000	0.180 / 0.000 / 0.000
Age FPR at alpha 0.10: <40 / 40-59 / >=60 (n = 27 / 29 / 25)	0.111 / 0.000 / 0.000	0.259 / 0.034 / 0.120
MSLUB Dice, threshold P4 at alpha 0.05	0.032	0.012
MSLUB Dice, threshold P3 (healthy percentile)	0.096	0.111
MSLUB Dice, threshold P2 (lesion-validation tuned)	0.107	0.121

Group values are point estimates; intervals are in audit.json.

What this supports

• The finite-sample false-positive guarantee holds empirically for both detectors (G0 passes). Healthy-only calibration works as a validity check.

What it does not support

• No confirmed group-bias claim. The one nominally significant term is ANDi at alpha 0.10, age 40-59 vs <40 (odds ratio 0.045, 95% CI 0.005 to 0.46, p = 0.009; 81 scans in the adjusted logistic model). It is one of many exploratory tests, uncorrected for multiplicity, with few flagged scans: a hypothesis for a larger cohort, not a finding. Other group differences are not distinguishable from noise at these sizes; the IOP site has 7 test scans and is not estimable.
• The DAE flag-model regression is degenerate (perfect separation; 36 non-finite values stored as null). Ignore it.
• DAE scan-level AUROC 0.99 (MSLUB vs IXI) reflects dataset shift (scanner/population), not lesion detection.
• Healthy-calibrated (P4) thresholds make the detectors nearly silent on MSLUB (Dice 0.012 to 0.032; batch-BH power 0 for ANDi). [Inference, untested] the IXI-calibrated threshold sits far from the MSLUB lesion-score range after domain shift.
• The checkpoints were chosen with MSLUB validation labels and MSLUB test results had already been seen: this is not a pre-registered confirmatory test.
• DAE is a denoising autoencoder, not a diffusion model. ANDi is an independent re-implementation (smaller network, 128x128, far fewer steps): numbers are not comparable to published ANDi results.

Reproducibility

• Code commits at run time: DAE fe81d3a10768, ANDi b32cb05f5648, calibration audit c841580910c2 (code is not published yet; ask the authors).
• Splits, checkpoint SHA-256s and subject-list hashes: dae_summary.json, andi_summary.json and the PDF.
• Files: calibration_audit.pdf (full report), audit.json, dae_summary.json, andi_summary.json.

Project results, time and cost (all experiments)

All runs use Runpod: one Pod at a time, one GPU per Pod (an NVIDIA L4, or an equivalent fallback GPU when the L4 was unavailable), controlled from a laptop. "Billed" amounts are from Runpod's billing records for the specific Pods; "est." amounts are wall-clock time x list price where the Pod could not be itemised.

Time and cost

#	Stage	Date	Pod time	Cost (USD)	Basis
1	DAE baseline: preprocess, train 20k steps, benchmark, upload	Nov 1	~42 min	~0.41	est.
2	ANDi 20k steps: train, benchmark, paired comparison with DAE	Nov 1	~1 h 50 min	~1.08	est.
3	Healthy-only calibration audit (DAE, ANDi-20k)	Nov 1	~1 h 11 min	0.68	billed
4	ANDi continued to 60k steps, benchmark	Nov 1-2	~3 h 16 min	1.94	billed
5	qmr-v1 study, first Pod (hit its 4.5 h runtime cap during evaluation)	Nov 2	4 h 30 min	3.26	billed
6	qmr-v1 two resume attempts that exited at once (stale finished-job marker)	Nov 2	~6 min	0.15	billed
7	qmr-v1 resume under a fresh marker (completed)	Nov 2	~51 min	0.57	billed
8	qmr-v2: float32 audit, training replay, pAUC-vs-BCE fix study	Nov 3	~2 h 58 min	1.61	billed
	Total itemised (rows 1-8)		~15.4 h	~9.70	8.21 billed + 1.49 est.

Not itemised and therefore not in the total: several short code-sync Pods (a few cents each), earlier development Pods run before the one-Pod-at-a-time rule was added (some were duplicates), and storage on the shared network volume (Runpod bills the whole volume, which is shared with other uses; its share for this project was not measured). Cost of a single scan at inference on an L4 at list price: ANDi about 4.9 s per volume ($0.0008); DAE 0.23 s ($0.00004).

Results

All lesion results use 20 held-out MS scans (MSLUB, a different scanner vendor from the IXI training data) and are exploratory after the first look.

Experiment	Headline result	Strength
DAE baseline (not a diffusion model)	Voxel AP 0.079, Dice 0.107, scan AUROC 0.600, 0.23 s per volume	benchmark
ANDi diffusion model, 20k steps	AP 0.084, Dice 0.120, AUROC 0.701, 4.84 s per volume	benchmark
ANDi, 60k steps	AP 0.090, Dice 0.125, AUROC 0.707; +0.0056 AP vs 20k (CI +0.002 to +0.010); vs DAE AP +0.011 (CI -0.036 to +0.056, not significant)	small gain; ~21x the DAE's cost
Healthy-only conformal calibration (guarantee)	False-positive rate inside the expected band on 86 healthy test scans for every detector tested	solid (healthy data only)
Group bias audit (site, age, sex)	No confirmed group disparity; one uncorrected nominal signal (ANDi, age 40-59 vs <40)	underpowered
Healthy-calibrated threshold on MS lesions	Dice 0.012 (ANDi-20k) and 0.032 (DAE) in the audit; ~0.02 for raw scores of every detector in qmr-v1	consistent across runs; likely scanner shift (untested)
qmr-v1: calibration-aligned objective (QMR)	Failed: Dice 0.000 at the healthy-calibrated threshold; worse than its BCE ablation (-0.0007)	clear
qmr-v1: per-voxel normalisation (CSS)	DAE Dice 0.023 -> 0.052 (+0.029, CI +0.013 to +0.048); no detectable effect for ANDi	one clear positive, n = 20
qmr-v1: synthetic lesions for model selection	Not a substitute for real labels (Dice 0.03-0.05 lower); selection signal nearly flat	exploratory
qmr-v2: float32 audit	Float16 sigmoid ceiling explains the exact zero (thresholds became finite) but not the poor detection (Dice ~0.002)	clear
qmr-v2: training replay	Learned threshold trailed the healthy tail while both saturated (one seed)	supports the failure mechanism in that run
qmr-v2: pre-specified fix (partial-AUC)	Failed: collapsed to a constant score; MSLUB Dice 0.0001 vs 0.021 for BCE on logits	clear
qmr-v2: BCE on float32 logits	Dice 0.029 and 0.014 (mean 0.021; DAE 0.023); original seed re-scored gave 0.0015	exploratory; seed spread as large as the effect