TamAko783
/

anomalies-diffusion-calibration

@@ -1,97 +1,114 @@
----
-license: cc-by-sa-3.0
-tags: [medical-imaging, anomaly-detection, mri, conformal-prediction, fairness-audit, reproducibility]
----
-# Healthy-only conformal calibration and bias audit of unsupervised brain-MRI anomaly detectors
-**Research only. Not validated for clinical use. No MRI data is stored in this repository** (aggregates only).
-Experimental Machines (non-profit). This repository holds the audit of two unsupervised detectors
-(`anomalies-diffusion-dae-baseline`, `anomalies-diffusion-andi`) using split-conformal calibration on healthy scans only.
-## Question
-Can an anomaly detector's alarm threshold be set from **healthy scans alone** with a finite-sample guarantee on the
-false-positive rate, and does that guarantee hold equally across scanner site, age band and sex?
-## Method
-- Per-scan score from the detector; conformal p-values from a held-out healthy calibration set (randomized and
-  conservative variants; Bates et al., Ann. Stat. 2023), Mondrian (per-group) variants, BH / Storey-BH for batches,
-  conformal risk control for a false-lesion-volume budget (Angelopoulos et al., ICLR 2024).
-- **G0 validity check:** the realised false-positive rate on an independent healthy test set must fall inside the 99%
-  binomial band around alpha.
-- Threshold protocols: P2 = Dice-optimal on lesion-validation scans; P3 = healthy 99.5th voxel percentile;
-  P4 = conformal-risk-control false-lesion guarantee at alpha 0.05 / 0.10.
-- Group FPRs by site, age band and sex, plus a site/age/sex-adjusted logistic model of the per-scan flag (`audit.json`).
-## Data
-| Role | Cohort | Scans |
-|---|---|---|
-| Detector training (earlier runs) | IXI T2, healthy | 319 |
-| Conformal calibration | IXI T2, healthy (3 sites: Guys, HH, IOP) | 173 |
-| Healthy test | IXI T2, healthy | 86 |
-| Lesion validation / test | MSLUB T2 (multiple sclerosis, Ljubljana) | 10 / 20 |
-Splits are subject-level and hashed; the pipeline asserts no training subject appears in calibration or test.
-IXI is CC BY-SA 3.0; MSLUB is CC-BY. Derived artifacts are labelled CC BY-SA 3.0 out of caution (whether trained weights
-are adaptations of the data is unsettled).
-## Results (healthy IXI test set, n = 86; checkpoints chosen earlier on MSLUB validation)
-| Check | DAE (step 5000) | ANDi (step 20000) |
-|---|---|---|
-| Realised FPR at alpha 0.05 (99% band 0 to 0.140) | 0.035 (inside) | 0.047 (inside) |
-| Realised FPR at alpha 0.10 (99% band 0.023 to 0.209) | 0.035 (inside) | 0.128 (inside) |
-| Site FPR at alpha 0.10, pooled threshold: Guys / HH / IOP (n = 61 / 18 / 7) | 0.033 / 0.056 / 0.000 | 0.180 / 0.000 / 0.000 |
-| Age FPR at alpha 0.10: <40 / 40-59 / >=60 (n = 27 / 29 / 25) | 0.111 / 0.000 / 0.000 | 0.259 / 0.034 / 0.120 |
-| MSLUB Dice, threshold P4 at alpha 0.05 | 0.032 | 0.012 |
-| MSLUB Dice, threshold P3 (healthy percentile) | 0.096 | 0.111 |
-| MSLUB Dice, threshold P2 (lesion-validation tuned) | 0.107 | 0.121 |
-Group values are point estimates; intervals are in `audit.json`.
-### What this supports
-- The finite-sample false-positive guarantee holds empirically for both detectors (G0 passes). Healthy-only calibration
-  works as a validity check.
-### What it does not support
-- **No confirmed group-bias claim.** The one nominally significant term is ANDi at alpha 0.10, age 40-59 vs <40
-  (odds ratio 0.045, 95% CI 0.005 to 0.46, p = 0.009; 81 scans in the adjusted logistic model). It is one of many
-  exploratory tests, uncorrected for multiplicity, with few flagged scans: a hypothesis for a larger cohort, not a
-  finding. Other group differences are not distinguishable from noise at these sizes; the IOP site has 7 test scans and is
-  not estimable.
-- The DAE flag-model regression is degenerate (perfect separation; 36 non-finite values stored as null). Ignore it.
-- DAE scan-level AUROC 0.99 (MSLUB vs IXI) reflects dataset shift (scanner/population), not lesion detection.
-- Healthy-calibrated (P4) thresholds make the detectors nearly silent on MSLUB (Dice 0.012 to 0.032; batch-BH power 0
-  for ANDi). [Inference, untested] the IXI-calibrated threshold sits far from the MSLUB lesion-score range after domain shift.
-- The checkpoints were chosen with MSLUB validation labels and MSLUB test results had already been seen: this is **not** a
-  pre-registered confirmatory test.
-- DAE is a denoising autoencoder, not a diffusion model. ANDi is an independent re-implementation (smaller network,
-  128x128, far fewer steps): numbers are not comparable to published ANDi results.
-## Compute and cloud cost (Runpod Secure Cloud, EU-RO-1)
-One Pod per stage, no helper or duplicate Pods, enforced in code (`UADCAL_MAX_PODS=1`, duplicate-name refusal, live Pod-list
-check), plus a runtime cap, self-terminating jobs and an external reaper. Results moved through the Runpod control plane and a
-network volume: no object storage, no egress fees.
-| Stage | GPU (list price/h) | Wall time | Estimated cost |
-|---|---|---|---|
-| DAE train (20k steps) + benchmark + PDF + upload | NVIDIA L4 ($0.59) | ~42 min (fit 17 min) | ~$0.41 |
-| ANDi train (20k steps) + benchmark + paired comparison | NVIDIA L4 ($0.59) | ~1 h 50 min (fit 89.5 min) | ~$1.08 |
-| Calibration audit (preprocess 259 IXI scans, score 289 scans with 2 detectors, audit, PDF) | not recorded (see notes) | ~70 min | ~$0.69 if L4; $0.28 to $1.05 across the allowed GPUs |
-| **Total for these three results** | | **~3 h 42 min** | **~$2.2** |
-Notes for reviewers:
-- Costs are **wall-clock x list price**, not an invoice. The GPU actually allocated to the calibration Pod was not logged
-  (the launcher tried L4 first, then other GPUs when out of stock); the range covers every allowed GPU.
-- The total excludes earlier development Pods run before the one-Pod discipline was added, including duplicates later
-  forbidden in code. They are not itemised here; the Runpod invoice is authoritative.
-- The shared 550 GB network volume bills $0.07/GB-month whether or not a Pod runs; this project's share was not measured.
-- Per-unit cost on an L4 at $0.59/h: ANDi inference ~$0.0008 per volume (4.84 s), DAE ~$0.00004 (0.23 s); preprocessing
-  (N4 + HD-BET + registration, largely CPU-bound) ~25-30 s per scan, ~$0.004 per scan.
-- Availability mattered as much as price: L4 and RTX 4090 capacity in the one datacenter the volume pins us to were each
-  unavailable at times. The client now waits for capacity (a rejected create request creates and bills nothing) instead of failing.
-## Reproducibility
-- Code commits at run time: DAE `fe81d3a10768`, ANDi `b32cb05f5648`, calibration audit `c841580910c2`
-  (code is not published yet; ask the authors).
-- Splits, checkpoint SHA-256s and subject-list hashes: `dae_summary.json`, `andi_summary.json` and the PDF.
-- Files: `calibration_audit.pdf` (full report), `audit.json`, `dae_summary.json`, `andi_summary.json`.

+---
+license: cc-by-sa-3.0
+tags: [medical-imaging, anomaly-detection, mri, conformal-prediction, fairness-audit, reproducibility]
+---
+# Healthy-only conformal calibration and bias audit of unsupervised brain-MRI anomaly detectors
+**Research only. Not validated for clinical use. No MRI data is stored in this repository** (aggregates only).
+Experimental Machines. This repository holds the audit of two unsupervised detectors
+(`anomalies-diffusion-dae-baseline`, `anomalies-diffusion-andi`) using split-conformal calibration on healthy scans only.
+## Question
+Can an anomaly detector's alarm threshold be set from **healthy scans alone** with a finite-sample guarantee on the
+false-positive rate, and does that guarantee hold equally across scanner site, age band and sex?
+## Method
+- Per-scan score from the detector; conformal p-values from a held-out healthy calibration set (randomized and
+  conservative variants; Bates et al., Ann. Stat. 2023), Mondrian (per-group) variants, BH / Storey-BH for batches,
+  conformal risk control for a false-lesion-volume budget (Angelopoulos et al., ICLR 2024).
+- **G0 validity check:** the realised false-positive rate on an independent healthy test set must fall inside the 99%
+  binomial band around alpha.
+- Threshold protocols: P2 = Dice-optimal on lesion-validation scans; P3 = healthy 99.5th voxel percentile;
+  P4 = conformal-risk-control false-lesion guarantee at alpha 0.05 / 0.10.
+- Group FPRs by site, age band and sex, plus a site/age/sex-adjusted logistic model of the per-scan flag (`audit.json`).
+## Data
+| Role | Cohort | Scans |
+|---|---|---|
+| Detector training (earlier runs) | IXI T2, healthy | 319 |
+| Conformal calibration | IXI T2, healthy (3 sites: Guys, HH, IOP) | 173 |
+| Healthy test | IXI T2, healthy | 86 |
+| Lesion validation / test | MSLUB T2 (multiple sclerosis, Ljubljana) | 10 / 20 |
+Splits are subject-level and hashed; the pipeline asserts no training subject appears in calibration or test.
+IXI is CC BY-SA 3.0; MSLUB is CC-BY. Derived artifacts are labelled CC BY-SA 3.0 out of caution (whether trained weights
+are adaptations of the data is unsettled).
+## Results (healthy IXI test set, n = 86; checkpoints chosen earlier on MSLUB validation)
+| Check | DAE (step 5000) | ANDi (step 20000) |
+|---|---|---|
+| Realised FPR at alpha 0.05 (99% band 0 to 0.140) | 0.035 (inside) | 0.047 (inside) |
+| Realised FPR at alpha 0.10 (99% band 0.023 to 0.209) | 0.035 (inside) | 0.128 (inside) |
+| Site FPR at alpha 0.10, pooled threshold: Guys / HH / IOP (n = 61 / 18 / 7) | 0.033 / 0.056 / 0.000 | 0.180 / 0.000 / 0.000 |
+| Age FPR at alpha 0.10: <40 / 40-59 / >=60 (n = 27 / 29 / 25) | 0.111 / 0.000 / 0.000 | 0.259 / 0.034 / 0.120 |
+| MSLUB Dice, threshold P4 at alpha 0.05 | 0.032 | 0.012 |
+| MSLUB Dice, threshold P3 (healthy percentile) | 0.096 | 0.111 |
+| MSLUB Dice, threshold P2 (lesion-validation tuned) | 0.107 | 0.121 |
+Group values are point estimates; intervals are in `audit.json`.
+### What this supports
+- The finite-sample false-positive guarantee holds empirically for both detectors (G0 passes). Healthy-only calibration
+  works as a validity check.
+### What it does not support
+- **No confirmed group-bias claim.** The one nominally significant term is ANDi at alpha 0.10, age 40-59 vs <40
+  (odds ratio 0.045, 95% CI 0.005 to 0.46, p = 0.009; 81 scans in the adjusted logistic model). It is one of many
+  exploratory tests, uncorrected for multiplicity, with few flagged scans: a hypothesis for a larger cohort, not a
+  finding. Other group differences are not distinguishable from noise at these sizes; the IOP site has 7 test scans and is
+  not estimable.
+- The DAE flag-model regression is degenerate (perfect separation; 36 non-finite values stored as null). Ignore it.
+- DAE scan-level AUROC 0.99 (MSLUB vs IXI) reflects dataset shift (scanner/population), not lesion detection.
+- Healthy-calibrated (P4) thresholds make the detectors nearly silent on MSLUB (Dice 0.012 to 0.032; batch-BH power 0
+  for ANDi). [Inference, untested] the IXI-calibrated threshold sits far from the MSLUB lesion-score range after domain shift.
+- The checkpoints were chosen with MSLUB validation labels and MSLUB test results had already been seen: this is **not** a
+  pre-registered confirmatory test.
+- DAE is a denoising autoencoder, not a diffusion model. ANDi is an independent re-implementation (smaller network,
+  128x128, far fewer steps): numbers are not comparable to published ANDi results.
+## Reproducibility
+- Code commits at run time: DAE `fe81d3a10768`, ANDi `b32cb05f5648`, calibration audit `c841580910c2`
+  (code is not published yet; ask the authors).
+- Splits, checkpoint SHA-256s and subject-list hashes: `dae_summary.json`, `andi_summary.json` and the PDF.
+- Files: `calibration_audit.pdf` (full report), `audit.json`, `dae_summary.json`, `andi_summary.json`.
+## Project results, time and cost (all experiments)
+All runs use Runpod: one Pod at a time, one GPU per Pod (an NVIDIA L4, or an equivalent fallback GPU when the L4 was unavailable), controlled from a laptop.
+"Billed" amounts are from Runpod's billing records for the specific Pods; "est." amounts are wall-clock time x list price where the Pod could not be itemised.
+### Time and cost
+| # | Stage | Date | Pod time | Cost (USD) | Basis |
+|---|---|---|---|---|---|
+| 1 | DAE baseline: preprocess, train 20k steps, benchmark, upload | Nov 1 | ~42 min | ~0.41 | est. |
+| 2 | ANDi 20k steps: train, benchmark, paired comparison with DAE | Nov 1 | ~1 h 50 min | ~1.08 | est. |
+| 3 | Healthy-only calibration audit (DAE, ANDi-20k) | Nov 1 | ~1 h 11 min | 0.68 | billed |
+| 4 | ANDi continued to 60k steps, benchmark | Nov 1-2 | ~3 h 16 min | 1.94 | billed |
+| 5 | qmr-v1 study, first Pod (hit its 4.5 h runtime cap during evaluation) | Nov 2 | 4 h 30 min | 3.26 | billed |
+| 6 | qmr-v1 two resume attempts that exited at once (stale finished-job marker) | Nov 2 | ~6 min | 0.15 | billed |
+| 7 | qmr-v1 resume under a fresh marker (completed) | Nov 2 | ~51 min | 0.57 | billed |
+| 8 | qmr-v2: float32 audit, training replay, pAUC-vs-BCE fix study | Nov 3 | ~2 h 58 min | 1.61 | billed |
+| | **Total itemised (rows 1-8)** | | **~15.4 h** | **~9.70** | 8.21 billed + 1.49 est. |
+Not itemised and therefore not in the total: several short code-sync Pods (a few cents each), earlier development Pods run before the one-Pod-at-a-time rule was added (some were duplicates), and storage on the shared network volume (Runpod bills the whole volume, which is shared with other uses; its share for this project was not measured).
+Cost of a single scan at inference on an L4 at list price: ANDi about 4.9 s per volume (~$0.0008); DAE 0.23 s (~$0.00004).
+### Results
+All lesion results use 20 held-out MS scans (MSLUB, a different scanner vendor from the IXI training data) and are exploratory after the first look.
+| Experiment | Headline result | Strength |
+|---|---|---|
+| DAE baseline (not a diffusion model) | Voxel AP 0.079, Dice 0.107, scan AUROC 0.600, 0.23 s per volume | benchmark |
+| ANDi diffusion model, 20k steps | AP 0.084, Dice 0.120, AUROC 0.701, 4.84 s per volume | benchmark |
+| ANDi, 60k steps | AP 0.090, Dice 0.125, AUROC 0.707; +0.0056 AP vs 20k (CI +0.002 to +0.010); vs DAE AP +0.011 (CI -0.036 to +0.056, not significant) | small gain; ~21x the DAE's cost |
+| Healthy-only conformal calibration (guarantee) | False-positive rate inside the expected band on 86 healthy test scans for every detector tested | solid (healthy data only) |
+| Group bias audit (site, age, sex) | No confirmed group disparity; one uncorrected nominal signal (ANDi, age 40-59 vs <40) | underpowered |
+| Healthy-calibrated threshold on MS lesions | Dice 0.012 (ANDi-20k) and 0.032 (DAE) in the audit; ~0.02 for raw scores of every detector in qmr-v1 | consistent across runs; likely scanner shift (untested) |
+| qmr-v1: calibration-aligned objective (QMR) | **Failed:** Dice 0.000 at the healthy-calibrated threshold; worse than its BCE ablation (-0.0007) | clear |
+| qmr-v1: per-voxel normalisation (CSS) | DAE Dice 0.023 -> 0.052 (+0.029, CI +0.013 to +0.048); no detectable effect for ANDi | one clear positive, n = 20 |
+| qmr-v1: synthetic lesions for model selection | Not a substitute for real labels (Dice 0.03-0.05 lower); selection signal nearly flat | exploratory |
+| qmr-v2: float32 audit | Float16 sigmoid ceiling explains the exact zero (thresholds became finite) but not the poor detection (Dice ~0.002) | clear |
+| qmr-v2: training replay | Learned threshold trailed the healthy tail while both saturated (one seed) | supports the failure mechanism in that run |
+| qmr-v2: pre-specified fix (partial-AUC) | **Failed:** collapsed to a constant score; MSLUB Dice 0.0001 vs 0.021 for BCE on logits | clear |
+| qmr-v2: BCE on float32 logits | Dice 0.029 and 0.014 (mean 0.021; DAE 0.023); original seed re-scored gave 0.0015 | exploratory; seed spread as large as the effect |