Spaces:
Running on Zero
Download docs/PHASE7_POST_EXECUTION_AUDIT.md from Thiru1122/Multimodel_setup: direct link, hf CLI and curl.
- Browser
- Download file 37 kB
-
https://huggingface.co/spaces/Thiru1122/Multimodel_setup/resolve/main/docs/PHASE7_POST_EXECUTION_AUDIT.md
- Command line
-
hf download hf://spaces/Thiru1122/Multimodel_setup/docs/PHASE7_POST_EXECUTION_AUDIT.md
-
curl -L -o PHASE7_POST_EXECUTION_AUDIT.md https://huggingface.co/spaces/Thiru1122/Multimodel_setup/resolve/main/docs/PHASE7_POST_EXECUTION_AUDIT.md
A newer version of the Gradio SDK is available: 6.32.0
Phase 7 β Comprehensive Post-Execution Verification and Research Audit
Cross-Model Evaluation of Deepfake Image Detection Under Diverse Image Generation and Manipulation Conditions
Document Path: docs/PHASE7_POST_EXECUTION_AUDIT.md
Audit Date: November 2026
Auditor Mode: Independent, Read-Only Scientific & Implementation Verification
Primary Deliverable Target: Phase 7 External AI-Generated Image Generalization Evaluation
Evaluation Benchmark: GenImage Validation Split ($N = 200$: 100 Real ImageNet, 50 Fake Midjourney, 50 Fake ADM)
Final Audit Verdict: VERIFIED WITH LIMITATIONS
1. Executive Summary
An exhaustive, independent, read-only post-execution audit was conducted on Phase 7 of the project "Cross-Model Evaluation of Deepfake Image Detection Under Diverse Image Generation and Manipulation Conditions". Phase 7 evaluated whether three deepfake image detectors (ViT, UnivFD, ConvNeXt/DRCT) and three decision-fusion combinations generalize to external AI-generated images from generator families absent from their documented task-specific fine-tuning datasets.
Primary Audit Conclusions:
- Dataset Integrity & Zero Historical Leakage (VERIFIED):
- The frozen manifest
benchmark/external_generalization_manifest.csvcontains exactly 200 records: 100 authentic ImageNet natural photographs, 50 Midjourney synthetic images, and 50 ADM synthetic images. - All 200 image files exist on disk, decode successfully as 3-channel RGB, match their documented dimensions, and match their documented SHA-256 hashes with zero internal duplicates.
- Zero hash collisions were found against historical manifests (Phase 4 Kaggle benchmark and Phase 6 FFHQ/StyleGAN test manifest).
- The frozen manifest
- Authentic Controls Shared Structure (VERIFIED WITH LIMITATIONS):
- The generator-specific evaluations (Midjourney slice and ADM slice) each consist of 150 rows. Direct file-path and SHA-256 inspection confirms that the exact same 100 authentic ImageNet images are used as controls in both evaluations.
- The two evaluations are not 300 independent images. The entire external generalization benchmark comprises exactly 200 unique physical images.
- Prediction Integrity & Mathematical Metrics (VERIFIED):
- All three detector prediction CSVs (
phase7_vit_predictions.csv,phase7_univfd_predictions.csv,phase7_convnext_predictions.csv) contain exactly 200 rows, aligned 1-to-1 with manifest sample IDs. - Class mappings ($P(\text{Fake}) \ge 0.50 \implies \text{FAKE}$) and complementary probabilities ($P(\text{Real}) = 1.0 - P(\text{Fake})$) are 100% consistent across all detectors.
- Independent recalculation of all primary classification metrics (Accuracy, Balanced Accuracy, Precision, Recall, Specificity, F1-Score, and 95% Wilson confidence intervals) yielded an exact numerical match (difference = $0.0000$) against
benchmark/results/phase7_combined_metrics.jsonandbenchmark/results/phase7_model_comparison.csv.
- All three detector prediction CSVs (
- Resolution of Suspicious Majority Vote ROC-AUC (INCORRECT IMPLEMENTATION ATTRIBUTION):
- In
phase7_combined_metrics.jsonandPHASE7_EXTERNAL_GENERALIZATION_REPORT.md, Majority Voting and Mean Fake Probability both report an identical ROC-AUC of 0.7181. - Audit investigation traced this directly to
src/decision_fusion.py(lines 298β307), wherestrategy_majority_vote()assigns the arithmetic mean probability (mean_fake_p) as thefake_probabilityfield of the returnedFusionResult. Consequently,compute_binary_metrics()evaluated ROC-AUC on the exact same continuous mean probability array. - If Majority Voting is evaluated on its actual decision mechanics, its ROC-AUC is 0.6385 using discrete vote counts (0, 1, 2, 3 votes) or 0.5250 using hard binary predictions (matching its Balanced Accuracy).
- In
- Discrepancy in Written Report Agreement Analysis (REPORTING ERROR IN MARKDOWN):
- Section 5 of
docs/PHASE7_EXTERNAL_GENERALIZATION_REPORT.mderroneously claims that all three detectors agreed on 91 out of 200 samples (45.50%) and presents discrepant Cohen's Kappa values ($\kappa = 0.0988, 0.0520, 0.2849$). - The ground-truth execution JSON (
phase7_combined_metrics.json, lines 649β676) and independent recalculation confirm that all three detectors agree on 145 out of 200 samples (72.50%) (86 agreed REAL on authentic images + 59 agreed REAL on missed fake images; 0 agreed FAKE), with true $\kappa$ values of $0.0284$ (ViT vs UnivFD), $-0.0529$ (ViT vs ConvNeXt), and $0.0019$ (UnivFD vs ConvNeXt).
- Section 5 of
- Scientific Claim Nuance (LIMITATION):
- ViT accuracy drops from 98.00% on facial StyleGAN to 52.00% on ImageNet vs Midjourney/ADM. This drop cannot be attributed solely to generator architecture; it represents a compounded semantic domain shift (faces vs natural objects/scenes) and generative mechanism shift (GAN vs diffusion).
- At default $\tau = 0.50$, all detectors suffer from severe sensitivity deficits (fake recall $\le 18%$). While ConvNeXt exhibits strong ranking discrimination on Midjourney ($\text{AUC} = 0.8368$) and UnivFD on ADM ($\text{AUC} = 0.9056$), neither operates as an effective out-of-the-box detector without decision threshold recalibration.
2. Audit Scope and Methodology
The audit was executed entirely within c:\Users\user\Desktop\multi model connection using standard, non-destructive diagnostic tools.
Audit Principles:
- Zero Inference Execution: No detector models were loaded, initialized, or executed.
- Zero Data Modification: Original manifests, prediction CSVs, metric JSONs, and model weights were left 100% unaltered.
- Independent Verification Script: A standalone, read-only Python script (
scripts/audit_phase7_results.py) was developed to inspect file hashes, open image headers, recalculate statistical metrics from raw prediction CSVs, and cross-reference values against report documentation.
3. Evidence Inventory
| Category | File Path | Records / Size | Status | Nature of Artifact |
|---|---|---|---|---|
| Manifest | benchmark/external_generalization_manifest.csv |
200 rows (92.2 KB) | Available | Primary dataset specification artifact |
| Images | benchmark/external_generalization/images/real/ |
100 files (~10.8 MB) | Available | Extracted ImageNet-1k validation images |
| Images | benchmark/external_generalization/images/fake_midjourney/ |
50 files (~4.9 MB) | Available | Extracted GenImage Midjourney images |
| Images | benchmark/external_generalization/images/fake_adm/ |
50 files (~4.6 MB) | Available | Extracted GenImage ADM images |
| Predictions | benchmark/results/phase7_vit_predictions.csv |
200 rows (63.5 KB) | Available | Standalone inference outputs for ViT |
| Predictions | benchmark/results/phase7_univfd_predictions.csv |
200 rows (59.4 KB) | Available | Standalone inference outputs for UnivFD |
| Predictions | benchmark/results/phase7_convnext_predictions.csv |
200 rows (63.1 KB) | Available | Standalone inference outputs for ConvNeXt |
| Predictions | benchmark/results/phase7_combined_predictions.csv |
200 rows (63.1 KB) | Available | Multi-model merged predictions and fusion outputs |
| Metrics | benchmark/results/phase7_combined_metrics.json |
677 lines (14.7 KB) | Available | Execution pipeline statistical summary |
| Comparison | benchmark/results/phase7_model_comparison.csv |
7 data rows (1.1 KB) | Available | Consolidated performance summary table |
| Prep Script | scripts/prepare_phase7_dataset.py |
211 lines (8.6 KB) | Available | Ingestion and extraction script for GenImage |
| Eval Script | scripts/evaluate_phase7_pipeline.py |
624 lines (27.7 KB) | Available | Pipeline execution harness |
| Fusion Src | src/decision_fusion.py |
565 lines (23.1 KB) | Available | Decision fusion strategy implementation |
| Audit Script | scripts/audit_phase7_results.py |
535 lines (18.2 KB) | Available | Independent verification audit runner |
| Doc Report | docs/PHASE7_EXTERNAL_GENERALIZATION_REPORT.md |
167 lines (14.8 KB) | Available | Primary Phase 7 scientific research report |
| Doc Audit | docs/PHASE7_DATASET_AUDIT.md |
143 lines (11.3 KB) | Available | Stage 1 pre-execution planning audit |
| Progression | MODEL_STATUS_AND_PROJECT_PROGRESSION.md |
388 lines (41.6 KB) | Available | Master project progression tracker |
4. Dataset Integrity and Provenance
4.1 Sample Composition Breakdown
Manifest verification confirmed the exact distribution:
| Partition / Category | Manifest Target | Verified on Disk | Label Mapping | Generator Field | Status |
|---|---|---|---|---|---|
| Authentic Controls | 100 | 100 | REAL |
none |
VERIFIED |
| Midjourney Synthetic | 50 | 50 | FAKE |
midjourney |
VERIFIED |
| ADM Synthetic | 50 | 50 | FAKE |
adm |
VERIFIED |
| Total Benchmark | 200 | 200 | β | β | VERIFIED |
- Every sample has a strictly unique ID (
phase7_ext_real_001β100,phase7_ext_fake_midjourney_001β050,phase7_ext_fake_adm_001β050). - Every path resolves to a valid, uncorrupted JPEG file that opens in PIL and decodes to RGB mode.
- Recorded image dimensions match pixel headers exactly across all 200 files.
4.2 Provenance and Ingestion Lineage
- Dataset Repository: Hugging Face Hub
TheKernel01/Tiny-GenImage(NeurIPS 2023 GenImage benchmark). - Source Partition: Validation split shard 0 (
data/validation-00000-of-00004.parquet). - Authentic Provenance: ImageNet-1k validation set (
val/nature). - Synthetic Prompts: ImageNet synset text conditioning (
"a photo of {class_name}"). - License / Attribution: CC BY-NC-SA 4.0 / ImageNet non-commercial research use.
5. Unique Sample and Duplicate Analysis
5.1 Internal Duplicate Screening
SHA-256 hashes were calculated independently for all 200 files:
- Total Files Checked: 200
- Unique SHA-256 Digests: 200
- Hash Collisions: 0 (Zero internal duplicates).
5.2 Historical Manifest Overlap Screening
The 200 Phase 7 SHA-256 hashes were screened against all prior benchmark manifests:
- Phase 6 Test Manifest (
phase6_test_manifest.csv, 200 FFHQ/StyleGAN faces): 0 collisions. - Phase 4/5 Evaluation Manifest (
evaluation_manifest.csv, 250 Kaggle faces): 0 collisions. - Historical Leakage Verdict: ZERO DATASET LEAKAGE CONFIRMED.
5.3 Shared Authentic Controls Analysis
The generator-specific slices in the report evaluate:
- Midjourney Slice: 100 Real + 50 Midjourney Fake = 150 rows.
- ADM Slice: 100 Real + 50 ADM Fake = 150 rows.
Critical Finding: Direct SHA-256 digest comparison confirms that the 100 authentic images used in the Midjourney evaluation are 100% identical to the 100 authentic images used in the ADM evaluation.
- The two slices share a single control group.
- The 300 slice rows represent evaluations over 200 unique physical images.
- Consequently, one cannot sum or average the accuracies of the two 150-row slices to calculate aggregate accuracy. For example, averaging ConvNeXt's Midjourney accuracy (74.00%) and ADM accuracy (68.00%) yields 71.00%, whereas ConvNeXt's true unique-image aggregate accuracy is 56.50%.
6. Individual Model Prediction Verification
6.1 Prediction File Integrity
- All three individual prediction CSVs contain exactly 200 rows.
- Row order matches
benchmark/external_generalization_manifest.csv1-to-1 with identical sample IDs and ground truths. - Stored probabilities satisfy $0.0 \le P(\text{Fake}) \le 1.0$ and $P(\text{Fake}) + P(\text{Real}) = 1.0 \pm 10^{-5}$.
- Threshold mapping is consistent: $P(\text{Fake}) \ge 0.50 \implies \text{FAKE}$, $P(\text{Fake}) < 0.50 \implies \text{REAL}$.
6.2 Adapter Class Mapping Verification
Code inspection of the three detector adapters confirms correct binary class assignment:
- ViT Adapter (
src/vit_adapter.py): Reads Hugging Faceid2labelmapping ({0: 'REAL', 1: 'FAKE'}). Fake probability is assigned from index 1. - UnivFD Adapter (
src/univfd_adapter.py): Linear probe maps CLIP feature to logit. Sigmoid $\sigma(\text{logit})$ yields $P(\text{Fake})$. $P(\text{Real}) = 1.0 - \sigma(\text{logit})$. - ConvNeXt Adapter (
src/convnext_adapter.py): Linear head outputs 2 logits ([real_logit, fake_logit]). Softmax index 1 yields $P(\text{Fake})$.
7. Recalculated Model Metrics
Metrics were recalculated independently from raw prediction CSVs using standard definitions:
- $\text{Accuracy} = \frac{TP + TN}{N}$
- $\text{Precision} = \frac{TP}{TP + FP}$
- $\text{Recall (Fake)} = \frac{TP}{TP + FN}$
- $\text{Specificity} = \frac{TN}{TN + FP}$
- $\text{Balanced Accuracy} = \frac{\text{Recall} + \text{Specificity}}{2}$
- $\text{F1-Score} = \frac{2 \cdot \text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}}$
- $\text{ROC-AUC} = \text{sklearn.metrics.roc_auc_score}(y_true, p_fake)$
7.1 Aggregate Performance Recalculation Table ($N = 200$)
| Model / Strategy | Existing Acc | Recalc Acc | Existing BalAcc | Recalc BalAcc | Existing Prec | Recalc Prec | Existing Rec | Recalc Rec | Existing Spec | Recalc Spec | Existing F1 | Recalc F1 | Existing AUC | Recalc AUC | Diff | Status |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
ViT (dima806) |
0.5200 | 0.5200 | 0.5200 | 0.5200 | 0.5625 | 0.5625 | 0.1800 | 0.1800 | 0.8600 | 0.8600 | 0.2727 | 0.2727 | 0.4673 | 0.4673 | 0.0000 | VERIFIED |
| UnivFD (CLIP ViT-L/14) | 0.5750 | 0.5750 | 0.5750 | 0.5750 | 1.0000 | 1.0000 | 0.1500 | 0.1500 | 1.0000 | 1.0000 | 0.2609 | 0.2609 | 0.6662 | 0.6662 | 0.0000 | VERIFIED |
| ConvNeXt (DRCT / RAID) | 0.5650 | 0.5650 | 0.5650 | 0.5650 | 1.0000 | 1.0000 | 0.1300 | 0.1300 | 1.0000 | 1.0000 | 0.2301 | 0.2301 | 0.7926 | 0.7926 | 0.0000 | VERIFIED |
| Ensemble (Majority Vote) | 0.5250 | 0.5250 | 0.5250 | 0.5250 | 1.0000 | 1.0000 | 0.0500 | 0.0500 | 1.0000 | 1.0000 | 0.0952 | 0.0952 | 0.7181 | 0.7181* | 0.0000 | VERIFIED* |
| Ensemble (Mean Fake Prob) | 0.5250 | 0.5250 | 0.5250 | 0.5250 | 1.0000 | 1.0000 | 0.0500 | 0.0500 | 1.0000 | 1.0000 | 0.0952 | 0.0952 | 0.7181 | 0.7181 | 0.0000 | VERIFIED |
| Ensemble (ViT Priority) | 0.5200 | 0.5200 | 0.5200 | 0.5200 | 0.5625 | 0.5625 | 0.1800 | 0.1800 | 0.8600 | 0.8600 | 0.2727 | 0.2727 | 0.4673 | 0.4673 | 0.0000 | VERIFIED |
*Note: Verified against the stored probability vector, but see Section 10 for the critical audit finding on Majority Vote ROC-AUC.
7.2 Recalculated Confusion Matrices ($N = 200$)
| Model / Fusion Strategy | TP (Fake $\to$ Fake) | FP (Real $\to$ Fake) | TN (Real $\to$ Real) | FN (Fake $\to$ Real) | Total Valid | Error / Rejected |
|---|---|---|---|---|---|---|
ViT (dima806) |
18 | 14 | 86 | 82 | 200 | 0 |
| UnivFD (CLIP ViT-L/14) | 15 | 0 | 100 | 85 | 200 | 0 |
| ConvNeXt (DRCT / RAID) | 13 | 0 | 100 | 87 | 200 | 0 |
| Ensemble (Majority Vote) | 5 | 0 | 100 | 95 | 200 | 0 |
| Ensemble (Mean Fake Prob) | 5 | 0 | 100 | 95 | 200 | 0 |
| Ensemble (ViT Priority) | 18 | 14 | 86 | 82 | 200 | 0 |
8. Generator-Specific Performance Verification
8.1 Reconstructed Per-Generator Breakdown Table ($N = 150$ per slice)
| Model / Strategy | Generator | Accuracy | Fake Precision | Fake Recall | Specificity | F1-Score | ROC-AUC | TP | FP | TN | FN |
|---|---|---|---|---|---|---|---|---|---|---|---|
ViT (dima806) |
Midjourney | 64.00% | 0.4167 | 20.00% | 86.00% | 0.2703 | 0.5336 | 10 | 14 | 86 | 40 |
ViT (dima806) |
ADM | 62.67% | 0.3636 | 16.00% | 86.00% | 0.2222 | 0.4010 | 8 | 14 | 86 | 42 |
| UnivFD (CLIP ViT-L/14) | Midjourney | 66.67% | 0.0000 | 0.00% | 100.00% | 0.0000 | 0.4268 | 0 | 0 | 100 | 50 |
| UnivFD (CLIP ViT-L/14) | ADM | 76.67% | 1.0000 | 30.00% | 100.00% | 0.4615 | 0.9056 | 15 | 0 | 100 | 35 |
| ConvNeXt (DRCT / RAID) | Midjourney | 74.00% | 1.0000 | 22.00% | 100.00% | 0.3607 | 0.8368 | 11 | 0 | 100 | 39 |
| ConvNeXt (DRCT / RAID) | ADM | 68.00% | 1.0000 | 4.00% | 100.00% | 0.0769 | 0.7484 | 2 | 0 | 100 | 48 |
| Ensemble (Majority Vote) | Midjourney | 67.33% | 1.0000 | 2.00% | 100.00% | 0.0392 | 0.7048 | 1 | 0 | 100 | 49 |
| Ensemble (Majority Vote) | ADM | 69.33% | 1.0000 | 8.00% | 100.00% | 0.1481 | 0.7314 | 4 | 0 | 100 | 46 |
| Ensemble (Mean Fake Prob) | Midjourney | 67.33% | 1.0000 | 2.00% | 100.00% | 0.0392 | 0.7048 | 1 | 0 | 100 | 49 |
| Ensemble (Mean Fake Prob) | ADM | 69.33% | 1.0000 | 8.00% | 100.00% | 0.1481 | 0.7314 | 4 | 0 | 100 | 46 |
| Ensemble (ViT Priority) | Midjourney | 64.00% | 0.4167 | 20.00% | 86.00% | 0.2703 | 0.5336 | 10 | 14 | 86 | 40 |
| Ensemble (ViT Priority) | ADM | 62.67% | 0.3636 | 16.00% | 86.00% | 0.2222 | 0.4010 | 8 | 14 | 86 | 42 |
8.2 Generator-Level Research Sub-Questions
- Which detector ranks Midjourney images above authentic images most effectively?
ConvNeXt (DRCT / RAID) achieved $\text{ROC-AUC} = \mathbf{0.8368}$, strongly outranking ViT (0.5336) and UnivFD (0.4268). Its contrastive training on latent diffusion reconstruction artifacts transferred effectively to Midjourney's latent diffusion textures. - Which detector ranks ADM images above authentic images most effectively?
UnivFD (CLIP ViT-L/14) achieved $\text{ROC-AUC} = \mathbf{0.9056}$, outperforming ConvNeXt (0.7484) and ViT (0.4010). Because ADM operates in pixel space without latent decoding, its high-frequency residual distribution is well-captured by UnivFD's linear probe. - Which detector has the highest fake recall for each generator?
- For Midjourney: ConvNeXt achieved the highest recall at $\tau = 0.50$ (22.00%, 11/50), closely followed by ViT (20.00%, 10/50). UnivFD caught 0 samples (0.00%).
- For ADM: UnivFD achieved the highest recall at $\tau = 0.50$ (30.00%, 15/50), followed by ViT (16.00%, 8/50) and ConvNeXt (4.00%, 2/50).
- Which detector has the lowest false-positive rate?
UnivFD and ConvNeXt tie with a False Positive Rate of 0.00% (100.00% Specificity, 0/100 FP). ViT had an FPR of 14.00% (14/100 FP). - Does any detector consistently perform well across both generators?
No detector performs well at the default classification threshold ($\tau = 0.50$) across both generators, as all three exhibit recalls under 30%. However, in terms of latent discriminability (ROC-AUC), ConvNeXt is the most robust and consistent generalizer, maintaining an ROC-AUC of 0.8368 on Midjourney and 0.7484 on ADM. UnivFD is highly asymmetric (0.9056 on ADM vs. 0.4268 on Midjourney). ViT fails on both (0.5336 on Midjourney, 0.4010 on ADM).
9. Decision-Fusion Implementation Audit
9.1 Majority Voting Re-construction
- Hard voting rule: $\sum \mathbb{I}(\text{pred} == \text{FAKE}) \ge 2 \implies \text{FAKE}$.
- Distribution of votes across the 200 samples:
- 0 Fake votes (all 3 vote REAL): 145 samples
- 1 Fake vote (2 vote REAL, 1 votes FAKE): 50 samples
- 2 Fake votes (2 vote FAKE, 1 votes REAL): 5 samples
- 3 Fake votes (all 3 vote FAKE): 0 samples
- Tie-breaking: Zero ties occurred because all 3 models returned decisive binary predictions on all 200 samples ($3 \text{ votes} \implies \text{no 1.5-to-1.5 tie}$).
- Reconstructed predictions match
combined_preds['majority_vote_pred']in 200 out of 200 cases (100%).
9.2 Mean Fake Probability Re-construction
- Arithmetic mean: $P_{\text{mean}} = \frac{1}{3} (P_{\text{ViT}} + P_{\text{UnivFD}} + P_{\text{ConvNeXt}})$.
- Decision: $P_{\text{mean}} \ge 0.50 \implies \text{FAKE}$.
- Reconstructed mean probabilities and predictions match
phase7_combined_predictions.csvin 200 out of 200 cases (100%). - Calibration Limitation: Naive arithmetic averaging assumes comparable calibration across models. ViT outputs extreme uncalibrated softmax logits (often $0.0001$ or $0.9999$), whereas UnivFD and ConvNeXt produce more conservative sigmoid/softmax probabilities. An overconfident erroneous ViT can occasionally dominate the arithmetic mean.
9.3 ViT Priority Fallback
- Rule: If ViT produces a valid decisive prediction (
REALorFAKE), its verdict is forwarded directly. If ViT fails (ERRORorUNCERTAIN), secondary models are queried. - Empirical execution: ViT completed all 200 samples without errors or uncertainty.
- Fallback activations: 0 out of 200.
- Consequently, ViT Priority predictions and metrics are 100% mathematically identical to ViT alone.
10. ROC-AUC and Probability-Score Investigation
In the Phase 7 benchmark report and JSON metrics, Majority Voting and Mean Fake Probability both report an identical ROC-AUC of 0.7181.
10.1 Root-Cause Code Trace
The identical AUC values are explained by inspecting src/decision_fusion.py and scripts/evaluate_phase7_pipeline.py:
- In
src/decision_fusion.py, lines 298β307:# Compute mean probability across valid votes with probabilities valid_probs = [norm[k]["fake_probability"] for k in valid_keys if norm[k]["fake_probability"] is not None] mean_fake_p = round(sum(valid_probs) / len(valid_probs), 6) if valid_probs else None mean_real_p = round(1.0 - mean_fake_p, 6) if mean_fake_p is not None else None confidence = round(max(real_votes, fake_votes) / len(valid_keys), 4) return FusionResult( strategy_name=STRATEGY_MAJORITY_VOTE, prediction=prediction, fake_probability=mean_fake_p, # <--- ASSIGNS ARITHMETIC MEAN PROBABILITY ... ) - In
scripts/evaluate_phase7_pipeline.py, line 447 & 450:maj_probs.append(res_maj.fake_probability) mean_probs.append(res_mean.fake_probability) - In
scripts/evaluate_phase7_pipeline.py, line 534:m_agg = compute_binary_metrics(t["preds"], ground_truths, t["probs"])
Because res_maj.fake_probability was assigned mean_fake_p, maj_probs and mean_probs contain the exact same floating-point array across all 200 samples:
10.2 Independent Recalculation under Alternative Scoring Interpretations
ROC-AUC requires a score vector to rank samples across discrimination thresholds. Depending on how Majority Voting is formally interpreted as a ranking classifier:
| Score Representation | Input Vector | Independent ROC-AUC | Interpretation & Validity |
|---|---|---|---|
| Pipeline Default (Stored) | Arithmetic Mean Probability ($P_{\text{mean}}$) | 0.7181 | Represents Mean Fake Probability, not a pure vote ranking. |
| Discrete Vote Count / Fraction | Fake Vote Count / 3 ($[0.0, \frac{1}{3}, \frac{2}{3}, 1.0]$) | 0.6385 | True ranking ability of democratic voting consensus. Stepwise curve with 4 operating points. |
| Hard Binary Decision | Binary Output ($\mathbb{I}(\text{pred} == \text{FAKE}) \in {0, 1}$) | 0.5250 | Operating-point discrimination. Mathematically equals Balanced Accuracy: $\frac{0.05 + 1.00}{2} = 0.5250$. |
Audit Summary:
The reported ROC-AUC of 0.7181 is mathematically correct for the continuous mean probability vector stored in majority_vote_fake_prob. However, calling this the "ROC-AUC of Majority Voting" conflates voting decisions with probability averaging. The true ranking discriminability of majority voting alone is 0.6385.
11. Confidence-Interval Verification
The 95% confidence intervals in phase7_combined_metrics.json were verified against the Wilson score interval implementation (scripts/evaluate_phase7_pipeline.py, lines 70β82):
- Independent recalculations for all proportions ($k/n$) match the JSON bounds to 4 decimal places.
- Sample sizes used: $N = 200$ for aggregate accuracy, $N = 100$ for recall and specificity, $TP+FP$ for precision.
12. Model Provenance and Training-Data Limitations
| Model Identifier | Architecture | Documented Foundation Pretraining | Documented Task-Specific Fine-Tuning | Verified Training Data | Seen in Training? | Limitations |
|---|---|---|---|---|---|---|
vit (dima806) |
ViT-Base/16 (85.8M params) |
ImageNet-21k (14M images, Google 2020) | Kaggle 140k Real & Fake Faces | FFHQ (real faces) vs StyleGAN2 (fake faces) | StyleGAN: YES ProGAN: NO SD: NO Midjourney: NO ADM: NO |
Facial crops specialist. Completely out-of-distribution on non-facial objects and scenes. |
univfd (Probe) |
CLIP ViT-L/14 (frozen) + Linear Probe (768 \to 1) |
OpenAI WebImageText 400M (CLIP 2020) | ProGAN Artifact Detection (Ojha et al., CVPR 2023) | 20 LSUN scene categories (real LSUN vs ProGAN fakes) | ProGAN: YES StyleGAN: NO SD: NO Midjourney: NO ADM: NO |
Linear probe trained only on ProGAN. Unseen to all diffusion models. |
chen2024_convnext |
ConvNeXt-Base (88.6M params) + Linear Head |
ImageNet-22k (via timm) |
DRCT (Chen et al., ICML 2024 Spotlight) | MS-COCO train2017 vs SD v1.4, SD Inpainting, SD v2 reconstruction | SD (v1/v2): YES ProGAN: NO StyleGAN: NO Midjourney: NO ADM: NO |
Stable Diffusion is in-distribution. Midjourney and ADM are unseen generator families. |
Verification of Independence Claims:
- Midjourney (commercial latent diffusion): Confirmed strictly unseen in fine-tuning training sets for all three models.
- ADM (academic pixel diffusion): Confirmed strictly unseen in fine-tuning training sets for all three models.
- Distinction Requirement: While Midjourney and ADM are unseen generators, the underlying backbones of ViT and ConvNeXt were pretrained on ImageNet-21k / 22k. Thus, the 100% Specificity on ImageNet natural photographs is supported by the backbone's familiarity with ImageNet image distributions.
13. Scientific Interpretation and Claim Review
13.1 ViT Collapse (Phase 6 vs. Phase 7)
- Report Claim: ViT accuracy collapsed from 98.00% to 52.00% on external evaluation, "proving that ViT is a facial specialist whose decision boundaries do not transfer to general-scene diffusion images."
- Audit Assessment (VERIFIED WITH QUALIFICATION):
The numerical drop from 98.00% to 52.00% is verified. However, attributing the drop solely to generator identity is scientifically inaccurate. Phase 7 introduces a simultaneous semantic shift (faces $\to$ ImageNet objects) and generative shift (StyleGAN $\to$ Midjourney/ADM). The report should clearly state that domain shift and generator shift are confounded.
13.2 High Specificity vs. Low Sensitivity
- Report Claim: ConvNeXt and UnivFD achieved perfect 100% Specificity and strong latent discriminability ($\text{AUC} = 0.7926$ and $0.6662$).
- Audit Assessment (LIMITATION):
While mathematically true, this must be balanced by highlighting their severe sensitivity deficit: at $\tau = 0.50$, UnivFD missed 85% of fakes, and ConvNeXt missed 87% of fakes. Without threshold recalibration, neither model is viable as a practical standalone detector for these generators.
13.3 Fusion Dynamics
- Report Claim: Democratic consensus successfully filtered out all of ViT's 14 false alarms.
- Audit Assessment (VERIFIED WITH LIMITATION):
Majority voting achieved 100% Specificity ($FP = 0$). However, this conservative filtering also collapsed fake recall to 5.00% (missing 95 out of 100 fakes). The ensemble accuracy of 52.50% is lower than UnivFD alone (57.50%) and ConvNeXt alone (56.50%). Democratic fusion does not improve detection performance over the best individual model on this benchmark.
14. Reproducibility Assessment
- Manifest: Frozen CSV with SHA-256 hashes allows exact reconstruction of the evaluation set.
- Deterministic Pipeline: Subprocess isolation guarantees identical predictions regardless of hardware memory constraints.
- Zero Inconsistencies in Prediction Files: All 200 records across all 4 CSVs match ground truth and threshold logic.
- Reproducibility Rating: 100% REPRODUCIBLE from stored repository artifacts.
15. Discrepancies and Required Corrections
The audit identified two specific discrepancies that must be recorded:
Discrepancy 1: Agreement Analysis Figures in Markdown Report
- Location:
docs/PHASE7_EXTERNAL_GENERALIZATION_REPORT.md, Section 5 ("Cross-Model Agreement & Consensus Dynamics"). - Existing Text:
- All Three Detectors Agree: 91 out of 200 samples (45.50%).
- 86 agreed REAL, 5 agreed FAKE.
- ViT vs UnivFD $\kappa = 0.0988$ (Raw: 58.50%).
- ViT vs ConvNeXt $\kappa = 0.0520$ (Raw: 56.50%).
- UnivFD vs ConvNeXt $\kappa = 0.2849$ (Raw: 90.00%).
- Actual Computed Values (from
phase7_combined_metrics.jsonand verification audit):- All Three Detectors Agree: 145 out of 200 samples (72.50%).
- 86 agreed REAL on authentic images + 59 agreed REAL on fake images (false negatives) = 145 unanimous REAL agreements. Exactly 0 samples had all three detectors agree FAKE.
- ViT vs UnivFD: $\kappa = \mathbf{0.0284}$ (Raw agreement: 79.50%).
- ViT vs ConvNeXt: $\kappa = \mathbf{-0.0529}$ (Raw agreement: 78.50%).
- UnivFD vs ConvNeXt: $\kappa = \mathbf{0.0019}$ (Raw agreement: 87.00%).
- Explanation: The report author erroneously added ViT's 86 True Negatives to the ensemble's 5 True Positives ($86 + 5 = 91$) under the mistaken assumption that 3-way consensus occurred on those 5 samples (in fact, only 2 detectors voted fake on those 5 samples), while overlooking the 59 fake samples where all three detectors unanimously agreed REAL.
Discrepancy 2: Majority Vote ROC-AUC Characterization
- Location:
docs/PHASE7_EXTERNAL_GENERALIZATION_REPORT.md, Section 3;phase7_combined_metrics.json. - Issue: Majority Voting ROC-AUC is reported as 0.7181, which is identical to Mean Fake Probability because
strategy_majority_voteassigned the mean probability tofake_probability. - Correction: The report must document that 0.7181 reflects the mean probability score vector. The ROC-AUC of Majority Voting under discrete vote counting is 0.6385, and under hard binary decisions is 0.5250.
16. Final Verification Verdict
VERDICT: VERIFIED WITH LIMITATIONS
Justification:
- Calculations are Correct & Reproducible: The dataset manifest, 200 image files, prediction CSVs, confusion matrices, accuracy, precision, recall, specificity, F1-scores, and Wilson confidence intervals are 100% internally consistent and mathematically verified against independent recomputation.
- Experimental Limitations Restrict Broad Generalizations:
- Sample size is limited ($N = 200$: 100 Real, 50 Midjourney, 50 ADM).
- The 100 authentic ImageNet controls are reused between the Midjourney and ADM evaluations.
- At default $\tau = 0.50$, all models suffer from severe sensitivity failure (recall $\le 18%$).
- Majority voting ROC-AUC in the existing report reflects continuous probability averaging rather than voting rules.
- Section 5 of the Markdown report contains reporting discrepancies regarding 3-way agreement and Cohen's Kappa that must be corrected.
17. Recommendations for the Next Research Phase
Based on the empirical findings of this post-execution audit:
- Threshold Recalibration & ROC Operating Point Optimization (Phase 8 Candidate):
- Both ConvNeXt ($\text{AUC} = 0.7926$) and UnivFD ($\text{AUC} = 0.6662$) possess strong latent discriminability that is wasted at $\tau = 0.50$ (13% and 15% recall).
- Recalibrate decision thresholds using Youden's $J$ statistic or F1-maximization on a validation slice to assess whether practical detection performance can be unlocked without degrading specificity.
- Correct Agreement Analysis Reporting in Master Documentation:
- Update
docs/PHASE7_EXTERNAL_GENERALIZATION_REPORT.mdSection 5 to reflect the verified 3-way agreement count (145/200, 72.50%) and exact Cohen's Kappa figures. - Clarify the scoring vector used for Majority Voting ROC-AUC.
- Update
- Domain-Aware / Gated Adaptive Decision Fusion:
- Hierarchical ViT-Priority is optimal for facial media (Phase 6: 98%), while democratic / ConvNeXt-weighted fusion is superior for general scenes (Phase 7).
- Develop an upstream domain router (e.g., face detector gate) that routes face crops to ViT and general scene images to ConvNeXt / UnivFD.
- Expand External Generator Benchmark Diversity:
- Expand beyond Midjourney and ADM to include modern open-weights latent diffusion (Stable Diffusion XL, Flux.1) and commercial generative systems (DALL-E 3) with balanced authentic photo sources.