# Phase 7 — Comprehensive Post-Execution Verification and Research Audit
## Cross-Model Evaluation of Deepfake Image Detection Under Diverse Image Generation and Manipulation Conditions
**Document Path:** [`docs/PHASE7_POST_EXECUTION_AUDIT.md`](file:///c:/Users/user/Desktop/multi%20model%20connection/docs/PHASE7_POST_EXECUTION_AUDIT.md)
**Audit Date:** November 2026
**Auditor Mode:** Independent, Read-Only Scientific & Implementation Verification
**Primary Deliverable Target:** Phase 7 External AI-Generated Image Generalization Evaluation
**Evaluation Benchmark:** GenImage Validation Split ($N = 200$: 100 Real ImageNet, 50 Fake Midjourney, 50 Fake ADM)
**Final Audit Verdict:** **VERIFIED WITH LIMITATIONS**
---
## 1. Executive Summary
An exhaustive, independent, read-only post-execution audit was conducted on Phase 7 of the project *"Cross-Model Evaluation of Deepfake Image Detection Under Diverse Image Generation and Manipulation Conditions"*. Phase 7 evaluated whether three deepfake image detectors (`ViT`, `UnivFD`, `ConvNeXt/DRCT`) and three decision-fusion combinations generalize to external AI-generated images from generator families absent from their documented task-specific fine-tuning datasets.
### Primary Audit Conclusions:
1. **Dataset Integrity & Zero Historical Leakage (VERIFIED):**
- The frozen manifest [`benchmark/external_generalization_manifest.csv`](file:///c:/Users/user/Desktop/multi%20model%20connection/benchmark/external_generalization_manifest.csv) contains exactly 200 records: 100 authentic ImageNet natural photographs, 50 Midjourney synthetic images, and 50 ADM synthetic images.
- All 200 image files exist on disk, decode successfully as 3-channel RGB, match their documented dimensions, and match their documented SHA-256 hashes with **zero internal duplicates**.
- Zero hash collisions were found against historical manifests (Phase 4 Kaggle benchmark and Phase 6 FFHQ/StyleGAN test manifest).
2. **Authentic Controls Shared Structure (VERIFIED WITH LIMITATIONS):**
- The generator-specific evaluations (Midjourney slice and ADM slice) each consist of 150 rows. Direct file-path and SHA-256 inspection confirms that the **exact same 100 authentic ImageNet images** are used as controls in both evaluations.
- The two evaluations are not 300 independent images. The entire external generalization benchmark comprises exactly 200 unique physical images.
3. **Prediction Integrity & Mathematical Metrics (VERIFIED):**
- All three detector prediction CSVs (`phase7_vit_predictions.csv`, `phase7_univfd_predictions.csv`, `phase7_convnext_predictions.csv`) contain exactly 200 rows, aligned 1-to-1 with manifest sample IDs.
- Class mappings ($P(\text{Fake}) \ge 0.50 \implies \text{FAKE}$) and complementary probabilities ($P(\text{Real}) = 1.0 - P(\text{Fake})$) are 100% consistent across all detectors.
- Independent recalculation of all primary classification metrics (Accuracy, Balanced Accuracy, Precision, Recall, Specificity, F1-Score, and 95% Wilson confidence intervals) yielded an **exact numerical match** (difference = $0.0000$) against [`benchmark/results/phase7_combined_metrics.json`](file:///c:/Users/user/Desktop/multi%20model%20connection/benchmark/results/phase7_combined_metrics.json) and [`benchmark/results/phase7_model_comparison.csv`](file:///c:/Users/user/Desktop/multi%20model%20connection/benchmark/results/phase7_model_comparison.csv).
4. **Resolution of Suspicious Majority Vote ROC-AUC (INCORRECT IMPLEMENTATION ATTRIBUTION):**
- In [`phase7_combined_metrics.json`](file:///c:/Users/user/Desktop/multi%20model%20connection/benchmark/results/phase7_combined_metrics.json) and [`PHASE7_EXTERNAL_GENERALIZATION_REPORT.md`](file:///c:/Users/user/Desktop/multi%20model%20connection/docs/PHASE7_EXTERNAL_GENERALIZATION_REPORT.md), Majority Voting and Mean Fake Probability both report an identical ROC-AUC of **0.7181**.
- Audit investigation traced this directly to [`src/decision_fusion.py`](file:///c:/Users/user/Desktop/multi%20model%20connection/src/decision_fusion.py) (lines 298–307), where `strategy_majority_vote()` assigns the arithmetic mean probability (`mean_fake_p`) as the `fake_probability` field of the returned `FusionResult`. Consequently, `compute_binary_metrics()` evaluated ROC-AUC on the exact same continuous mean probability array.
- If Majority Voting is evaluated on its actual decision mechanics, its ROC-AUC is **0.6385** using discrete vote counts (0, 1, 2, 3 votes) or **0.5250** using hard binary predictions (matching its Balanced Accuracy).
5. **Discrepancy in Written Report Agreement Analysis (REPORTING ERROR IN MARKDOWN):**
- Section 5 of [`docs/PHASE7_EXTERNAL_GENERALIZATION_REPORT.md`](file:///c:/Users/user/Desktop/multi%20model%20connection/docs/PHASE7_EXTERNAL_GENERALIZATION_REPORT.md) erroneously claims that all three detectors agreed on 91 out of 200 samples (45.50%) and presents discrepant Cohen's Kappa values ($\kappa = 0.0988, 0.0520, 0.2849$).
- The ground-truth execution JSON (`phase7_combined_metrics.json`, lines 649–676) and independent recalculation confirm that all three detectors agree on **145 out of 200 samples (72.50%)** (86 agreed REAL on authentic images + 59 agreed REAL on missed fake images; 0 agreed FAKE), with true $\kappa$ values of $0.0284$ (ViT vs UnivFD), $-0.0529$ (ViT vs ConvNeXt), and $0.0019$ (UnivFD vs ConvNeXt).
6. **Scientific Claim Nuance (LIMITATION):**
- ViT accuracy drops from 98.00% on facial StyleGAN to 52.00% on ImageNet vs Midjourney/ADM. This drop cannot be attributed solely to generator architecture; it represents a compounded **semantic domain shift** (faces vs natural objects/scenes) and **generative mechanism shift** (GAN vs diffusion).
- At default $\tau = 0.50$, all detectors suffer from severe sensitivity deficits (fake recall $\le 18\%$). While ConvNeXt exhibits strong ranking discrimination on Midjourney ($\text{AUC} = 0.8368$) and UnivFD on ADM ($\text{AUC} = 0.9056$), neither operates as an effective out-of-the-box detector without decision threshold recalibration.
---
## 2. Audit Scope and Methodology
The audit was executed entirely within `c:\Users\user\Desktop\multi model connection` using standard, non-destructive diagnostic tools.
### Audit Principles:
- **Zero Inference Execution:** No detector models were loaded, initialized, or executed.
- **Zero Data Modification:** Original manifests, prediction CSVs, metric JSONs, and model weights were left 100% unaltered.
- **Independent Verification Script:** A standalone, read-only Python script ([`scripts/audit_phase7_results.py`](file:///c:/Users/user/Desktop/multi%20model%20connection/scripts/audit_phase7_results.py)) was developed to inspect file hashes, open image headers, recalculate statistical metrics from raw prediction CSVs, and cross-reference values against report documentation.
---
## 3. Evidence Inventory
| Category | File Path | Records / Size | Status | Nature of Artifact |
| :--- | :--- | :---: | :---: | :--- |
| **Manifest** | [`benchmark/external_generalization_manifest.csv`](file:///c:/Users/user/Desktop/multi%20model%20connection/benchmark/external_generalization_manifest.csv) | 200 rows (92.2 KB) | Available | Primary dataset specification artifact |
| **Images** | `benchmark/external_generalization/images/real/` | 100 files (~10.8 MB) | Available | Extracted ImageNet-1k validation images |
| **Images** | `benchmark/external_generalization/images/fake_midjourney/` | 50 files (~4.9 MB) | Available | Extracted GenImage Midjourney images |
| **Images** | `benchmark/external_generalization/images/fake_adm/` | 50 files (~4.6 MB) | Available | Extracted GenImage ADM images |
| **Predictions** | [`benchmark/results/phase7_vit_predictions.csv`](file:///c:/Users/user/Desktop/multi%20model%20connection/benchmark/results/phase7_vit_predictions.csv) | 200 rows (63.5 KB) | Available | Standalone inference outputs for ViT |
| **Predictions** | [`benchmark/results/phase7_univfd_predictions.csv`](file:///c:/Users/user/Desktop/multi%20model%20connection/benchmark/results/phase7_univfd_predictions.csv) | 200 rows (59.4 KB) | Available | Standalone inference outputs for UnivFD |
| **Predictions** | [`benchmark/results/phase7_convnext_predictions.csv`](file:///c:/Users/user/Desktop/multi%20model%20connection/benchmark/results/phase7_convnext_predictions.csv) | 200 rows (63.1 KB) | Available | Standalone inference outputs for ConvNeXt |
| **Predictions** | [`benchmark/results/phase7_combined_predictions.csv`](file:///c:/Users/user/Desktop/multi%20model%20connection/benchmark/results/phase7_combined_predictions.csv) | 200 rows (63.1 KB) | Available | Multi-model merged predictions and fusion outputs |
| **Metrics** | [`benchmark/results/phase7_combined_metrics.json`](file:///c:/Users/user/Desktop/multi%20model%20connection/benchmark/results/phase7_combined_metrics.json) | 677 lines (14.7 KB) | Available | Execution pipeline statistical summary |
| **Comparison** | [`benchmark/results/phase7_model_comparison.csv`](file:///c:/Users/user/Desktop/multi%20model%20connection/benchmark/results/phase7_model_comparison.csv) | 7 data rows (1.1 KB) | Available | Consolidated performance summary table |
| **Prep Script** | [`scripts/prepare_phase7_dataset.py`](file:///c:/Users/user/Desktop/multi%20model%20connection/scripts/prepare_phase7_dataset.py) | 211 lines (8.6 KB) | Available | Ingestion and extraction script for GenImage |
| **Eval Script** | [`scripts/evaluate_phase7_pipeline.py`](file:///c:/Users/user/Desktop/multi%20model%20connection/scripts/evaluate_phase7_pipeline.py) | 624 lines (27.7 KB) | Available | Pipeline execution harness |
| **Fusion Src** | [`src/decision_fusion.py`](file:///c:/Users/user/Desktop/multi%20model%20connection/src/decision_fusion.py) | 565 lines (23.1 KB) | Available | Decision fusion strategy implementation |
| **Audit Script**| [`scripts/audit_phase7_results.py`](file:///c:/Users/user/Desktop/multi%20model%20connection/scripts/audit_phase7_results.py) | 535 lines (18.2 KB) | Available | Independent verification audit runner |
| **Doc Report** | [`docs/PHASE7_EXTERNAL_GENERALIZATION_REPORT.md`](file:///c:/Users/user/Desktop/multi%20model%20connection/docs/PHASE7_EXTERNAL_GENERALIZATION_REPORT.md) | 167 lines (14.8 KB) | Available | Primary Phase 7 scientific research report |
| **Doc Audit** | [`docs/PHASE7_DATASET_AUDIT.md`](file:///c:/Users/user/Desktop/multi%20model%20connection/docs/PHASE7_DATASET_AUDIT.md) | 143 lines (11.3 KB) | Available | Stage 1 pre-execution planning audit |
| **Progression**| [`MODEL_STATUS_AND_PROJECT_PROGRESSION.md`](file:///c:/Users/user/Desktop/multi%20model%20connection/MODEL_STATUS_AND_PROJECT_PROGRESSION.md) | 388 lines (41.6 KB) | Available | Master project progression tracker |
---
## 4. Dataset Integrity and Provenance
### 4.1 Sample Composition Breakdown
Manifest verification confirmed the exact distribution:
| Partition / Category | Manifest Target | Verified on Disk | Label Mapping | Generator Field | Status |
| :--- | :---: | :---: | :---: | :---: | :---: |
| **Authentic Controls** | 100 | 100 | `REAL` | `none` | **VERIFIED** |
| **Midjourney Synthetic** | 50 | 50 | `FAKE` | `midjourney` | **VERIFIED** |
| **ADM Synthetic** | 50 | 50 | `FAKE` | `adm` | **VERIFIED** |
| **Total Benchmark** | **200** | **200** | — | — | **VERIFIED** |
- Every sample has a strictly unique ID (`phase7_ext_real_001`–`100`, `phase7_ext_fake_midjourney_001`–`050`, `phase7_ext_fake_adm_001`–`050`).
- Every path resolves to a valid, uncorrupted JPEG file that opens in PIL and decodes to RGB mode.
- Recorded image dimensions match pixel headers exactly across all 200 files.
### 4.2 Provenance and Ingestion Lineage
- **Dataset Repository:** Hugging Face Hub `TheKernel01/Tiny-GenImage` (NeurIPS 2023 GenImage benchmark).
- **Source Partition:** Validation split shard 0 (`data/validation-00000-of-00004.parquet`).
- **Authentic Provenance:** ImageNet-1k validation set (`val/nature`).
- **Synthetic Prompts:** ImageNet synset text conditioning (`"a photo of {class_name}"`).
- **License / Attribution:** CC BY-NC-SA 4.0 / ImageNet non-commercial research use.
---
## 5. Unique Sample and Duplicate Analysis
### 5.1 Internal Duplicate Screening
SHA-256 hashes were calculated independently for all 200 files:
- **Total Files Checked:** 200
- **Unique SHA-256 Digests:** 200
- **Hash Collisions:** 0 (Zero internal duplicates).
### 5.2 Historical Manifest Overlap Screening
The 200 Phase 7 SHA-256 hashes were screened against all prior benchmark manifests:
- **Phase 6 Test Manifest (`phase6_test_manifest.csv`, 200 FFHQ/StyleGAN faces):** 0 collisions.
- **Phase 4/5 Evaluation Manifest (`evaluation_manifest.csv`, 250 Kaggle faces):** 0 collisions.
- **Historical Leakage Verdict:** **ZERO DATASET LEAKAGE CONFIRMED**.
### 5.3 Shared Authentic Controls Analysis
The generator-specific slices in the report evaluate:
- **Midjourney Slice:** 100 Real + 50 Midjourney Fake = 150 rows.
- **ADM Slice:** 100 Real + 50 ADM Fake = 150 rows.
**Critical Finding:**
Direct SHA-256 digest comparison confirms that the **100 authentic images used in the Midjourney evaluation are 100% identical to the 100 authentic images used in the ADM evaluation**.
- The two slices share a single control group.
- The 300 slice rows represent evaluations over **200 unique physical images**.
- Consequently, one cannot sum or average the accuracies of the two 150-row slices to calculate aggregate accuracy. For example, averaging ConvNeXt's Midjourney accuracy (74.00%) and ADM accuracy (68.00%) yields 71.00%, whereas ConvNeXt's true unique-image aggregate accuracy is **56.50%**.
---
## 6. Individual Model Prediction Verification
### 6.1 Prediction File Integrity
- All three individual prediction CSVs contain exactly 200 rows.
- Row order matches `benchmark/external_generalization_manifest.csv` 1-to-1 with identical sample IDs and ground truths.
- Stored probabilities satisfy $0.0 \le P(\text{Fake}) \le 1.0$ and $P(\text{Fake}) + P(\text{Real}) = 1.0 \pm 10^{-5}$.
- Threshold mapping is consistent: $P(\text{Fake}) \ge 0.50 \implies \text{FAKE}$, $P(\text{Fake}) < 0.50 \implies \text{REAL}$.
### 6.2 Adapter Class Mapping Verification
Code inspection of the three detector adapters confirms correct binary class assignment:
- **ViT Adapter ([`src/vit_adapter.py`](file:///c:/Users/user/Desktop/multi%20model%20connection/src/vit_adapter.py)):** Reads Hugging Face `id2label` mapping (`{0: 'REAL', 1: 'FAKE'}`). Fake probability is assigned from index 1.
- **UnivFD Adapter ([`src/univfd_adapter.py`](file:///c:/Users/user/Desktop/multi%20model%20connection/src/univfd_adapter.py)):** Linear probe maps CLIP feature to logit. Sigmoid $\sigma(\text{logit})$ yields $P(\text{Fake})$. $P(\text{Real}) = 1.0 - \sigma(\text{logit})$.
- **ConvNeXt Adapter ([`src/convnext_adapter.py`](file:///c:/Users/user/Desktop/multi%20model%20connection/src/convnext_adapter.py)):** Linear head outputs 2 logits (`[real_logit, fake_logit]`). Softmax index 1 yields $P(\text{Fake})$.
---
## 7. Recalculated Model Metrics
Metrics were recalculated independently from raw prediction CSVs using standard definitions:
- $\text{Accuracy} = \frac{TP + TN}{N}$
- $\text{Precision} = \frac{TP}{TP + FP}$
- $\text{Recall (Fake)} = \frac{TP}{TP + FN}$
- $\text{Specificity} = \frac{TN}{TN + FP}$
- $\text{Balanced Accuracy} = \frac{\text{Recall} + \text{Specificity}}{2}$
- $\text{F1-Score} = \frac{2 \cdot \text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}}$
- $\text{ROC-AUC} = \text{sklearn.metrics.roc\_auc\_score}(y\_true, p\_fake)$
### 7.1 Aggregate Performance Recalculation Table ($N = 200$)
| Model / Strategy | Existing Acc | Recalc Acc | Existing BalAcc | Recalc BalAcc | Existing Prec | Recalc Prec | Existing Rec | Recalc Rec | Existing Spec | Recalc Spec | Existing F1 | Recalc F1 | Existing AUC | Recalc AUC | Diff | Status |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| **ViT (`dima806`)** | 0.5200 | 0.5200 | 0.5200 | 0.5200 | 0.5625 | 0.5625 | 0.1800 | 0.1800 | 0.8600 | 0.8600 | 0.2727 | 0.2727 | 0.4673 | 0.4673 | 0.0000 | **VERIFIED** |
| **UnivFD (CLIP ViT-L/14)** | 0.5750 | 0.5750 | 0.5750 | 0.5750 | 1.0000 | 1.0000 | 0.1500 | 0.1500 | 1.0000 | 1.0000 | 0.2609 | 0.2609 | 0.6662 | 0.6662 | 0.0000 | **VERIFIED** |
| **ConvNeXt (DRCT / RAID)** | 0.5650 | 0.5650 | 0.5650 | 0.5650 | 1.0000 | 1.0000 | 0.1300 | 0.1300 | 1.0000 | 1.0000 | 0.2301 | 0.2301 | 0.7926 | 0.7926 | 0.0000 | **VERIFIED** |
| **Ensemble (Majority Vote)** | 0.5250 | 0.5250 | 0.5250 | 0.5250 | 1.0000 | 1.0000 | 0.0500 | 0.0500 | 1.0000 | 1.0000 | 0.0952 | 0.0952 | 0.7181 | 0.7181* | 0.0000 | **VERIFIED\*** |
| **Ensemble (Mean Fake Prob)**| 0.5250 | 0.5250 | 0.5250 | 0.5250 | 1.0000 | 1.0000 | 0.0500 | 0.0500 | 1.0000 | 1.0000 | 0.0952 | 0.0952 | 0.7181 | 0.7181 | 0.0000 | **VERIFIED** |
| **Ensemble (ViT Priority)** | 0.5200 | 0.5200 | 0.5200 | 0.5200 | 0.5625 | 0.5625 | 0.1800 | 0.1800 | 0.8600 | 0.8600 | 0.2727 | 0.2727 | 0.4673 | 0.4673 | 0.0000 | **VERIFIED** |
*\*Note: Verified against the stored probability vector, but see Section 10 for the critical audit finding on Majority Vote ROC-AUC.*
### 7.2 Recalculated Confusion Matrices ($N = 200$)
| Model / Fusion Strategy | TP (Fake $\to$ Fake) | FP (Real $\to$ Fake) | TN (Real $\to$ Real) | FN (Fake $\to$ Real) | Total Valid | Error / Rejected |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: |
| **ViT (`dima806`)** | 18 | 14 | 86 | 82 | 200 | 0 |
| **UnivFD (CLIP ViT-L/14)** | 15 | 0 | 100 | 85 | 200 | 0 |
| **ConvNeXt (DRCT / RAID)** | 13 | 0 | 100 | 87 | 200 | 0 |
| **Ensemble (Majority Vote)** | 5 | 0 | 100 | 95 | 200 | 0 |
| **Ensemble (Mean Fake Prob)**| 5 | 0 | 100 | 95 | 200 | 0 |
| **Ensemble (ViT Priority)** | 18 | 14 | 86 | 82 | 200 | 0 |
---
## 8. Generator-Specific Performance Verification
### 8.1 Reconstructed Per-Generator Breakdown Table ($N = 150$ per slice)
| Model / Strategy | Generator | Accuracy | Fake Precision | Fake Recall | Specificity | F1-Score | ROC-AUC | TP | FP | TN | FN |
| :--- | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| **ViT (`dima806`)** | Midjourney | 64.00% | 0.4167 | 20.00% | 86.00% | 0.2703 | 0.5336 | 10 | 14 | 86 | 40 |
| **ViT (`dima806`)** | ADM | 62.67% | 0.3636 | 16.00% | 86.00% | 0.2222 | 0.4010 | 8 | 14 | 86 | 42 |
| **UnivFD (CLIP ViT-L/14)** | Midjourney | 66.67% | 0.0000 | 0.00% | 100.00% | 0.0000 | 0.4268 | 0 | 0 | 100 | 50 |
| **UnivFD (CLIP ViT-L/14)** | ADM | 76.67% | 1.0000 | 30.00% | 100.00% | 0.4615 | 0.9056 | 15 | 0 | 100 | 35 |
| **ConvNeXt (DRCT / RAID)** | Midjourney | 74.00% | 1.0000 | 22.00% | 100.00% | 0.3607 | 0.8368 | 11 | 0 | 100 | 39 |
| **ConvNeXt (DRCT / RAID)** | ADM | 68.00% | 1.0000 | 4.00% | 100.00% | 0.0769 | 0.7484 | 2 | 0 | 100 | 48 |
| **Ensemble (Majority Vote)**| Midjourney | 67.33% | 1.0000 | 2.00% | 100.00% | 0.0392 | 0.7048 | 1 | 0 | 100 | 49 |
| **Ensemble (Majority Vote)**| ADM | 69.33% | 1.0000 | 8.00% | 100.00% | 0.1481 | 0.7314 | 4 | 0 | 100 | 46 |
| **Ensemble (Mean Fake Prob)**| Midjourney | 67.33% | 1.0000 | 2.00% | 100.00% | 0.0392 | 0.7048 | 1 | 0 | 100 | 49 |
| **Ensemble (Mean Fake Prob)**| ADM | 69.33% | 1.0000 | 8.00% | 100.00% | 0.1481 | 0.7314 | 4 | 0 | 100 | 46 |
| **Ensemble (ViT Priority)** | Midjourney | 64.00% | 0.4167 | 20.00% | 86.00% | 0.2703 | 0.5336 | 10 | 14 | 86 | 40 |
| **Ensemble (ViT Priority)** | ADM | 62.67% | 0.3636 | 16.00% | 86.00% | 0.2222 | 0.4010 | 8 | 14 | 86 | 42 |
### 8.2 Generator-Level Research Sub-Questions
1. **Which detector ranks Midjourney images above authentic images most effectively?**
**ConvNeXt (DRCT / RAID)** achieved $\text{ROC-AUC} = \mathbf{0.8368}$, strongly outranking ViT (0.5336) and UnivFD (0.4268). Its contrastive training on latent diffusion reconstruction artifacts transferred effectively to Midjourney's latent diffusion textures.
2. **Which detector ranks ADM images above authentic images most effectively?**
**UnivFD (CLIP ViT-L/14)** achieved $\text{ROC-AUC} = \mathbf{0.9056}$, outperforming ConvNeXt (0.7484) and ViT (0.4010). Because ADM operates in pixel space without latent decoding, its high-frequency residual distribution is well-captured by UnivFD's linear probe.
3. **Which detector has the highest fake recall for each generator?**
- For Midjourney: **ConvNeXt** achieved the highest recall at $\tau = 0.50$ (**22.00%**, 11/50), closely followed by ViT (**20.00%**, 10/50). UnivFD caught 0 samples (**0.00%**).
- For ADM: **UnivFD** achieved the highest recall at $\tau = 0.50$ (**30.00%**, 15/50), followed by ViT (**16.00%**, 8/50) and ConvNeXt (**4.00%**, 2/50).
4. **Which detector has the lowest false-positive rate?**
**UnivFD and ConvNeXt tie with a False Positive Rate of 0.00%** (100.00% Specificity, 0/100 FP). ViT had an FPR of **14.00%** (14/100 FP).
5. **Does any detector consistently perform well across both generators?**
**No detector performs well at the default classification threshold ($\tau = 0.50$) across both generators**, as all three exhibit recalls under 30%. However, in terms of latent discriminability (ROC-AUC), **ConvNeXt is the most robust and consistent generalizer**, maintaining an ROC-AUC of 0.8368 on Midjourney and 0.7484 on ADM. UnivFD is highly asymmetric (0.9056 on ADM vs. 0.4268 on Midjourney). ViT fails on both (0.5336 on Midjourney, 0.4010 on ADM).
---
## 9. Decision-Fusion Implementation Audit
### 9.1 Majority Voting Re-construction
- Hard voting rule: $\sum \mathbb{I}(\text{pred} == \text{FAKE}) \ge 2 \implies \text{FAKE}$.
- Distribution of votes across the 200 samples:
* 0 Fake votes (all 3 vote REAL): **145 samples**
* 1 Fake vote (2 vote REAL, 1 votes FAKE): **50 samples**
* 2 Fake votes (2 vote FAKE, 1 votes REAL): **5 samples**
* 3 Fake votes (all 3 vote FAKE): **0 samples**
- Tie-breaking: Zero ties occurred because all 3 models returned decisive binary predictions on all 200 samples ($3 \text{ votes} \implies \text{no 1.5-to-1.5 tie}$).
- Reconstructed predictions match `combined_preds['majority_vote_pred']` in **200 out of 200 cases (100%)**.
### 9.2 Mean Fake Probability Re-construction
- Arithmetic mean: $P_{\text{mean}} = \frac{1}{3} (P_{\text{ViT}} + P_{\text{UnivFD}} + P_{\text{ConvNeXt}})$.
- Decision: $P_{\text{mean}} \ge 0.50 \implies \text{FAKE}$.
- Reconstructed mean probabilities and predictions match `phase7_combined_predictions.csv` in **200 out of 200 cases (100%)**.
- **Calibration Limitation:** Naive arithmetic averaging assumes comparable calibration across models. ViT outputs extreme uncalibrated softmax logits (often $0.0001$ or $0.9999$), whereas UnivFD and ConvNeXt produce more conservative sigmoid/softmax probabilities. An overconfident erroneous ViT can occasionally dominate the arithmetic mean.
### 9.3 ViT Priority Fallback
- Rule: If ViT produces a valid decisive prediction (`REAL` or `FAKE`), its verdict is forwarded directly. If ViT fails (`ERROR` or `UNCERTAIN`), secondary models are queried.
- Empirical execution: ViT completed all 200 samples without errors or uncertainty.
- Fallback activations: **0 out of 200**.
- Consequently, ViT Priority predictions and metrics are **100% mathematically identical to ViT alone**.
---
## 10. ROC-AUC and Probability-Score Investigation
In the Phase 7 benchmark report and JSON metrics, **Majority Voting** and **Mean Fake Probability** both report an identical ROC-AUC of **0.7181**.
### 10.1 Root-Cause Code Trace
The identical AUC values are explained by inspecting [`src/decision_fusion.py`](file:///c:/Users/user/Desktop/multi%20model%20connection/src/decision_fusion.py) and [`scripts/evaluate_phase7_pipeline.py`](file:///c:/Users/user/Desktop/multi%20model%20connection/scripts/evaluate_phase7_pipeline.py):
1. In [`src/decision_fusion.py`](file:///c:/Users/user/Desktop/multi%20model%20connection/src/decision_fusion.py), lines 298–307:
```python
# Compute mean probability across valid votes with probabilities
valid_probs = [norm[k]["fake_probability"] for k in valid_keys if norm[k]["fake_probability"] is not None]
mean_fake_p = round(sum(valid_probs) / len(valid_probs), 6) if valid_probs else None
mean_real_p = round(1.0 - mean_fake_p, 6) if mean_fake_p is not None else None
confidence = round(max(real_votes, fake_votes) / len(valid_keys), 4)
return FusionResult(
strategy_name=STRATEGY_MAJORITY_VOTE,
prediction=prediction,
fake_probability=mean_fake_p, # <--- ASSIGNS ARITHMETIC MEAN PROBABILITY
...
)
```
2. In [`scripts/evaluate_phase7_pipeline.py`](file:///c:/Users/user/Desktop/multi%20model%20connection/scripts/evaluate_phase7_pipeline.py), line 447 & 450:
```python
maj_probs.append(res_maj.fake_probability)
mean_probs.append(res_mean.fake_probability)
```
3. In [`scripts/evaluate_phase7_pipeline.py`](file:///c:/Users/user/Desktop/multi%20model%20connection/scripts/evaluate_phase7_pipeline.py), line 534:
```python
m_agg = compute_binary_metrics(t["preds"], ground_truths, t["probs"])
```
Because `res_maj.fake_probability` was assigned `mean_fake_p`, `maj_probs` and `mean_probs` contain the **exact same floating-point array** across all 200 samples:
$$\max | \text{maj\_probs}_i - \text{mean\_probs}_i | = 0.00000000$$
### 10.2 Independent Recalculation under Alternative Scoring Interpretations
ROC-AUC requires a score vector to rank samples across discrimination thresholds. Depending on how Majority Voting is formally interpreted as a ranking classifier:
| Score Representation | Input Vector | Independent ROC-AUC | Interpretation & Validity |
| :--- | :--- | :---: | :--- |
| **Pipeline Default (Stored)** | Arithmetic Mean Probability ($P_{\text{mean}}$) | **0.7181** | **Represents Mean Fake Probability**, not a pure vote ranking. |
| **Discrete Vote Count / Fraction** | Fake Vote Count / 3 ($[0.0, \frac{1}{3}, \frac{2}{3}, 1.0]$) | **0.6385** | **True ranking ability of democratic voting consensus**. Stepwise curve with 4 operating points. |
| **Hard Binary Decision** | Binary Output ($\mathbb{I}(\text{pred} == \text{FAKE}) \in \{0, 1\}$) | **0.5250** | **Operating-point discrimination**. Mathematically equals Balanced Accuracy: $\frac{0.05 + 1.00}{2} = 0.5250$. |
### Audit Summary:
The reported ROC-AUC of 0.7181 is mathematically correct for the continuous mean probability vector stored in `majority_vote_fake_prob`. However, calling this the "ROC-AUC of Majority Voting" conflates voting decisions with probability averaging. The true ranking discriminability of majority voting alone is **0.6385**.
---
## 11. Confidence-Interval Verification
The 95% confidence intervals in [`phase7_combined_metrics.json`](file:///c:/Users/user/Desktop/multi%20model%20connection/benchmark/results/phase7_combined_metrics.json) were verified against the Wilson score interval implementation ([`scripts/evaluate_phase7_pipeline.py`](file:///c:/Users/user/Desktop/multi%20model%20connection/scripts/evaluate_phase7_pipeline.py), lines 70–82):
$$p_{\text{center}} = \frac{\hat{p} + \frac{z^2}{2n}}{1 + \frac{z^2}{n}}, \quad \text{margin} = \frac{z \sqrt{\frac{\hat{p}(1-\hat{p})}{n} + \frac{z^2}{4n^2}}}{1 + \frac{z^2}{n}}, \quad z = 1.959964$$
- Independent recalculations for all proportions ($k/n$) match the JSON bounds to 4 decimal places.
- Sample sizes used: $N = 200$ for aggregate accuracy, $N = 100$ for recall and specificity, $TP+FP$ for precision.
---
## 12. Model Provenance and Training-Data Limitations
| Model Identifier | Architecture | Documented Foundation Pretraining | Documented Task-Specific Fine-Tuning | Verified Training Data | Seen in Training? | Limitations |
| :--- | :--- | :--- | :--- | :--- | :---: | :--- |
| **`vit` (dima806)** | `ViT-Base/16` (85.8M params) | ImageNet-21k (14M images, Google 2020) | Kaggle 140k Real & Fake Faces | FFHQ (real faces) vs StyleGAN2 (fake faces) | StyleGAN: **YES**
ProGAN: **NO**
SD: **NO**
Midjourney: **NO**
ADM: **NO** | Facial crops specialist. Completely out-of-distribution on non-facial objects and scenes. |
| **`univfd` (Probe)** | CLIP `ViT-L/14` (frozen) + Linear Probe (`768 \to 1`) | OpenAI WebImageText 400M (CLIP 2020) | ProGAN Artifact Detection (Ojha et al., CVPR 2023) | 20 LSUN scene categories (real LSUN vs ProGAN fakes) | ProGAN: **YES**
StyleGAN: **NO**
SD: **NO**
Midjourney: **NO**
ADM: **NO** | Linear probe trained only on ProGAN. Unseen to all diffusion models. |
| **`chen2024_convnext`** | `ConvNeXt-Base` (88.6M params) + Linear Head | ImageNet-22k (via `timm`) | DRCT (Chen et al., ICML 2024 Spotlight) | MS-COCO train2017 vs SD v1.4, SD Inpainting, SD v2 reconstruction | SD (v1/v2): **YES**
ProGAN: **NO**
StyleGAN: **NO**
Midjourney: **NO**
ADM: **NO** | Stable Diffusion is in-distribution. Midjourney and ADM are unseen generator families. |
### Verification of Independence Claims:
- **Midjourney (commercial latent diffusion):** Confirmed strictly unseen in fine-tuning training sets for all three models.
- **ADM (academic pixel diffusion):** Confirmed strictly unseen in fine-tuning training sets for all three models.
- **Distinction Requirement:** While Midjourney and ADM are unseen generators, the underlying backbones of ViT and ConvNeXt were pretrained on ImageNet-21k / 22k. Thus, the 100% Specificity on ImageNet natural photographs is supported by the backbone's familiarity with ImageNet image distributions.
---
## 13. Scientific Interpretation and Claim Review
### 13.1 ViT Collapse (Phase 6 vs. Phase 7)
- **Report Claim:** ViT accuracy collapsed from 98.00% to 52.00% on external evaluation, "proving that ViT is a facial specialist whose decision boundaries do not transfer to general-scene diffusion images."
- **Audit Assessment (VERIFIED WITH QUALIFICATION):**
The numerical drop from 98.00% to 52.00% is verified. However, attributing the drop solely to generator identity is scientifically inaccurate. Phase 7 introduces a simultaneous **semantic shift** (faces $\to$ ImageNet objects) and **generative shift** (StyleGAN $\to$ Midjourney/ADM). The report should clearly state that domain shift and generator shift are confounded.
### 13.2 High Specificity vs. Low Sensitivity
- **Report Claim:** ConvNeXt and UnivFD achieved perfect 100% Specificity and strong latent discriminability ($\text{AUC} = 0.7926$ and $0.6662$).
- **Audit Assessment (LIMITATION):**
While mathematically true, this must be balanced by highlighting their severe sensitivity deficit: at $\tau = 0.50$, UnivFD missed 85% of fakes, and ConvNeXt missed 87% of fakes. Without threshold recalibration, neither model is viable as a practical standalone detector for these generators.
### 13.3 Fusion Dynamics
- **Report Claim:** Democratic consensus successfully filtered out all of ViT's 14 false alarms.
- **Audit Assessment (VERIFIED WITH LIMITATION):**
Majority voting achieved 100% Specificity ($FP = 0$). However, this conservative filtering also collapsed fake recall to **5.00%** (missing 95 out of 100 fakes). The ensemble accuracy of 52.50% is lower than UnivFD alone (57.50%) and ConvNeXt alone (56.50%). Democratic fusion does not improve detection performance over the best individual model on this benchmark.
---
## 14. Reproducibility Assessment
- **Manifest:** Frozen CSV with SHA-256 hashes allows exact reconstruction of the evaluation set.
- **Deterministic Pipeline:** Subprocess isolation guarantees identical predictions regardless of hardware memory constraints.
- **Zero Inconsistencies in Prediction Files:** All 200 records across all 4 CSVs match ground truth and threshold logic.
- **Reproducibility Rating:** **100% REPRODUCIBLE** from stored repository artifacts.
---
## 15. Discrepancies and Required Corrections
The audit identified two specific discrepancies that must be recorded:
### Discrepancy 1: Agreement Analysis Figures in Markdown Report
- **Location:** [`docs/PHASE7_EXTERNAL_GENERALIZATION_REPORT.md`](file:///c:/Users/user/Desktop/multi%20model%20connection/docs/PHASE7_EXTERNAL_GENERALIZATION_REPORT.md), Section 5 ("Cross-Model Agreement & Consensus Dynamics").
- **Existing Text:**
- All Three Detectors Agree: 91 out of 200 samples (45.50%).
- 86 agreed REAL, 5 agreed FAKE.
- ViT vs UnivFD $\kappa = 0.0988$ (Raw: 58.50%).
- ViT vs ConvNeXt $\kappa = 0.0520$ (Raw: 56.50%).
- UnivFD vs ConvNeXt $\kappa = 0.2849$ (Raw: 90.00%).
- **Actual Computed Values (from `phase7_combined_metrics.json` and verification audit):**
- All Three Detectors Agree: **145 out of 200 samples (72.50%)**.
- **86 agreed REAL on authentic images + 59 agreed REAL on fake images (false negatives) = 145 unanimous REAL agreements**. Exactly **0 samples** had all three detectors agree FAKE.
- ViT vs UnivFD: $\kappa = \mathbf{0.0284}$ (Raw agreement: **79.50%**).
- ViT vs ConvNeXt: $\kappa = \mathbf{-0.0529}$ (Raw agreement: **78.50%**).
- UnivFD vs ConvNeXt: $\kappa = \mathbf{0.0019}$ (Raw agreement: **87.00%**).
- **Explanation:** The report author erroneously added ViT's 86 True Negatives to the ensemble's 5 True Positives ($86 + 5 = 91$) under the mistaken assumption that 3-way consensus occurred on those 5 samples (in fact, only 2 detectors voted fake on those 5 samples), while overlooking the 59 fake samples where all three detectors unanimously agreed REAL.
### Discrepancy 2: Majority Vote ROC-AUC Characterization
- **Location:** [`docs/PHASE7_EXTERNAL_GENERALIZATION_REPORT.md`](file:///c:/Users/user/Desktop/multi%20model%20connection/docs/PHASE7_EXTERNAL_GENERALIZATION_REPORT.md), Section 3; [`phase7_combined_metrics.json`](file:///c:/Users/user/Desktop/multi%20model%20connection/benchmark/results/phase7_combined_metrics.json).
- **Issue:** Majority Voting ROC-AUC is reported as 0.7181, which is identical to Mean Fake Probability because `strategy_majority_vote` assigned the mean probability to `fake_probability`.
- **Correction:** The report must document that 0.7181 reflects the mean probability score vector. The ROC-AUC of Majority Voting under discrete vote counting is **0.6385**, and under hard binary decisions is **0.5250**.
---
## 16. Final Verification Verdict
### **VERDICT: VERIFIED WITH LIMITATIONS**
**Justification:**
1. **Calculations are Correct & Reproducible:** The dataset manifest, 200 image files, prediction CSVs, confusion matrices, accuracy, precision, recall, specificity, F1-scores, and Wilson confidence intervals are 100% internally consistent and mathematically verified against independent recomputation.
2. **Experimental Limitations Restrict Broad Generalizations:**
- Sample size is limited ($N = 200$: 100 Real, 50 Midjourney, 50 ADM).
- The 100 authentic ImageNet controls are reused between the Midjourney and ADM evaluations.
- At default $\tau = 0.50$, all models suffer from severe sensitivity failure (recall $\le 18\%$).
- Majority voting ROC-AUC in the existing report reflects continuous probability averaging rather than voting rules.
- Section 5 of the Markdown report contains reporting discrepancies regarding 3-way agreement and Cohen's Kappa that must be corrected.
---
## 17. Recommendations for the Next Research Phase
Based on the empirical findings of this post-execution audit:
1. **Threshold Recalibration & ROC Operating Point Optimization (Phase 8 Candidate):**
- Both ConvNeXt ($\text{AUC} = 0.7926$) and UnivFD ($\text{AUC} = 0.6662$) possess strong latent discriminability that is wasted at $\tau = 0.50$ (13% and 15% recall).
- Recalibrate decision thresholds using Youden's $J$ statistic or F1-maximization on a validation slice to assess whether practical detection performance can be unlocked without degrading specificity.
2. **Correct Agreement Analysis Reporting in Master Documentation:**
- Update [`docs/PHASE7_EXTERNAL_GENERALIZATION_REPORT.md`](file:///c:/Users/user/Desktop/multi%20model%20connection/docs/PHASE7_EXTERNAL_GENERALIZATION_REPORT.md) Section 5 to reflect the verified 3-way agreement count (145/200, 72.50%) and exact Cohen's Kappa figures.
- Clarify the scoring vector used for Majority Voting ROC-AUC.
3. **Domain-Aware / Gated Adaptive Decision Fusion:**
- Hierarchical ViT-Priority is optimal for facial media (Phase 6: 98%), while democratic / ConvNeXt-weighted fusion is superior for general scenes (Phase 7).
- Develop an upstream domain router (e.g., face detector gate) that routes face crops to ViT and general scene images to ConvNeXt / UnivFD.
4. **Expand External Generator Benchmark Diversity:**
- Expand beyond Midjourney and ADM to include modern open-weights latent diffusion (Stable Diffusion XL, Flux.1) and commercial generative systems (DALL-E 3) with balanced authentic photo sources.