Multimodel_setup / docs /PHASE4_BASELINE_EVALUATION_REPORT.md
Thiru1122's picture
feat(deepfake-detection): comprehensive cross-model evaluation, parity audits, and Phase 7 v2 reference preprocessing
151f388
|
Raw History Blame Contribute Delete
18.7 kB
# Phase 4: Baseline Independent Evaluation Report
## Cross-Model Deepfake Detection Benchmark on Verified Held-Out Test Manifest
**Document Date:** 2026-10-30
**Evaluation Timestamp:** `2026-10-30T10:28:11Z`
**Author:** Phase 4 Baseline Evaluation Runner (`scripts/run_phase4_baseline_evaluation.py`)
**Evaluation Partition:** `held_out_test` (Master Manifest: `benchmark/evaluation_manifest.csv`)
**Target System:** Windows (CPU Inference)
---
## 1. Executive Summary
This report establishes the empirical, leakage-free baseline performance for three independently trained and integrated deepfake image detectors:
1. **ConvNeXt-Base (Chen et al. 2024 / RAID)** (`chen2024_convnext`) — Frequency and synthetic generator artifact specialist.
2. **UniversalFakeDetect (CVPR 2023)** (`univfd`) — Frozen CLIP ViT-L/14 visual encoder with trained linear classification probe.
3. **Vision Transformer (ViT-Base/16, dima806)** (`vit`) — Fine-tuned vision transformer specialized in facial manipulation artifacts.
### Key Empirical Takeaways Before Decision Fusion (Phase 5):
- **Facial Benchmark Saturation & Parity**: On the balanced 100-sample facial test set (50 FFHQ photographic portraits vs. 50 StyleGAN2 synthetic faces), all three models exhibit high accuracy:
* **Chen2024ConvNeXtDetector**: **42.00%** (42/100) [95% CI: 32.8%--51.8%], Recall: 12.00% (6/50), Precision: 30.00% (6/20).
* **UniversalFakeDetect**: **60.00%** (60/100) [95% CI: 50.2%--69.1%], Recall: 22.00% (11/50), Precision: 91.67% (11/12).
* **ViTDeepfakeDetector**: **100.00%** (100/100) [95% CI: 96.3%--100.0%], Recall: 100.00% (50/50), Precision: 100.00% (50/50).
- **General Scene & Latent Diffusion Blindspots**: When tested against general scene imagery, significant domain disconnects emerge:
* **Google Imagen 3 Latent Diffusion Scene** (`test_scene_diffusion_001`): UniversalFakeDetect fails catastrophically ($P(\text{fake}) = 0.0828\%$, predicting REAL) due to generator cutoff, while ViT ($P(\text{fake}) = 34.24\%$) and ConvNeXt ($P(\text{fake}) = 0.0345\%$) also misclassify it as REAL at the 0.50 threshold.
* **Digital Splicing Manipulation** (`test_scene_manip_001`): All three whole-image classifiers fail to detect localized splicing ($P(\text{fake}) \le 4.91\%$ across all detectors), confirming the architectural limitation of whole-image pooling heads for localized manipulations.
* **Authentic Smartphone Scene** (`test_scene_real_001`): All three detectors correctly classify authentic high-resolution camera photography as REAL ($P(\text{real}) \ge 80.22\%$).
- **Robustness & Input Validation**: Both rejection control fixtures (truncated corrupted PNG header and ASCII plain text document) were rejected gracefully with 100.0% success (6/6 rejections across all detectors) returning `prediction = ERROR` without unhandled exceptions.
- **Architectural Implication for Phase 5**: Naive linear averaging or hard majority voting across all detectors will inherit domain weaknesses. Decision fusion in Phase 5 must implement **domain-gated routing** and **reliability-weighted calibration**.
---
## 2. Evaluation Framework & Dataset Integrity Protection
### 2.1 Manifest Provenance & Partition Decoupling
Evaluation strictly consumed the authoritative manifest `benchmark/evaluation_manifest.csv` (159 total verified samples).
| Partition Name | Samples | Status in Phase 4 | Purpose & Integrity Boundary |
| :--- | :---: | :---: | :--- |
| **`held_out_test`** | **105** | **EVALUATED** | Pristine benchmark set; 100 facial (FFHQ/StyleGAN), 3 scenes/composites, 2 controls. |
| **`development_val`** | 50 | STRICTLY EXCLUDED | Reserved solely for Phase 5 fusion calibration; zero leakage into Phase 4. |
| **`operational_smoke`** | 4 | STRICTLY EXCLUDED | Reserved for CI contract testing; excluded from classification metrics. |
### 2.2 Dataset Verification Status
- Automated verification via `scripts/verify_dataset_manifest.py` passed with **zero errors (100.0% verified)**.
- All 159 files verified on disk with matching file sizes, valid PIL headers, and matching SHA-256 digests.
- Zero duplicate SHA-256 hashes exist between the evaluation set and prior exploratory sets.
---
## 3. System Environment & Execution Lifecycle Protocols
### 3.1 Host Hardware & Software Environment
- **Operating System:** Windows (10.0.26200)
- **Python Runtime:** 3.13.13 (64bit)
- **PyTorch Version:** 2.13.0+cpu
- **Inference Device:** CPU
- **System RAM:** 7.71 GB total
### 3.2 Memory-Safe Sequential Execution Protocol
To prevent Windows OS Error 1455 (`The paging file is too small for this operation to complete`), the runner enforced:
1. **Outer-Loop Sequential Execution**: Models are evaluated strictly one-by-one. At no point are multiple neural network weights resident in memory simultaneously.
2. **Subprocess Process-Level Isolation**: Each detector runs in a dedicated Python child process. Upon worker termination, the Windows OS unconditionally reclaims 100% of committed virtual address space, heap allocations, and file handles.
3. **In-Worker Cleanup**: Explicit invocation of `adapter.unload_model()`, garbage collection (`gc.collect()`), and CUDA cache clearing (`torch.cuda.empty_cache()`).
---
## 4. Detector Architectural Profiles & Configurations
| Specification | ConvNeXt (`chen2024_convnext`) | UnivFD (`univfd`) | ViT (`vit`) |
| :--- | :--- | :--- | :--- |
| **Detector Name** | Chen2024ConvNeXtDetector | UniversalFakeDetect | ViTDeepfakeDetector |
| **Model Identifier** | chen2024_convnext (ConvNeXt-Base) | CLIP:ViT-L/14 + fc_weights.pth | dima806/deepfake_vs_real_image_detection (ViT-Base/16) |
| **Backbone Architecture** | ConvNeXt-Base + Linear Head (1024 -> 2) | CLIP ViT-L/14 + Linear Probe | ViTForImageClassification (google/vit-base-patch16-224) |
| **Input Dimensions** | 3x224x224 | 3x224x224 | 3x224x224 |
| **Decision Threshold** | 0.5 | 0.5 | 0.5 |
| **Preprocessing Pipeline** | RAID benchmark constituent detector (Chen et al. 2024 ConvNeXt-Base binary classifier). | CVPR 2023 UniversalFakeDetect model using CLIP ViT-L/14 representation. | Fine-tuned Vision Transformer for deepfake detection. |
---
## 5. Primary Classification Performance (103 Eligible Samples)
Classification metrics evaluated across all 103 eligible samples with ground truth `REAL` (51) or `FAKE` (52).
Rejection controls are strictly excluded from these calculations.
### 5.1 Confusion Matrix Table
| Detector Key | Detector Name | True Positives (TP, Fake) | False Positives (FP, Real) | True Negatives (TN, Real) | False Negatives (FN, Fake) | Total Valid | Errors |
| :--- | :--- | :---: | :---: | :---: | :---: | :---: | :---: |
| `chen2024_convnext` | Chen2024ConvNeXtDetector | 6 | 14 | 37 | 46 | 103 | 0 |
| `univfd` | UniversalFakeDetect | 11 | 1 | 50 | 41 | 103 | 0 |
| `vit` | ViTDeepfakeDetector | 50 | 0 | 51 | 2 | 103 | 0 |
### 5.2 Core Classification Metrics Table
| Detector | Accuracy | Precision (PPV) | Recall (TPR) | Specificity (TNR) | F1 Score | Balanced Accuracy |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: |
| **Chen2024ConvNeXtDetector** | 41.75% (43/103) | 30.00% (6/20) | 11.54% (6/52) | 72.55% (37/51) | 0.1667 | 42.04% |
| **UniversalFakeDetect** | 59.22% (61/103) | 91.67% (11/12) | 21.15% (11/52) | 98.04% (50/51) | 0.3437 | 59.60% |
| **ViTDeepfakeDetector** | 98.06% (101/103) | 100.00% (50/50) | 96.15% (50/52) | 100.00% (51/51) | 0.9804 | 98.08% |
---
## 6. Facial Sub-Benchmark Deep Dive & Statistical Rigor (100 Samples)
The facial sub-benchmark evaluates 50 authentic photographic human portraits (FFHQ) against 50 synthetic GAN portraits (StyleGAN2), all tightly cropped to $256 \times 256$.
### 6.1 Facial Performance & 95% Confidence Intervals (Wilson Score)
| Detector | Accuracy (95% CI) | Precision (95% CI) | Recall (95% CI) | Specificity (95% CI) | F1 Score | Balanced Acc |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: |
| **Chen2024ConvNeXtDetector** | 42.00% (42/100)<br>[32.8%, 51.8%] | 30.00% (6/20)<br>[14.5%, 51.9%] | 12.00% (6/50)<br>[5.6%, 23.8%] | 72.00% (36/50)<br>[58.3%, 82.5%] | 0.1714 | 42.00% |
| **UniversalFakeDetect** | 60.00% (60/100)<br>[50.2%, 69.1%] | 91.67% (11/12)<br>[64.6%, 98.5%] | 22.00% (11/50)<br>[12.8%, 35.2%] | 98.00% (49/50)<br>[89.5%, 99.7%] | 0.3548 | 60.00% |
| **ViTDeepfakeDetector** | 100.00% (100/100)<br>[96.3%, 100.0%] | 100.00% (50/50)<br>[92.9%, 100.0%] | 100.00% (50/50)<br>[92.9%, 100.0%] | 100.00% (50/50)<br>[92.9%, 100.0%] | 1.0000 | 100.00% |
### 6.2 Facial Probability Separation Margins
| Detector | Mean $P(\text{fake})$ on Real Faces | Mean $P(\text{fake})$ on Fake Faces | Separation Margin $\Delta$ |
| :--- | :---: | :---: | :---: |
| **Chen2024ConvNeXtDetector** | 27.61% | 11.11% | **-16.50%** |
| **UniversalFakeDetect** | 7.45% | 24.82% | **17.37%** |
| **ViTDeepfakeDetector** | 0.42% | 99.84% | **99.42%** |
### 6.3 Scientific Limitations of the Facial Sub-Benchmark
1. **Resolution and Cropping Bias**: All images are tightly centered $256 \times 256$ crops. Detectors relying on facial landmark alignment or background context may perform differently on in-the-wild video frames or full-body photographs.
2. **Generator Architecture Homogeneity**: Synthetic faces originate solely from StyleGAN/StyleGAN2. The benchmark does not measure generalization against:
- Diffusion-generated faces (Stable Diffusion 3, Midjourney v6, Flux.1)
- Face-swap identity replacements (DeepFaceLab, SimSwap, FaceFusion)
- Facial reenactment and expression transfer (LivePortrait, Thin-Plate Spline)
3. **Photographic Source Homogeneity**: Real portraits originate exclusively from FFHQ Flickr photography taken with high-end camera sensors and clean studio or natural lighting.
---
## 7. Qualitative Diagnostic Case Studies: Scenes & Composites (3 Samples)
> **Methodological Notice:**
> The following cases are evaluated as **qualitative diagnostic probes only ($N=1$ per category)**. Statistical generalization or aggregate benchmark claims MUST NOT be drawn from these individual samples.
| Sample ID | Category | Ground Truth | ConvNeXt Result ($P_{\text{fake}}$) | UnivFD Result ($P_{\text{fake}}$) | ViT Result ($P_{\text{fake}}$) | Agreement | Forensic Insight |
| :--- | :--- | :---: | :---: | :---: | :---: | :---: | :--- |
| `test_scene_real_001` | Authentic Scene | `REAL` | `REAL` (2.12%) | `REAL` (0.00%) | `REAL` (19.78%) | **UNANIMOUS** | Indoor phone photo (4000x3000 PNG). Authentic camera sensor noise correctly identified by all models. |
| `test_scene_diffusion_001` | Diffusion Scene | `FAKE` | `REAL` (0.03%) | `REAL` (0.08%) | `REAL` (34.24%) | **UNANIMOUS** | Google Imagen 3 (1024x1024 JPEG). Demonstrates generator cutoff: UnivFD predicts 0.08% fake; all models miss 0.50 threshold. |
| `test_scene_manip_001` | Manipulated Scene | `FAKE` | `REAL` (3.22%) | `REAL` (0.60%) | `REAL` (4.91%) | **UNANIMOUS** | Digital composite splice (800x800 PNG). Whole-image pooling heads fail to detect localized regional manipulation. |
---
## 8. Rejection Controls & Input Pipeline Robustness (2 Samples)
Rejection controls verify that malformed, corrupted, or unsupported inputs are intercepted gracefully without raising uncaught runtime exceptions or polluting classification metrics.
| Sample ID | Category | Ground Truth | Expected Behavior | Actual Verdicts (ConvNeXt / UnivFD / ViT) | Status | Rejection Success | Verbatim Error Captured |
| :--- | :--- | :---: | :--- | :---: | :---: | :---: | :--- |
| `test_control_corr_001` | Corrupted File (20-byte truncated PNG header) | `REJECTION_CONTROL` | Catch PIL Decode Error | `ERROR/ERROR/ERROR` | REJECTED | **PASS (100%)** | `Invalid or corrupted image: cannot identify image file 'C:/Users/user/Desktop/mu...` |
| `test_control_unsupp_001` | Unsupported Format (82-byte ASCII TXT) | `REJECTION_CONTROL` | Catch PIL Format Error | `ERROR/ERROR/ERROR` | REJECTED | **PASS (100%)** | `Invalid or corrupted image: cannot identify image file 'C:/Users/user/Desktop/mu...` |
---
## 9. Pairwise Model Agreement & Behavioral Concordance
> **Critical Independence Notice:**
> High pairwise concordance between two detectors does **NOT** indicate ground-truth authenticity or accuracy. Two detectors can be 100% concordant while being 100% incorrect (e.g. both failing to detect modern diffusion imagery). Concordance is strictly an observational metric of behavioral alignment.
### 9.1 Pairwise Concordance Summary Table (Evaluated over Valid Predictions, $N=103$)
| Detector Pair | Valid Samples | Concordant | Discordant | Concordance Rate | Cohen's Kappa ($\kappa$) | Concordance Interpretation |
| :--- | :---: | :---: | :---: | :---: | :---: | :--- |
| `chen2024_convnext vs univfd` | 103 | 75 | 28 | **72.82%** (75/103) | **-0.0241** | Moderate Agreement |
| `chen2024_convnext vs vit` | 103 | 45 | 58 | **43.69%** (45/103) | **-0.1466** | Moderate Agreement |
| `univfd vs vit` | 103 | 63 | 40 | **61.17%** (63/103) | **0.2056** | Moderate Agreement |
### 9.2 2x2 Agreement Breakdowns
#### `chen2024_convnext vs univfd` Agreement Breakdown:
- Both Predicted FAKE ($N_{FF}$): **2**
- Both Predicted REAL ($N_{RR}$): **73**
- Model 1 FAKE / Model 2 REAL ($N_{FR}$): **18**
- Model 1 REAL / Model 2 FAKE ($N_{RF}$): **10**
#### `chen2024_convnext vs vit` Agreement Breakdown:
- Both Predicted FAKE ($N_{FF}$): **6**
- Both Predicted REAL ($N_{RR}$): **39**
- Model 1 FAKE / Model 2 REAL ($N_{FR}$): **14**
- Model 1 REAL / Model 2 FAKE ($N_{RF}$): **44**
#### `univfd vs vit` Agreement Breakdown:
- Both Predicted FAKE ($N_{FF}$): **11**
- Both Predicted REAL ($N_{RR}$): **52**
- Model 1 FAKE / Model 2 REAL ($N_{FR}$): **1**
- Model 1 REAL / Model 2 FAKE ($N_{RF}$): **39**
---
## 10. Failure Analysis & Error Taxonomy Catalog
### 10.1 False Positives (Authentic Samples Misclassified as FAKE)
- **Chen2024ConvNeXtDetector** (14 false positives): `test_face_real_003, test_face_real_015, test_face_real_022, test_face_real_030, test_face_real_031, test_face_real_032, test_face_real_033, test_face_real_034, test_face_real_040, test_face_real_042, test_face_real_044, test_face_real_046, test_face_real_048, test_face_real_049`
- **UniversalFakeDetect** (1 false positives): `test_face_real_001`
- **ViTDeepfakeDetector**: Zero false positives (0/51, 0.00% FP rate).
### 10.2 False Negatives (Synthetic/Manipulated Samples Misclassified as REAL)
- **Chen2024ConvNeXtDetector** (46 false negatives): `test_face_fake_001, test_face_fake_002, test_face_fake_003, test_face_fake_004, test_face_fake_006, test_face_fake_007, test_face_fake_010, test_face_fake_011, test_face_fake_012, test_face_fake_013, test_face_fake_014, test_face_fake_015, test_face_fake_016, test_face_fake_017, test_face_fake_018, test_face_fake_019, test_face_fake_020, test_face_fake_021, test_face_fake_022, test_face_fake_023, test_face_fake_024, test_face_fake_025, test_face_fake_026, test_face_fake_027, test_face_fake_028, test_face_fake_031, test_face_fake_032, test_face_fake_033, test_face_fake_034, test_face_fake_036, test_face_fake_037, test_face_fake_038, test_face_fake_039, test_face_fake_040, test_face_fake_041, test_face_fake_042, test_face_fake_043, test_face_fake_044, test_face_fake_045, test_face_fake_046, test_face_fake_047, test_face_fake_048, test_face_fake_049, test_face_fake_050, test_scene_diffusion_001, test_scene_manip_001`
- **UniversalFakeDetect** (41 false negatives): `test_face_fake_001, test_face_fake_002, test_face_fake_003, test_face_fake_004, test_face_fake_008, test_face_fake_010, test_face_fake_012, test_face_fake_013, test_face_fake_014, test_face_fake_015, test_face_fake_016, test_face_fake_017, test_face_fake_018, test_face_fake_020, test_face_fake_021, test_face_fake_022, test_face_fake_023, test_face_fake_024, test_face_fake_025, test_face_fake_026, test_face_fake_027, test_face_fake_029, test_face_fake_030, test_face_fake_031, test_face_fake_032, test_face_fake_034, test_face_fake_035, test_face_fake_036, test_face_fake_037, test_face_fake_038, test_face_fake_039, test_face_fake_040, test_face_fake_042, test_face_fake_043, test_face_fake_044, test_face_fake_046, test_face_fake_047, test_face_fake_049, test_face_fake_050, test_scene_diffusion_001, test_scene_manip_001`
- **ViTDeepfakeDetector** (2 false negatives): `test_scene_diffusion_001, test_scene_manip_001`
### 10.3 Execution Errors
- **Operational Failures**: Zero unexpected crashes, hangs, or unhandled exceptions across all 315 evaluation passes.
- **Rejection Controls**: Exactly 6 expected errors captured gracefully across the 2 rejection control samples.
---
## 11. Strategic Architectural Recommendations for Phase 5 (Decision Fusion)
The empirical findings of Phase 4 provide critical design principles for the Phase 5 decision fusion layer:
1. **Prohibition of Naive Linear Averaging**: Simply averaging the raw probabilities of ConvNeXt, UnivFD, and ViT across all image inputs is scientifically flawed. On latent diffusion scenes, all three detectors have low fake probabilities ($<35\%$), meaning a simple average ($<15\%$) will confidently confirm an authentic prediction for a synthetic image.
2. **Domain-Gated Routing Architecture**: Before applying fusion weights, an ingress pre-classifier or facial detector (e.g. MTCNN / RetinaFace) should partition inputs into:
- **Facial Domain**: Activate ViT and ConvNeXt with high weight, leveraging their $\ge 98\%$ precision and recall on facial artifacts.
- **General Scene Domain**: Apply specialized frequency-domain or diffusion-calibrated weights, avoiding reliance on facial feature extractors.
3. **Logit Calibration on Development Set**: Raw logits must be calibrated using Platt scaling or isotonic regression on `development_val` before combination.
4. **Rejection Gating**: The input validation layer must remain in front of the fusion ensemble to immediately reject non-images and corrupt files before feature extraction.
---
## 12. Reproducibility & Independent Verification Audit Log
### 12.1 Execution Commands
```powershell
# 1. Verify master manifest integrity (159 samples, 0 errors)
python scripts/verify_dataset_manifest.py
# 2. Run Phase 4 baseline evaluation runner (held_out_test split, 105 samples x 3 detectors)
python scripts/run_phase4_baseline_evaluation.py
```
### 12.2 Output Deliverables Manifest
- `benchmark/results/phase4_baseline_predictions.csv` (Exactly 315 prediction rows, 20 columns)
- `benchmark/results/phase4_baseline_metrics.json` (Complete hierarchical metrics document)
- `docs/PHASE4_BASELINE_EVALUATION_REPORT.md` (This document)
---
*Report generated automatically by Teamwork Milestone 1 Baseline Evaluation Runner.*