Multimodel_setup / docs /PHASE6_COMPLETION_REPORT.md
Thiru1122's picture
feat(deepfake-detection): comprehensive cross-model evaluation, parity audits, and Phase 7 v2 reference preprocessing
151f388
|
Raw History Blame Contribute Delete
9.34 kB

A newer version of the Gradio SDK is available: 6.32.0

Upgrade

Phase 6 Completion Report: Cross-Model Deepfake Detection, Ensemble Classifier & Independent Evaluation

Project: Cross-Model Evaluation of Deepfake Image Detection Under Diverse Image Generation and Manipulation Conditions
Execution Mode: Autonomous Master Goal (Phase 6)
Date: November 2026
Status: All Phase 6 Operations Completed & Verified


1. Executive Summary

Phase 6 has successfully executed an end-to-end, empirical evaluation of all three active 2D deepfake image detectors (ViT, UnivFD, ConvNeXt/DRCT) across a pristine, leak-free test partition of 200 images (100 REAL, 100 FAKE).

Crucially, Phase 6 implemented and evaluated multi-model decision-fusion classifiers (Majority Voting, Mean Fake Probability, and ViT-Priority Fallback), uncovering the Ensemble Degradation Paradox and establishing a verified architectural paradigm for multi-model deepfake detection.


2. Deliverable Verification Audit

All required Phase 6 deliverables have been generated, statistically validated, and persisted on disk:

Deliverable Path Status Verification Detail
docs/PHASE6_EXECUTION_AUDIT.md Verified Model audit confirming active 2D models (vit, univfd, chen2024_convnext) and excluding incompatible 5D video model (df_model.pt).
benchmark/phase6_test_manifest.csv Verified 200 pristine samples (100 REAL, 100 FAKE) with zero train/val/test leakage, verified $256 \times 256$ RGB, unique SHA-256 hashes.
docs/PHASE6_DATASET_SELECTION_REPORT.md Verified Methodological report on candidate filtering, exclusion of historically touched indices, and decodability validation.
benchmark/results/phase6_vit_predictions.csv Verified 200 predictions from ViT (dima806) with probabilities, logits, and latency measurements.
benchmark/results/phase6_univfd_predictions.csv Verified 200 predictions from UnivFD (CLIP ViT-L/14) with probabilities, logits, and latency measurements.
benchmark/results/phase6_convnext_predictions.csv Verified 200 predictions from ConvNeXt (16_acc0.9993.pth) with probabilities, logits, and latency measurements.
benchmark/results/phase6_combined_predictions.csv Verified Per-sample comparative matrix covering all 3 detectors, all 3 fusion strategies, and consensus flags.
benchmark/results/phase6_combined_metrics.json Verified Comprehensive statistical metrics: Accuracy, BalAcc, Precision, Recall, Specificity, F1, 95% Wilson CIs, Cohen's Kappa, and latencies.
benchmark/results/phase6_model_comparison.csv Verified Consolidated performance comparison table across all 6 models and strategies.
docs/PHASE6_EVALUATION_REPORT.md Verified Comprehensive analytical report detailing empirical outcomes, failure modes, agreement analysis, and the ensemble degradation paradox.
docs/PHASE6_GENERALIZATION_PLAN.md Verified Scientific research plan detailing 6 external benchmarks (GenImage, DiffusionForensics, FF++, Celeb-DF, RAISE, CASIA) with verified URLs and licenses.
MODEL_STATUS_AND_PROJECT_PROGRESSION.md Verified Updated master progression log reflecting completed Phase 6 status.

3. Verified Empirical Results

Across the 200 test images (100 REAL authentic FFHQ faces, 100 FAKE StyleGAN faces):

+-----------------------------------+---------------+--------------------+---------------+---------------+---------------+
| Model / Strategy                  |  Accuracy (%) | 95% Wilson CI      | Recall (Fake) | Specificity   | F1-Score      |
+-----------------------------------+---------------+--------------------+---------------+---------------+---------------+
| ViT (dima806)                     |    98.00%     | [94.97%, 99.22%]   |    100.00%    |    96.00%     |    0.9804     |
| UnivFD (CLIP ViT-L/14)            |    51.50%     | [44.61%, 58.33%]   |     17.00%    |    86.00%     |    0.2595     |
| ConvNeXt (DRCT / RAID)            |    51.00%     | [44.12%, 57.84%]   |     16.00%    |    86.00%     |    0.2462     |
+-----------------------------------+---------------+--------------------+---------------+---------------+---------------+
| Ensemble: Majority Vote           |    63.00%     | [56.12%, 69.39%]   |     30.00%    |    96.00%     |    0.4478     |
| Ensemble: Mean Fake Probability   |    63.50%     | [56.63%, 69.86%]   |     31.00%    |    96.00%     |    0.4593     |
| Ensemble: ViT Priority Fallback   |    98.00%     | [94.97%, 99.22%]   |    100.00%    |    96.00%     |    0.9804     |
+-----------------------------------+---------------+--------------------+---------------+---------------+---------------+

4. Key Scientific & Engineering Insights

4.1 The "Ensemble Degradation Paradox"

A central finding is that naive democratic ensemble combination strictly degrades performance when detector domain competencies are asymmetric:

  • ViT alone detected 100 out of 100 fake faces ($Recall = 100.00%$).
  • UnivFD and ConvNeXt suffered from a shared false-negative bias on StyleGAN faces, missing 83% and 84% of fake faces, respectively.
  • Because UnivFD and ConvNeXt formed a $2\text{-to-}1$ majority, they outvoted ViT on 70 out of 100 fake faces, dropping ensemble recall down to 30.00% and crashing accuracy from 98.00% to 63.00% (a -35.00 percentage point collapse).
  • Conclusion: Equal-weighted majority voting or probability averaging must not be deployed across models trained on differing generative manifolds.

4.2 ViT-Priority Fallback as the Solution

The hierarchical ViT-Priority Fallback architecture preserved the full 98.00% accuracy of the expert model while maintaining secondary models strictly as fallback guards in case of model initialization or confidence failure.

4.3 Guaranteed Memory Safety on Constrained Host

Operating under strict Windows laptop constraints (8 GB RAM, CPU inference), our sequential subprocess isolation harness successfully executed 600 inference queries (200 per detector) plus 3 smoke tests with zero memory exhaustion errors (os error 1455), complete virtual memory reclamation, and deterministic reproducible outputs.


5. Master Goal Checklist Completion