Multimodel_setup / docs /PHASE6_GENERALIZATION_PLAN.md
Thiru1122's picture
feat(deepfake-detection): comprehensive cross-model evaluation, parity audits, and Phase 7 v2 reference preprocessing
151f388
|
Raw History Blame Contribute Delete
9.68 kB

A newer version of the Gradio SDK is available: 6.32.0

Upgrade

Phase 6 Generalization Plan: Independent Cross-Generator & Multi-Domain Evaluation

Project: Cross-Model Evaluation of Deepfake Image Detection Under Diverse Image Generation and Manipulation Conditions
Date: November 2026
Status: Approved Research & Execution Plan
Target: Generalization beyond Facial GANs to Multi-Generator Scenes, Diffusion Models, and Local Manipulations


1. Context & Motivation

Phase 4, Phase 5, and Phase 6 empirically confirmed the Domain-Specialization Dilemma in deepfake detection:

  1. ViT (dima806) dominates facial GAN detection with 98.00% accuracy on FFHQ vs. StyleGAN, but completely failed on diagnostic general scenes generated by modern diffusion systems (e.g., Google Imagen 3) and cut-and-paste manipulations.
  2. UnivFD (CLIP ViT-L/14) and ConvNeXt (DRCT / RAID) performed poorly on facial StyleGAN benchmarks (~51% accuracy), yet were designed specifically for cross-generator artifacts (ProGAN, LDM, Diffusion).
  3. Naive fusion (majority voting and mean probability) degrades facial accuracy by -35.00 percentage points due to correlated false-negative biases.

To build an authentic, generalizable detection architecture, the evaluation benchmark must expand beyond faces to encompass:

  • Diffusion Models (Stable Diffusion, Midjourney, DALL-E, Imagen).
  • Diverse Domains (Landscapes, Urban Scenes, Objects, Textures).
  • Local Manipulations (Face-swapping, Inpainting, Splicing, Copy-Move).
  • Pruned Uncompressed Baselines (Camera-native RAW imagery).

2. Target External Datasets & Benchmark Specifications

The evaluation suite will integrate 6 standardized external benchmarks across three functional axes:

                                  [Cross-Generator Benchmarks]
                                                |
         +--------------------------------------+--------------------------------------+
         |                                      |                                      |
   [Full-Frame Diffusion & GANs]        [Facial Deepfake Video/Frames]        [Local Splicing & RAW]
     - GenImage (NeurIPS 2023)             - FaceForensics++ (ICCV 2019)         - RAISE (Authentic RAW)
     - DiffusionForensics (AAAI 2024)      - Celeb-DF v2 (CVPR 2020)             - CASIA v2.0 (Splicing)

2.1 Full-Frame Multi-Generator Benchmarks

Dataset 1: GenImage (NeurIPS 2023)

  • Primary Focus: Broad-spectrum cross-generator evaluation across diverse general-scene categories.
  • Reference: Yang et al., "GenImage: A Million-Scale Dataset for Detecting AI-Generated Image", NeurIPS 2023.
  • Official Repository: https://github.com/GenImage-Dataset/GenImage
  • License: Research / Academic Non-Commercial Use (subject to ImageNet Terms).
  • Data Modality: Full general scenes ($256 \times 256$ to $512 \times 512$ RGB images).
  • Real Images: Subsampled from ImageNet (ILSVRC 2012 validation/train sets).
  • Generators Represented (8 Generators):
    1. Stable Diffusion v1.4 (CompVis)
    2. Stable Diffusion v1.5 (RunwayML)
    3. Midjourney (Commercial closed-source text-to-image)
    4. ADM (Ablated Diffusion Model)
    5. Glide (OpenAI text-guided diffusion)
    6. Wukong (Multilingual text-to-image diffusion)
    7. VQDM (Vector Quantized Diffusion Model)
    8. BigGAN (Large-scale GAN)
  • Strategic Role in Evaluation: Tests whether UnivFD and ConvNeXt outperform ViT on general-scene diffusion images.

Dataset 2: DiffusionForensics (AAAI 2024 / DIRE)

  • Primary Focus: Latent diffusion and pixel diffusion artifacts.
  • Reference: Ma et al., "DiffusionForensics: Deepfake Detection for Diffusion Models", AAAI 2024.
  • Official Repository: https://github.com/VCL-Lecun/DiffusionForensics
  • License: MIT License / Open Academic Research.
  • Data Modality: General scenes and portraits ($256 \times 256$ and $512 \times 512$).
  • Generators Represented:
    • LDM (Latent Diffusion Models)
    • Stable Diffusion v2.1
    • DALL-E 2
    • Guided Diffusion
    • Midjourney v5
  • Strategic Role in Evaluation: Probes high-frequency reconstruction errors and residual artifacts.

2.2 Advanced Facial Deepfake & Video Frame Benchmarks

Dataset 3: FaceForensics++ (FF++) (ICCV 2019)

  • Primary Focus: Facial reenactment and swapping across standardized compression levels.
  • Reference: Rössler et al., "FaceForensics++: Learning to Detect Manipulated Facial Images", ICCV 2019.
  • Official Repository: https://github.com/ondyari/FaceForensics
  • License: Academic Use Agreement (Non-Commercial).
  • Data Modality: High-resolution facial video frames (extracted at pristine and compressed rates).
  • Manipulation Algorithms:
    1. Deepfakes (Autoencoder-based facial replacement)
    2. Face2Face (Real-time facial reenactment)
    3. FaceSwap (Graphics-based 3D landmark swapping)
    4. NeuralTextures (Photometric reenactment)
    5. FaceShifter (Two-stage facial swap with occlusions)
  • Strategic Role in Evaluation: Evaluates robustness under video compression (c0 raw, c23 high-quality, c40 low-quality).

Dataset 4: Celeb-DF (v2) (CVPR 2020)

  • Primary Focus: Next-generation high-visual-quality deepfake face swapping.
  • Reference: Li et al., "Celeb-DF: A Large-scale Challenging Dataset for Deepfake Detection", CVPR 2020.
  • Official Repository: https://github.com/yuezunli/celeb-deepfakeforensic
  • License: Non-Commercial Research License.
  • Data Modality: 5,639 high-definition YouTube celebrity interview frames.
  • Strategic Role in Evaluation: Mitigates common low-resolution boundary artifacts present in older benchmarks; tests detector performance against advanced boundary blending.

2.3 Local Tampering, Splicing & Pristine Uncompressed Baselines

Dataset 5: RAISE (Raw Images for Authenticity Evaluation)

  • Primary Focus: Pristine, camera-sensor native ground truth for false-positive validation.
  • Reference: Pasquini et al., "RAISE: A Raw Images Dataset for Digital Image Forensics", ACM MMSys 2015.
  • Official Repository: http://raise.diei.unipv.it/
  • License: Creative Commons Attribution-NonCommercial (CC BY-NC 4.0).
  • Data Modality: 8,156 uncompressed camera-native TIFF/DNG images with complete EXIF sensor parameters.
  • Strategic Role in Evaluation: Serves as the ultimate negative (REAL) calibration control to establish authentic false alarm rates when compression artifacts are completely absent.

Dataset 6: CASIA Image Tampering Database (v2.0)

  • Primary Focus: Splicing, copy-move, and local image composition.
  • Reference: Dong et al., CASIA TIDE v2.0, Institute of Automation, Chinese Academy of Sciences.
  • Official Repository: https://github.com/namtpham/casia2groundtruth
  • License: Academic Research License.
  • Data Modality: 12,614 natural scene images with complex cut-and-paste manipulation boundaries.
  • Strategic Role in Evaluation: Evaluates whether detectors can identify localized tampered regions embedded within authentic natural backgrounds.

3. Sampling Strategy & Manifest Integration

To respect host hardware constraints (8 GB RAM, CPU inference) while ensuring statistically rigorous representation:

3.1 Stratified Subsample Architecture

For each external dataset, a frozen evaluation split of 100 REAL and 100 FAKE images will be sampled:

  • GenImage: 50 SD-v1.5, 50 Midjourney vs. 100 ImageNet Real.
  • DiffusionForensics: 50 LDM, 50 DALL-E 2 vs. 100 LSUN Real.
  • FaceForensics++: 25 Deepfakes, 25 Face2Face, 25 FaceSwap, 25 NeuralTextures vs. 100 Authentic YouTube frames (c23 compression).
  • Celeb-DF: 100 Celeb-DF swap frames vs. 100 Celeb-Real frames.
  • RAISE: 100 pristine sensor captures (all REAL negative controls).
  • CASIA v2.0: 100 local spliced/tampered scenes vs. 100 authentic scenes.

Total Subsampled Evaluation Corpus: 1,200 verified images across 6 datasets.

3.2 Automated Integrity Pipeline

Prior to evaluation, all external downloads must pass through:

  1. Image.open().verify() and RGB mode conversion.
  2. Canonical resizing to standardized aspect ratios.
  3. Cryptographic SHA-256 fingerprinting.
  4. Schema validation into benchmark/external_generalization_manifest.csv.

4. Hypothesis & Multi-Model Routing Strategy

Based on the empirical findings of Phase 6, we establish the following testable hypotheses:

Dataset Modality Predicted Strongest Model Predicted Weakest Model Proposed Ensemble Strategy
Facial GANs (FFHQ/StyleGAN) ViT (dima806) ConvNeXt / UnivFD ViT-Priority Routing
General-Scene Diffusion (GenImage) UnivFD / ConvNeXt ViT (dima806) CLIP/ConvNeXt Consensus
Video Deepfakes (FaceForensics++) ViT / ConvNeXt UnivFD Confidence-Weighted Softmax
Local Splicing (CASIA v2.0) ConvNeXt (DRCT) ViT (dima806) Local Artifact Detection
Pristine RAW (RAISE Controls) All (Target: Spec > 98%) None Conservative Real Thresholding

Next Step: Domain-Aware Dynamic Router

Instead of static voting, the multi-model engine will implement a lightweight Domain-Aware Dynamic Router:

  1. An input image is scanned for face presence (via MTCNN / RetinaFace or lightweight landmark check).
  2. If Facial Portrait: Route to ViT-Priority (preserving 98% accuracy).
  3. If General Scene / Landscape: Route to UnivFD / ConvNeXt consensus.
  4. If Ambiguous / Borderline: Query full ensemble with uncertainty thresholding.