Multimodel_setup / docs /PHASE6_GENERALIZATION_PLAN.md
Thiru1122's picture
feat(deepfake-detection): comprehensive cross-model evaluation, parity audits, and Phase 7 v2 reference preprocessing
151f388
|
Raw History Blame Contribute Delete
9.68 kB
# Phase 6 Generalization Plan: Independent Cross-Generator & Multi-Domain Evaluation
**Project:** Cross-Model Evaluation of Deepfake Image Detection Under Diverse Image Generation and Manipulation Conditions
**Date:** November 2026
**Status:** Approved Research & Execution Plan
**Target:** Generalization beyond Facial GANs to Multi-Generator Scenes, Diffusion Models, and Local Manipulations
---
## 1. Context & Motivation
Phase 4, Phase 5, and Phase 6 empirically confirmed the **Domain-Specialization Dilemma** in deepfake detection:
1. **ViT (`dima806`)** dominates facial GAN detection with **98.00% accuracy** on FFHQ vs. StyleGAN, but completely failed on diagnostic general scenes generated by modern diffusion systems (e.g., Google Imagen 3) and cut-and-paste manipulations.
2. **UnivFD (`CLIP ViT-L/14`)** and **ConvNeXt (`DRCT / RAID`)** performed poorly on facial StyleGAN benchmarks (~51% accuracy), yet were designed specifically for cross-generator artifacts (ProGAN, LDM, Diffusion).
3. Naive fusion (majority voting and mean probability) degrades facial accuracy by **-35.00 percentage points** due to correlated false-negative biases.
To build an authentic, generalizable detection architecture, the evaluation benchmark must expand beyond faces to encompass:
- **Diffusion Models** (Stable Diffusion, Midjourney, DALL-E, Imagen).
- **Diverse Domains** (Landscapes, Urban Scenes, Objects, Textures).
- **Local Manipulations** (Face-swapping, Inpainting, Splicing, Copy-Move).
- **Pruned Uncompressed Baselines** (Camera-native RAW imagery).
---
## 2. Target External Datasets & Benchmark Specifications
The evaluation suite will integrate 6 standardized external benchmarks across three functional axes:
```
[Cross-Generator Benchmarks]
|
+--------------------------------------+--------------------------------------+
| | |
[Full-Frame Diffusion & GANs] [Facial Deepfake Video/Frames] [Local Splicing & RAW]
- GenImage (NeurIPS 2023) - FaceForensics++ (ICCV 2019) - RAISE (Authentic RAW)
- DiffusionForensics (AAAI 2024) - Celeb-DF v2 (CVPR 2020) - CASIA v2.0 (Splicing)
```
---
### 2.1 Full-Frame Multi-Generator Benchmarks
#### Dataset 1: GenImage (NeurIPS 2023)
- **Primary Focus:** Broad-spectrum cross-generator evaluation across diverse general-scene categories.
- **Reference:** Yang et al., *"GenImage: A Million-Scale Dataset for Detecting AI-Generated Image"*, NeurIPS 2023.
- **Official Repository:** `https://github.com/GenImage-Dataset/GenImage`
- **License:** Research / Academic Non-Commercial Use (subject to ImageNet Terms).
- **Data Modality:** Full general scenes ($256 \times 256$ to $512 \times 512$ RGB images).
- **Real Images:** Subsampled from ImageNet (ILSVRC 2012 validation/train sets).
- **Generators Represented (8 Generators):**
1. *Stable Diffusion v1.4* (CompVis)
2. *Stable Diffusion v1.5* (RunwayML)
3. *Midjourney* (Commercial closed-source text-to-image)
4. *ADM* (Ablated Diffusion Model)
5. *Glide* (OpenAI text-guided diffusion)
6. *Wukong* (Multilingual text-to-image diffusion)
7. *VQDM* (Vector Quantized Diffusion Model)
8. *BigGAN* (Large-scale GAN)
- **Strategic Role in Evaluation:** Tests whether UnivFD and ConvNeXt outperform ViT on general-scene diffusion images.
#### Dataset 2: DiffusionForensics (AAAI 2024 / DIRE)
- **Primary Focus:** Latent diffusion and pixel diffusion artifacts.
- **Reference:** Ma et al., *"DiffusionForensics: Deepfake Detection for Diffusion Models"*, AAAI 2024.
- **Official Repository:** `https://github.com/VCL-Lecun/DiffusionForensics`
- **License:** MIT License / Open Academic Research.
- **Data Modality:** General scenes and portraits ($256 \times 256$ and $512 \times 512$).
- **Generators Represented:**
- LDM (Latent Diffusion Models)
- Stable Diffusion v2.1
- DALL-E 2
- Guided Diffusion
- Midjourney v5
- **Strategic Role in Evaluation:** Probes high-frequency reconstruction errors and residual artifacts.
---
### 2.2 Advanced Facial Deepfake & Video Frame Benchmarks
#### Dataset 3: FaceForensics++ (FF++) (ICCV 2019)
- **Primary Focus:** Facial reenactment and swapping across standardized compression levels.
- **Reference:** Rössler et al., *"FaceForensics++: Learning to Detect Manipulated Facial Images"*, ICCV 2019.
- **Official Repository:** `https://github.com/ondyari/FaceForensics`
- **License:** Academic Use Agreement (Non-Commercial).
- **Data Modality:** High-resolution facial video frames (extracted at pristine and compressed rates).
- **Manipulation Algorithms:**
1. *Deepfakes* (Autoencoder-based facial replacement)
2. *Face2Face* (Real-time facial reenactment)
3. *FaceSwap* (Graphics-based 3D landmark swapping)
4. *NeuralTextures* (Photometric reenactment)
5. *FaceShifter* (Two-stage facial swap with occlusions)
- **Strategic Role in Evaluation:** Evaluates robustness under video compression (c0 raw, c23 high-quality, c40 low-quality).
#### Dataset 4: Celeb-DF (v2) (CVPR 2020)
- **Primary Focus:** Next-generation high-visual-quality deepfake face swapping.
- **Reference:** Li et al., *"Celeb-DF: A Large-scale Challenging Dataset for Deepfake Detection"*, CVPR 2020.
- **Official Repository:** `https://github.com/yuezunli/celeb-deepfakeforensic`
- **License:** Non-Commercial Research License.
- **Data Modality:** 5,639 high-definition YouTube celebrity interview frames.
- **Strategic Role in Evaluation:** Mitigates common low-resolution boundary artifacts present in older benchmarks; tests detector performance against advanced boundary blending.
---
### 2.3 Local Tampering, Splicing & Pristine Uncompressed Baselines
#### Dataset 5: RAISE (Raw Images for Authenticity Evaluation)
- **Primary Focus:** Pristine, camera-sensor native ground truth for false-positive validation.
- **Reference:** Pasquini et al., *"RAISE: A Raw Images Dataset for Digital Image Forensics"*, ACM MMSys 2015.
- **Official Repository:** `http://raise.diei.unipv.it/`
- **License:** Creative Commons Attribution-NonCommercial (CC BY-NC 4.0).
- **Data Modality:** 8,156 uncompressed camera-native TIFF/DNG images with complete EXIF sensor parameters.
- **Strategic Role in Evaluation:** Serves as the ultimate negative (REAL) calibration control to establish authentic false alarm rates when compression artifacts are completely absent.
#### Dataset 6: CASIA Image Tampering Database (v2.0)
- **Primary Focus:** Splicing, copy-move, and local image composition.
- **Reference:** Dong et al., CASIA TIDE v2.0, Institute of Automation, Chinese Academy of Sciences.
- **Official Repository:** `https://github.com/namtpham/casia2groundtruth`
- **License:** Academic Research License.
- **Data Modality:** 12,614 natural scene images with complex cut-and-paste manipulation boundaries.
- **Strategic Role in Evaluation:** Evaluates whether detectors can identify localized tampered regions embedded within authentic natural backgrounds.
---
## 3. Sampling Strategy & Manifest Integration
To respect host hardware constraints (8 GB RAM, CPU inference) while ensuring statistically rigorous representation:
### 3.1 Stratified Subsample Architecture
For each external dataset, a frozen evaluation split of **100 REAL and 100 FAKE** images will be sampled:
- **GenImage:** 50 SD-v1.5, 50 Midjourney vs. 100 ImageNet Real.
- **DiffusionForensics:** 50 LDM, 50 DALL-E 2 vs. 100 LSUN Real.
- **FaceForensics++:** 25 Deepfakes, 25 Face2Face, 25 FaceSwap, 25 NeuralTextures vs. 100 Authentic YouTube frames (c23 compression).
- **Celeb-DF:** 100 Celeb-DF swap frames vs. 100 Celeb-Real frames.
- **RAISE:** 100 pristine sensor captures (all REAL negative controls).
- **CASIA v2.0:** 100 local spliced/tampered scenes vs. 100 authentic scenes.
**Total Subsampled Evaluation Corpus:** 1,200 verified images across 6 datasets.
### 3.2 Automated Integrity Pipeline
Prior to evaluation, all external downloads must pass through:
1. `Image.open().verify()` and RGB mode conversion.
2. Canonical resizing to standardized aspect ratios.
3. Cryptographic SHA-256 fingerprinting.
4. Schema validation into `benchmark/external_generalization_manifest.csv`.
---
## 4. Hypothesis & Multi-Model Routing Strategy
Based on the empirical findings of Phase 6, we establish the following testable hypotheses:
| Dataset Modality | Predicted Strongest Model | Predicted Weakest Model | Proposed Ensemble Strategy |
| :--- | :--- | :--- | :--- |
| **Facial GANs (FFHQ/StyleGAN)** | ViT (`dima806`) | ConvNeXt / UnivFD | **ViT-Priority Routing** |
| **General-Scene Diffusion (GenImage)** | UnivFD / ConvNeXt | ViT (`dima806`) | **CLIP/ConvNeXt Consensus** |
| **Video Deepfakes (FaceForensics++)** | ViT / ConvNeXt | UnivFD | **Confidence-Weighted Softmax** |
| **Local Splicing (CASIA v2.0)** | ConvNeXt (DRCT) | ViT (`dima806`) | **Local Artifact Detection** |
| **Pristine RAW (RAISE Controls)** | All (Target: Spec > 98%) | None | **Conservative Real Thresholding** |
### Next Step: Domain-Aware Dynamic Router
Instead of static voting, the multi-model engine will implement a lightweight **Domain-Aware Dynamic Router**:
1. An input image is scanned for face presence (via MTCNN / RetinaFace or lightweight landmark check).
2. If **Facial Portrait:** Route to **ViT-Priority** (preserving 98% accuracy).
3. If **General Scene / Landscape:** Route to **UnivFD / ConvNeXt** consensus.
4. If **Ambiguous / Borderline:** Query full ensemble with uncertainty thresholding.