Spaces:
Running on Zero
Running on Zero
File size: 9,679 Bytes
151f388 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 | # Phase 6 Generalization Plan: Independent Cross-Generator & Multi-Domain Evaluation
**Project:** Cross-Model Evaluation of Deepfake Image Detection Under Diverse Image Generation and Manipulation Conditions
**Date:** November 2026
**Status:** Approved Research & Execution Plan
**Target:** Generalization beyond Facial GANs to Multi-Generator Scenes, Diffusion Models, and Local Manipulations
---
## 1. Context & Motivation
Phase 4, Phase 5, and Phase 6 empirically confirmed the **Domain-Specialization Dilemma** in deepfake detection:
1. **ViT (`dima806`)** dominates facial GAN detection with **98.00% accuracy** on FFHQ vs. StyleGAN, but completely failed on diagnostic general scenes generated by modern diffusion systems (e.g., Google Imagen 3) and cut-and-paste manipulations.
2. **UnivFD (`CLIP ViT-L/14`)** and **ConvNeXt (`DRCT / RAID`)** performed poorly on facial StyleGAN benchmarks (~51% accuracy), yet were designed specifically for cross-generator artifacts (ProGAN, LDM, Diffusion).
3. Naive fusion (majority voting and mean probability) degrades facial accuracy by **-35.00 percentage points** due to correlated false-negative biases.
To build an authentic, generalizable detection architecture, the evaluation benchmark must expand beyond faces to encompass:
- **Diffusion Models** (Stable Diffusion, Midjourney, DALL-E, Imagen).
- **Diverse Domains** (Landscapes, Urban Scenes, Objects, Textures).
- **Local Manipulations** (Face-swapping, Inpainting, Splicing, Copy-Move).
- **Pruned Uncompressed Baselines** (Camera-native RAW imagery).
---
## 2. Target External Datasets & Benchmark Specifications
The evaluation suite will integrate 6 standardized external benchmarks across three functional axes:
```
[Cross-Generator Benchmarks]
|
+--------------------------------------+--------------------------------------+
| | |
[Full-Frame Diffusion & GANs] [Facial Deepfake Video/Frames] [Local Splicing & RAW]
- GenImage (NeurIPS 2023) - FaceForensics++ (ICCV 2019) - RAISE (Authentic RAW)
- DiffusionForensics (AAAI 2024) - Celeb-DF v2 (CVPR 2020) - CASIA v2.0 (Splicing)
```
---
### 2.1 Full-Frame Multi-Generator Benchmarks
#### Dataset 1: GenImage (NeurIPS 2023)
- **Primary Focus:** Broad-spectrum cross-generator evaluation across diverse general-scene categories.
- **Reference:** Yang et al., *"GenImage: A Million-Scale Dataset for Detecting AI-Generated Image"*, NeurIPS 2023.
- **Official Repository:** `https://github.com/GenImage-Dataset/GenImage`
- **License:** Research / Academic Non-Commercial Use (subject to ImageNet Terms).
- **Data Modality:** Full general scenes ($256 \times 256$ to $512 \times 512$ RGB images).
- **Real Images:** Subsampled from ImageNet (ILSVRC 2012 validation/train sets).
- **Generators Represented (8 Generators):**
1. *Stable Diffusion v1.4* (CompVis)
2. *Stable Diffusion v1.5* (RunwayML)
3. *Midjourney* (Commercial closed-source text-to-image)
4. *ADM* (Ablated Diffusion Model)
5. *Glide* (OpenAI text-guided diffusion)
6. *Wukong* (Multilingual text-to-image diffusion)
7. *VQDM* (Vector Quantized Diffusion Model)
8. *BigGAN* (Large-scale GAN)
- **Strategic Role in Evaluation:** Tests whether UnivFD and ConvNeXt outperform ViT on general-scene diffusion images.
#### Dataset 2: DiffusionForensics (AAAI 2024 / DIRE)
- **Primary Focus:** Latent diffusion and pixel diffusion artifacts.
- **Reference:** Ma et al., *"DiffusionForensics: Deepfake Detection for Diffusion Models"*, AAAI 2024.
- **Official Repository:** `https://github.com/VCL-Lecun/DiffusionForensics`
- **License:** MIT License / Open Academic Research.
- **Data Modality:** General scenes and portraits ($256 \times 256$ and $512 \times 512$).
- **Generators Represented:**
- LDM (Latent Diffusion Models)
- Stable Diffusion v2.1
- DALL-E 2
- Guided Diffusion
- Midjourney v5
- **Strategic Role in Evaluation:** Probes high-frequency reconstruction errors and residual artifacts.
---
### 2.2 Advanced Facial Deepfake & Video Frame Benchmarks
#### Dataset 3: FaceForensics++ (FF++) (ICCV 2019)
- **Primary Focus:** Facial reenactment and swapping across standardized compression levels.
- **Reference:** Rössler et al., *"FaceForensics++: Learning to Detect Manipulated Facial Images"*, ICCV 2019.
- **Official Repository:** `https://github.com/ondyari/FaceForensics`
- **License:** Academic Use Agreement (Non-Commercial).
- **Data Modality:** High-resolution facial video frames (extracted at pristine and compressed rates).
- **Manipulation Algorithms:**
1. *Deepfakes* (Autoencoder-based facial replacement)
2. *Face2Face* (Real-time facial reenactment)
3. *FaceSwap* (Graphics-based 3D landmark swapping)
4. *NeuralTextures* (Photometric reenactment)
5. *FaceShifter* (Two-stage facial swap with occlusions)
- **Strategic Role in Evaluation:** Evaluates robustness under video compression (c0 raw, c23 high-quality, c40 low-quality).
#### Dataset 4: Celeb-DF (v2) (CVPR 2020)
- **Primary Focus:** Next-generation high-visual-quality deepfake face swapping.
- **Reference:** Li et al., *"Celeb-DF: A Large-scale Challenging Dataset for Deepfake Detection"*, CVPR 2020.
- **Official Repository:** `https://github.com/yuezunli/celeb-deepfakeforensic`
- **License:** Non-Commercial Research License.
- **Data Modality:** 5,639 high-definition YouTube celebrity interview frames.
- **Strategic Role in Evaluation:** Mitigates common low-resolution boundary artifacts present in older benchmarks; tests detector performance against advanced boundary blending.
---
### 2.3 Local Tampering, Splicing & Pristine Uncompressed Baselines
#### Dataset 5: RAISE (Raw Images for Authenticity Evaluation)
- **Primary Focus:** Pristine, camera-sensor native ground truth for false-positive validation.
- **Reference:** Pasquini et al., *"RAISE: A Raw Images Dataset for Digital Image Forensics"*, ACM MMSys 2015.
- **Official Repository:** `http://raise.diei.unipv.it/`
- **License:** Creative Commons Attribution-NonCommercial (CC BY-NC 4.0).
- **Data Modality:** 8,156 uncompressed camera-native TIFF/DNG images with complete EXIF sensor parameters.
- **Strategic Role in Evaluation:** Serves as the ultimate negative (REAL) calibration control to establish authentic false alarm rates when compression artifacts are completely absent.
#### Dataset 6: CASIA Image Tampering Database (v2.0)
- **Primary Focus:** Splicing, copy-move, and local image composition.
- **Reference:** Dong et al., CASIA TIDE v2.0, Institute of Automation, Chinese Academy of Sciences.
- **Official Repository:** `https://github.com/namtpham/casia2groundtruth`
- **License:** Academic Research License.
- **Data Modality:** 12,614 natural scene images with complex cut-and-paste manipulation boundaries.
- **Strategic Role in Evaluation:** Evaluates whether detectors can identify localized tampered regions embedded within authentic natural backgrounds.
---
## 3. Sampling Strategy & Manifest Integration
To respect host hardware constraints (8 GB RAM, CPU inference) while ensuring statistically rigorous representation:
### 3.1 Stratified Subsample Architecture
For each external dataset, a frozen evaluation split of **100 REAL and 100 FAKE** images will be sampled:
- **GenImage:** 50 SD-v1.5, 50 Midjourney vs. 100 ImageNet Real.
- **DiffusionForensics:** 50 LDM, 50 DALL-E 2 vs. 100 LSUN Real.
- **FaceForensics++:** 25 Deepfakes, 25 Face2Face, 25 FaceSwap, 25 NeuralTextures vs. 100 Authentic YouTube frames (c23 compression).
- **Celeb-DF:** 100 Celeb-DF swap frames vs. 100 Celeb-Real frames.
- **RAISE:** 100 pristine sensor captures (all REAL negative controls).
- **CASIA v2.0:** 100 local spliced/tampered scenes vs. 100 authentic scenes.
**Total Subsampled Evaluation Corpus:** 1,200 verified images across 6 datasets.
### 3.2 Automated Integrity Pipeline
Prior to evaluation, all external downloads must pass through:
1. `Image.open().verify()` and RGB mode conversion.
2. Canonical resizing to standardized aspect ratios.
3. Cryptographic SHA-256 fingerprinting.
4. Schema validation into `benchmark/external_generalization_manifest.csv`.
---
## 4. Hypothesis & Multi-Model Routing Strategy
Based on the empirical findings of Phase 6, we establish the following testable hypotheses:
| Dataset Modality | Predicted Strongest Model | Predicted Weakest Model | Proposed Ensemble Strategy |
| :--- | :--- | :--- | :--- |
| **Facial GANs (FFHQ/StyleGAN)** | ViT (`dima806`) | ConvNeXt / UnivFD | **ViT-Priority Routing** |
| **General-Scene Diffusion (GenImage)** | UnivFD / ConvNeXt | ViT (`dima806`) | **CLIP/ConvNeXt Consensus** |
| **Video Deepfakes (FaceForensics++)** | ViT / ConvNeXt | UnivFD | **Confidence-Weighted Softmax** |
| **Local Splicing (CASIA v2.0)** | ConvNeXt (DRCT) | ViT (`dima806`) | **Local Artifact Detection** |
| **Pristine RAW (RAISE Controls)** | All (Target: Spec > 98%) | None | **Conservative Real Thresholding** |
### Next Step: Domain-Aware Dynamic Router
Instead of static voting, the multi-model engine will implement a lightweight **Domain-Aware Dynamic Router**:
1. An input image is scanned for face presence (via MTCNN / RetinaFace or lightweight landmark check).
2. If **Facial Portrait:** Route to **ViT-Priority** (preserving 98% accuracy).
3. If **General Scene / Landscape:** Route to **UnivFD / ConvNeXt** consensus.
4. If **Ambiguous / Borderline:** Query full ensemble with uncertainty thresholding.
|