File size: 9,679 Bytes
151f388
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
# Phase 6 Generalization Plan: Independent Cross-Generator & Multi-Domain Evaluation

**Project:** Cross-Model Evaluation of Deepfake Image Detection Under Diverse Image Generation and Manipulation Conditions  
**Date:** November 2026  
**Status:** Approved Research & Execution Plan  
**Target:** Generalization beyond Facial GANs to Multi-Generator Scenes, Diffusion Models, and Local Manipulations

---

## 1. Context & Motivation

Phase 4, Phase 5, and Phase 6 empirically confirmed the **Domain-Specialization Dilemma** in deepfake detection:
1. **ViT (`dima806`)** dominates facial GAN detection with **98.00% accuracy** on FFHQ vs. StyleGAN, but completely failed on diagnostic general scenes generated by modern diffusion systems (e.g., Google Imagen 3) and cut-and-paste manipulations.
2. **UnivFD (`CLIP ViT-L/14`)** and **ConvNeXt (`DRCT / RAID`)** performed poorly on facial StyleGAN benchmarks (~51% accuracy), yet were designed specifically for cross-generator artifacts (ProGAN, LDM, Diffusion).
3. Naive fusion (majority voting and mean probability) degrades facial accuracy by **-35.00 percentage points** due to correlated false-negative biases.

To build an authentic, generalizable detection architecture, the evaluation benchmark must expand beyond faces to encompass:
- **Diffusion Models** (Stable Diffusion, Midjourney, DALL-E, Imagen).
- **Diverse Domains** (Landscapes, Urban Scenes, Objects, Textures).
- **Local Manipulations** (Face-swapping, Inpainting, Splicing, Copy-Move).
- **Pruned Uncompressed Baselines** (Camera-native RAW imagery).

---

## 2. Target External Datasets & Benchmark Specifications

The evaluation suite will integrate 6 standardized external benchmarks across three functional axes:

```
                                  [Cross-Generator Benchmarks]
                                                |
         +--------------------------------------+--------------------------------------+
         |                                      |                                      |
   [Full-Frame Diffusion & GANs]        [Facial Deepfake Video/Frames]        [Local Splicing & RAW]
     - GenImage (NeurIPS 2023)             - FaceForensics++ (ICCV 2019)         - RAISE (Authentic RAW)
     - DiffusionForensics (AAAI 2024)      - Celeb-DF v2 (CVPR 2020)             - CASIA v2.0 (Splicing)
```

---

### 2.1 Full-Frame Multi-Generator Benchmarks

#### Dataset 1: GenImage (NeurIPS 2023)
- **Primary Focus:** Broad-spectrum cross-generator evaluation across diverse general-scene categories.
- **Reference:** Yang et al., *"GenImage: A Million-Scale Dataset for Detecting AI-Generated Image"*, NeurIPS 2023.
- **Official Repository:** `https://github.com/GenImage-Dataset/GenImage`
- **License:** Research / Academic Non-Commercial Use (subject to ImageNet Terms).
- **Data Modality:** Full general scenes ($256 \times 256$ to $512 \times 512$ RGB images).
- **Real Images:** Subsampled from ImageNet (ILSVRC 2012 validation/train sets).
- **Generators Represented (8 Generators):**
  1. *Stable Diffusion v1.4* (CompVis)
  2. *Stable Diffusion v1.5* (RunwayML)
  3. *Midjourney* (Commercial closed-source text-to-image)
  4. *ADM* (Ablated Diffusion Model)
  5. *Glide* (OpenAI text-guided diffusion)
  6. *Wukong* (Multilingual text-to-image diffusion)
  7. *VQDM* (Vector Quantized Diffusion Model)
  8. *BigGAN* (Large-scale GAN)
- **Strategic Role in Evaluation:** Tests whether UnivFD and ConvNeXt outperform ViT on general-scene diffusion images.

#### Dataset 2: DiffusionForensics (AAAI 2024 / DIRE)
- **Primary Focus:** Latent diffusion and pixel diffusion artifacts.
- **Reference:** Ma et al., *"DiffusionForensics: Deepfake Detection for Diffusion Models"*, AAAI 2024.
- **Official Repository:** `https://github.com/VCL-Lecun/DiffusionForensics`
- **License:** MIT License / Open Academic Research.
- **Data Modality:** General scenes and portraits ($256 \times 256$ and $512 \times 512$).
- **Generators Represented:**
  - LDM (Latent Diffusion Models)
  - Stable Diffusion v2.1
  - DALL-E 2
  - Guided Diffusion
  - Midjourney v5
- **Strategic Role in Evaluation:** Probes high-frequency reconstruction errors and residual artifacts.

---

### 2.2 Advanced Facial Deepfake & Video Frame Benchmarks

#### Dataset 3: FaceForensics++ (FF++) (ICCV 2019)
- **Primary Focus:** Facial reenactment and swapping across standardized compression levels.
- **Reference:** Rössler et al., *"FaceForensics++: Learning to Detect Manipulated Facial Images"*, ICCV 2019.
- **Official Repository:** `https://github.com/ondyari/FaceForensics`
- **License:** Academic Use Agreement (Non-Commercial).
- **Data Modality:** High-resolution facial video frames (extracted at pristine and compressed rates).
- **Manipulation Algorithms:**
  1. *Deepfakes* (Autoencoder-based facial replacement)
  2. *Face2Face* (Real-time facial reenactment)
  3. *FaceSwap* (Graphics-based 3D landmark swapping)
  4. *NeuralTextures* (Photometric reenactment)
  5. *FaceShifter* (Two-stage facial swap with occlusions)
- **Strategic Role in Evaluation:** Evaluates robustness under video compression (c0 raw, c23 high-quality, c40 low-quality).

#### Dataset 4: Celeb-DF (v2) (CVPR 2020)
- **Primary Focus:** Next-generation high-visual-quality deepfake face swapping.
- **Reference:** Li et al., *"Celeb-DF: A Large-scale Challenging Dataset for Deepfake Detection"*, CVPR 2020.
- **Official Repository:** `https://github.com/yuezunli/celeb-deepfakeforensic`
- **License:** Non-Commercial Research License.
- **Data Modality:** 5,639 high-definition YouTube celebrity interview frames.
- **Strategic Role in Evaluation:** Mitigates common low-resolution boundary artifacts present in older benchmarks; tests detector performance against advanced boundary blending.

---

### 2.3 Local Tampering, Splicing & Pristine Uncompressed Baselines

#### Dataset 5: RAISE (Raw Images for Authenticity Evaluation)
- **Primary Focus:** Pristine, camera-sensor native ground truth for false-positive validation.
- **Reference:** Pasquini et al., *"RAISE: A Raw Images Dataset for Digital Image Forensics"*, ACM MMSys 2015.
- **Official Repository:** `http://raise.diei.unipv.it/`
- **License:** Creative Commons Attribution-NonCommercial (CC BY-NC 4.0).
- **Data Modality:** 8,156 uncompressed camera-native TIFF/DNG images with complete EXIF sensor parameters.
- **Strategic Role in Evaluation:** Serves as the ultimate negative (REAL) calibration control to establish authentic false alarm rates when compression artifacts are completely absent.

#### Dataset 6: CASIA Image Tampering Database (v2.0)
- **Primary Focus:** Splicing, copy-move, and local image composition.
- **Reference:** Dong et al., CASIA TIDE v2.0, Institute of Automation, Chinese Academy of Sciences.
- **Official Repository:** `https://github.com/namtpham/casia2groundtruth`
- **License:** Academic Research License.
- **Data Modality:** 12,614 natural scene images with complex cut-and-paste manipulation boundaries.
- **Strategic Role in Evaluation:** Evaluates whether detectors can identify localized tampered regions embedded within authentic natural backgrounds.

---

## 3. Sampling Strategy & Manifest Integration

To respect host hardware constraints (8 GB RAM, CPU inference) while ensuring statistically rigorous representation:

### 3.1 Stratified Subsample Architecture
For each external dataset, a frozen evaluation split of **100 REAL and 100 FAKE** images will be sampled:
- **GenImage:** 50 SD-v1.5, 50 Midjourney vs. 100 ImageNet Real.
- **DiffusionForensics:** 50 LDM, 50 DALL-E 2 vs. 100 LSUN Real.
- **FaceForensics++:** 25 Deepfakes, 25 Face2Face, 25 FaceSwap, 25 NeuralTextures vs. 100 Authentic YouTube frames (c23 compression).
- **Celeb-DF:** 100 Celeb-DF swap frames vs. 100 Celeb-Real frames.
- **RAISE:** 100 pristine sensor captures (all REAL negative controls).
- **CASIA v2.0:** 100 local spliced/tampered scenes vs. 100 authentic scenes.

**Total Subsampled Evaluation Corpus:** 1,200 verified images across 6 datasets.

### 3.2 Automated Integrity Pipeline
Prior to evaluation, all external downloads must pass through:
1. `Image.open().verify()` and RGB mode conversion.
2. Canonical resizing to standardized aspect ratios.
3. Cryptographic SHA-256 fingerprinting.
4. Schema validation into `benchmark/external_generalization_manifest.csv`.

---

## 4. Hypothesis & Multi-Model Routing Strategy

Based on the empirical findings of Phase 6, we establish the following testable hypotheses:

| Dataset Modality | Predicted Strongest Model | Predicted Weakest Model | Proposed Ensemble Strategy |
| :--- | :--- | :--- | :--- |
| **Facial GANs (FFHQ/StyleGAN)** | ViT (`dima806`) | ConvNeXt / UnivFD | **ViT-Priority Routing** |
| **General-Scene Diffusion (GenImage)** | UnivFD / ConvNeXt | ViT (`dima806`) | **CLIP/ConvNeXt Consensus** |
| **Video Deepfakes (FaceForensics++)** | ViT / ConvNeXt | UnivFD | **Confidence-Weighted Softmax** |
| **Local Splicing (CASIA v2.0)** | ConvNeXt (DRCT) | ViT (`dima806`) | **Local Artifact Detection** |
| **Pristine RAW (RAISE Controls)** | All (Target: Spec > 98%) | None | **Conservative Real Thresholding** |

### Next Step: Domain-Aware Dynamic Router
Instead of static voting, the multi-model engine will implement a lightweight **Domain-Aware Dynamic Router**:
1. An input image is scanned for face presence (via MTCNN / RetinaFace or lightweight landmark check).
2. If **Facial Portrait:** Route to **ViT-Priority** (preserving 98% accuracy).
3. If **General Scene / Landscape:** Route to **UnivFD / ConvNeXt** consensus.
4. If **Ambiguous / Borderline:** Query full ensemble with uncertainty thresholding.