# Phase 6 Generalization Plan: Independent Cross-Generator & Multi-Domain Evaluation **Project:** Cross-Model Evaluation of Deepfake Image Detection Under Diverse Image Generation and Manipulation Conditions **Date:** November 2026 **Status:** Approved Research & Execution Plan **Target:** Generalization beyond Facial GANs to Multi-Generator Scenes, Diffusion Models, and Local Manipulations --- ## 1. Context & Motivation Phase 4, Phase 5, and Phase 6 empirically confirmed the **Domain-Specialization Dilemma** in deepfake detection: 1. **ViT (`dima806`)** dominates facial GAN detection with **98.00% accuracy** on FFHQ vs. StyleGAN, but completely failed on diagnostic general scenes generated by modern diffusion systems (e.g., Google Imagen 3) and cut-and-paste manipulations. 2. **UnivFD (`CLIP ViT-L/14`)** and **ConvNeXt (`DRCT / RAID`)** performed poorly on facial StyleGAN benchmarks (~51% accuracy), yet were designed specifically for cross-generator artifacts (ProGAN, LDM, Diffusion). 3. Naive fusion (majority voting and mean probability) degrades facial accuracy by **-35.00 percentage points** due to correlated false-negative biases. To build an authentic, generalizable detection architecture, the evaluation benchmark must expand beyond faces to encompass: - **Diffusion Models** (Stable Diffusion, Midjourney, DALL-E, Imagen). - **Diverse Domains** (Landscapes, Urban Scenes, Objects, Textures). - **Local Manipulations** (Face-swapping, Inpainting, Splicing, Copy-Move). - **Pruned Uncompressed Baselines** (Camera-native RAW imagery). --- ## 2. Target External Datasets & Benchmark Specifications The evaluation suite will integrate 6 standardized external benchmarks across three functional axes: ``` [Cross-Generator Benchmarks] | +--------------------------------------+--------------------------------------+ | | | [Full-Frame Diffusion & GANs] [Facial Deepfake Video/Frames] [Local Splicing & RAW] - GenImage (NeurIPS 2023) - FaceForensics++ (ICCV 2019) - RAISE (Authentic RAW) - DiffusionForensics (AAAI 2024) - Celeb-DF v2 (CVPR 2020) - CASIA v2.0 (Splicing) ``` --- ### 2.1 Full-Frame Multi-Generator Benchmarks #### Dataset 1: GenImage (NeurIPS 2023) - **Primary Focus:** Broad-spectrum cross-generator evaluation across diverse general-scene categories. - **Reference:** Yang et al., *"GenImage: A Million-Scale Dataset for Detecting AI-Generated Image"*, NeurIPS 2023. - **Official Repository:** `https://github.com/GenImage-Dataset/GenImage` - **License:** Research / Academic Non-Commercial Use (subject to ImageNet Terms). - **Data Modality:** Full general scenes ($256 \times 256$ to $512 \times 512$ RGB images). - **Real Images:** Subsampled from ImageNet (ILSVRC 2012 validation/train sets). - **Generators Represented (8 Generators):** 1. *Stable Diffusion v1.4* (CompVis) 2. *Stable Diffusion v1.5* (RunwayML) 3. *Midjourney* (Commercial closed-source text-to-image) 4. *ADM* (Ablated Diffusion Model) 5. *Glide* (OpenAI text-guided diffusion) 6. *Wukong* (Multilingual text-to-image diffusion) 7. *VQDM* (Vector Quantized Diffusion Model) 8. *BigGAN* (Large-scale GAN) - **Strategic Role in Evaluation:** Tests whether UnivFD and ConvNeXt outperform ViT on general-scene diffusion images. #### Dataset 2: DiffusionForensics (AAAI 2024 / DIRE) - **Primary Focus:** Latent diffusion and pixel diffusion artifacts. - **Reference:** Ma et al., *"DiffusionForensics: Deepfake Detection for Diffusion Models"*, AAAI 2024. - **Official Repository:** `https://github.com/VCL-Lecun/DiffusionForensics` - **License:** MIT License / Open Academic Research. - **Data Modality:** General scenes and portraits ($256 \times 256$ and $512 \times 512$). - **Generators Represented:** - LDM (Latent Diffusion Models) - Stable Diffusion v2.1 - DALL-E 2 - Guided Diffusion - Midjourney v5 - **Strategic Role in Evaluation:** Probes high-frequency reconstruction errors and residual artifacts. --- ### 2.2 Advanced Facial Deepfake & Video Frame Benchmarks #### Dataset 3: FaceForensics++ (FF++) (ICCV 2019) - **Primary Focus:** Facial reenactment and swapping across standardized compression levels. - **Reference:** Rössler et al., *"FaceForensics++: Learning to Detect Manipulated Facial Images"*, ICCV 2019. - **Official Repository:** `https://github.com/ondyari/FaceForensics` - **License:** Academic Use Agreement (Non-Commercial). - **Data Modality:** High-resolution facial video frames (extracted at pristine and compressed rates). - **Manipulation Algorithms:** 1. *Deepfakes* (Autoencoder-based facial replacement) 2. *Face2Face* (Real-time facial reenactment) 3. *FaceSwap* (Graphics-based 3D landmark swapping) 4. *NeuralTextures* (Photometric reenactment) 5. *FaceShifter* (Two-stage facial swap with occlusions) - **Strategic Role in Evaluation:** Evaluates robustness under video compression (c0 raw, c23 high-quality, c40 low-quality). #### Dataset 4: Celeb-DF (v2) (CVPR 2020) - **Primary Focus:** Next-generation high-visual-quality deepfake face swapping. - **Reference:** Li et al., *"Celeb-DF: A Large-scale Challenging Dataset for Deepfake Detection"*, CVPR 2020. - **Official Repository:** `https://github.com/yuezunli/celeb-deepfakeforensic` - **License:** Non-Commercial Research License. - **Data Modality:** 5,639 high-definition YouTube celebrity interview frames. - **Strategic Role in Evaluation:** Mitigates common low-resolution boundary artifacts present in older benchmarks; tests detector performance against advanced boundary blending. --- ### 2.3 Local Tampering, Splicing & Pristine Uncompressed Baselines #### Dataset 5: RAISE (Raw Images for Authenticity Evaluation) - **Primary Focus:** Pristine, camera-sensor native ground truth for false-positive validation. - **Reference:** Pasquini et al., *"RAISE: A Raw Images Dataset for Digital Image Forensics"*, ACM MMSys 2015. - **Official Repository:** `http://raise.diei.unipv.it/` - **License:** Creative Commons Attribution-NonCommercial (CC BY-NC 4.0). - **Data Modality:** 8,156 uncompressed camera-native TIFF/DNG images with complete EXIF sensor parameters. - **Strategic Role in Evaluation:** Serves as the ultimate negative (REAL) calibration control to establish authentic false alarm rates when compression artifacts are completely absent. #### Dataset 6: CASIA Image Tampering Database (v2.0) - **Primary Focus:** Splicing, copy-move, and local image composition. - **Reference:** Dong et al., CASIA TIDE v2.0, Institute of Automation, Chinese Academy of Sciences. - **Official Repository:** `https://github.com/namtpham/casia2groundtruth` - **License:** Academic Research License. - **Data Modality:** 12,614 natural scene images with complex cut-and-paste manipulation boundaries. - **Strategic Role in Evaluation:** Evaluates whether detectors can identify localized tampered regions embedded within authentic natural backgrounds. --- ## 3. Sampling Strategy & Manifest Integration To respect host hardware constraints (8 GB RAM, CPU inference) while ensuring statistically rigorous representation: ### 3.1 Stratified Subsample Architecture For each external dataset, a frozen evaluation split of **100 REAL and 100 FAKE** images will be sampled: - **GenImage:** 50 SD-v1.5, 50 Midjourney vs. 100 ImageNet Real. - **DiffusionForensics:** 50 LDM, 50 DALL-E 2 vs. 100 LSUN Real. - **FaceForensics++:** 25 Deepfakes, 25 Face2Face, 25 FaceSwap, 25 NeuralTextures vs. 100 Authentic YouTube frames (c23 compression). - **Celeb-DF:** 100 Celeb-DF swap frames vs. 100 Celeb-Real frames. - **RAISE:** 100 pristine sensor captures (all REAL negative controls). - **CASIA v2.0:** 100 local spliced/tampered scenes vs. 100 authentic scenes. **Total Subsampled Evaluation Corpus:** 1,200 verified images across 6 datasets. ### 3.2 Automated Integrity Pipeline Prior to evaluation, all external downloads must pass through: 1. `Image.open().verify()` and RGB mode conversion. 2. Canonical resizing to standardized aspect ratios. 3. Cryptographic SHA-256 fingerprinting. 4. Schema validation into `benchmark/external_generalization_manifest.csv`. --- ## 4. Hypothesis & Multi-Model Routing Strategy Based on the empirical findings of Phase 6, we establish the following testable hypotheses: | Dataset Modality | Predicted Strongest Model | Predicted Weakest Model | Proposed Ensemble Strategy | | :--- | :--- | :--- | :--- | | **Facial GANs (FFHQ/StyleGAN)** | ViT (`dima806`) | ConvNeXt / UnivFD | **ViT-Priority Routing** | | **General-Scene Diffusion (GenImage)** | UnivFD / ConvNeXt | ViT (`dima806`) | **CLIP/ConvNeXt Consensus** | | **Video Deepfakes (FaceForensics++)** | ViT / ConvNeXt | UnivFD | **Confidence-Weighted Softmax** | | **Local Splicing (CASIA v2.0)** | ConvNeXt (DRCT) | ViT (`dima806`) | **Local Artifact Detection** | | **Pristine RAW (RAISE Controls)** | All (Target: Spec > 98%) | None | **Conservative Real Thresholding** | ### Next Step: Domain-Aware Dynamic Router Instead of static voting, the multi-model engine will implement a lightweight **Domain-Aware Dynamic Router**: 1. An input image is scanned for face presence (via MTCNN / RetinaFace or lightweight landmark check). 2. If **Facial Portrait:** Route to **ViT-Priority** (preserving 98% accuracy). 3. If **General Scene / Landscape:** Route to **UnivFD / ConvNeXt** consensus. 4. If **Ambiguous / Borderline:** Query full ensemble with uncertainty thresholding.