Spaces:
Running on Zero
Download docs/PHASE6_GENERALIZATION_PLAN.md from Thiru1122/Multimodel_setup: direct link, hf CLI and curl.
- Browser
- Download file 9.68 kB
-
https://huggingface.co/spaces/Thiru1122/Multimodel_setup/resolve/main/docs/PHASE6_GENERALIZATION_PLAN.md
- Command line
-
hf download hf://spaces/Thiru1122/Multimodel_setup/docs/PHASE6_GENERALIZATION_PLAN.md
-
curl -L -o PHASE6_GENERALIZATION_PLAN.md https://huggingface.co/spaces/Thiru1122/Multimodel_setup/resolve/main/docs/PHASE6_GENERALIZATION_PLAN.md
A newer version of the Gradio SDK is available: 6.32.0
Phase 6 Generalization Plan: Independent Cross-Generator & Multi-Domain Evaluation
Project: Cross-Model Evaluation of Deepfake Image Detection Under Diverse Image Generation and Manipulation Conditions
Date: November 2026
Status: Approved Research & Execution Plan
Target: Generalization beyond Facial GANs to Multi-Generator Scenes, Diffusion Models, and Local Manipulations
1. Context & Motivation
Phase 4, Phase 5, and Phase 6 empirically confirmed the Domain-Specialization Dilemma in deepfake detection:
- ViT (
dima806) dominates facial GAN detection with 98.00% accuracy on FFHQ vs. StyleGAN, but completely failed on diagnostic general scenes generated by modern diffusion systems (e.g., Google Imagen 3) and cut-and-paste manipulations. - UnivFD (
CLIP ViT-L/14) and ConvNeXt (DRCT / RAID) performed poorly on facial StyleGAN benchmarks (~51% accuracy), yet were designed specifically for cross-generator artifacts (ProGAN, LDM, Diffusion). - Naive fusion (majority voting and mean probability) degrades facial accuracy by -35.00 percentage points due to correlated false-negative biases.
To build an authentic, generalizable detection architecture, the evaluation benchmark must expand beyond faces to encompass:
- Diffusion Models (Stable Diffusion, Midjourney, DALL-E, Imagen).
- Diverse Domains (Landscapes, Urban Scenes, Objects, Textures).
- Local Manipulations (Face-swapping, Inpainting, Splicing, Copy-Move).
- Pruned Uncompressed Baselines (Camera-native RAW imagery).
2. Target External Datasets & Benchmark Specifications
The evaluation suite will integrate 6 standardized external benchmarks across three functional axes:
[Cross-Generator Benchmarks]
|
+--------------------------------------+--------------------------------------+
| | |
[Full-Frame Diffusion & GANs] [Facial Deepfake Video/Frames] [Local Splicing & RAW]
- GenImage (NeurIPS 2023) - FaceForensics++ (ICCV 2019) - RAISE (Authentic RAW)
- DiffusionForensics (AAAI 2024) - Celeb-DF v2 (CVPR 2020) - CASIA v2.0 (Splicing)
2.1 Full-Frame Multi-Generator Benchmarks
Dataset 1: GenImage (NeurIPS 2023)
- Primary Focus: Broad-spectrum cross-generator evaluation across diverse general-scene categories.
- Reference: Yang et al., "GenImage: A Million-Scale Dataset for Detecting AI-Generated Image", NeurIPS 2023.
- Official Repository:
https://github.com/GenImage-Dataset/GenImage - License: Research / Academic Non-Commercial Use (subject to ImageNet Terms).
- Data Modality: Full general scenes ($256 \times 256$ to $512 \times 512$ RGB images).
- Real Images: Subsampled from ImageNet (ILSVRC 2012 validation/train sets).
- Generators Represented (8 Generators):
- Stable Diffusion v1.4 (CompVis)
- Stable Diffusion v1.5 (RunwayML)
- Midjourney (Commercial closed-source text-to-image)
- ADM (Ablated Diffusion Model)
- Glide (OpenAI text-guided diffusion)
- Wukong (Multilingual text-to-image diffusion)
- VQDM (Vector Quantized Diffusion Model)
- BigGAN (Large-scale GAN)
- Strategic Role in Evaluation: Tests whether UnivFD and ConvNeXt outperform ViT on general-scene diffusion images.
Dataset 2: DiffusionForensics (AAAI 2024 / DIRE)
- Primary Focus: Latent diffusion and pixel diffusion artifacts.
- Reference: Ma et al., "DiffusionForensics: Deepfake Detection for Diffusion Models", AAAI 2024.
- Official Repository:
https://github.com/VCL-Lecun/DiffusionForensics - License: MIT License / Open Academic Research.
- Data Modality: General scenes and portraits ($256 \times 256$ and $512 \times 512$).
- Generators Represented:
- LDM (Latent Diffusion Models)
- Stable Diffusion v2.1
- DALL-E 2
- Guided Diffusion
- Midjourney v5
- Strategic Role in Evaluation: Probes high-frequency reconstruction errors and residual artifacts.
2.2 Advanced Facial Deepfake & Video Frame Benchmarks
Dataset 3: FaceForensics++ (FF++) (ICCV 2019)
- Primary Focus: Facial reenactment and swapping across standardized compression levels.
- Reference: Rössler et al., "FaceForensics++: Learning to Detect Manipulated Facial Images", ICCV 2019.
- Official Repository:
https://github.com/ondyari/FaceForensics - License: Academic Use Agreement (Non-Commercial).
- Data Modality: High-resolution facial video frames (extracted at pristine and compressed rates).
- Manipulation Algorithms:
- Deepfakes (Autoencoder-based facial replacement)
- Face2Face (Real-time facial reenactment)
- FaceSwap (Graphics-based 3D landmark swapping)
- NeuralTextures (Photometric reenactment)
- FaceShifter (Two-stage facial swap with occlusions)
- Strategic Role in Evaluation: Evaluates robustness under video compression (c0 raw, c23 high-quality, c40 low-quality).
Dataset 4: Celeb-DF (v2) (CVPR 2020)
- Primary Focus: Next-generation high-visual-quality deepfake face swapping.
- Reference: Li et al., "Celeb-DF: A Large-scale Challenging Dataset for Deepfake Detection", CVPR 2020.
- Official Repository:
https://github.com/yuezunli/celeb-deepfakeforensic - License: Non-Commercial Research License.
- Data Modality: 5,639 high-definition YouTube celebrity interview frames.
- Strategic Role in Evaluation: Mitigates common low-resolution boundary artifacts present in older benchmarks; tests detector performance against advanced boundary blending.
2.3 Local Tampering, Splicing & Pristine Uncompressed Baselines
Dataset 5: RAISE (Raw Images for Authenticity Evaluation)
- Primary Focus: Pristine, camera-sensor native ground truth for false-positive validation.
- Reference: Pasquini et al., "RAISE: A Raw Images Dataset for Digital Image Forensics", ACM MMSys 2015.
- Official Repository:
http://raise.diei.unipv.it/ - License: Creative Commons Attribution-NonCommercial (CC BY-NC 4.0).
- Data Modality: 8,156 uncompressed camera-native TIFF/DNG images with complete EXIF sensor parameters.
- Strategic Role in Evaluation: Serves as the ultimate negative (REAL) calibration control to establish authentic false alarm rates when compression artifacts are completely absent.
Dataset 6: CASIA Image Tampering Database (v2.0)
- Primary Focus: Splicing, copy-move, and local image composition.
- Reference: Dong et al., CASIA TIDE v2.0, Institute of Automation, Chinese Academy of Sciences.
- Official Repository:
https://github.com/namtpham/casia2groundtruth - License: Academic Research License.
- Data Modality: 12,614 natural scene images with complex cut-and-paste manipulation boundaries.
- Strategic Role in Evaluation: Evaluates whether detectors can identify localized tampered regions embedded within authentic natural backgrounds.
3. Sampling Strategy & Manifest Integration
To respect host hardware constraints (8 GB RAM, CPU inference) while ensuring statistically rigorous representation:
3.1 Stratified Subsample Architecture
For each external dataset, a frozen evaluation split of 100 REAL and 100 FAKE images will be sampled:
- GenImage: 50 SD-v1.5, 50 Midjourney vs. 100 ImageNet Real.
- DiffusionForensics: 50 LDM, 50 DALL-E 2 vs. 100 LSUN Real.
- FaceForensics++: 25 Deepfakes, 25 Face2Face, 25 FaceSwap, 25 NeuralTextures vs. 100 Authentic YouTube frames (c23 compression).
- Celeb-DF: 100 Celeb-DF swap frames vs. 100 Celeb-Real frames.
- RAISE: 100 pristine sensor captures (all REAL negative controls).
- CASIA v2.0: 100 local spliced/tampered scenes vs. 100 authentic scenes.
Total Subsampled Evaluation Corpus: 1,200 verified images across 6 datasets.
3.2 Automated Integrity Pipeline
Prior to evaluation, all external downloads must pass through:
Image.open().verify()and RGB mode conversion.- Canonical resizing to standardized aspect ratios.
- Cryptographic SHA-256 fingerprinting.
- Schema validation into
benchmark/external_generalization_manifest.csv.
4. Hypothesis & Multi-Model Routing Strategy
Based on the empirical findings of Phase 6, we establish the following testable hypotheses:
| Dataset Modality | Predicted Strongest Model | Predicted Weakest Model | Proposed Ensemble Strategy |
|---|---|---|---|
| Facial GANs (FFHQ/StyleGAN) | ViT (dima806) |
ConvNeXt / UnivFD | ViT-Priority Routing |
| General-Scene Diffusion (GenImage) | UnivFD / ConvNeXt | ViT (dima806) |
CLIP/ConvNeXt Consensus |
| Video Deepfakes (FaceForensics++) | ViT / ConvNeXt | UnivFD | Confidence-Weighted Softmax |
| Local Splicing (CASIA v2.0) | ConvNeXt (DRCT) | ViT (dima806) |
Local Artifact Detection |
| Pristine RAW (RAISE Controls) | All (Target: Spec > 98%) | None | Conservative Real Thresholding |
Next Step: Domain-Aware Dynamic Router
Instead of static voting, the multi-model engine will implement a lightweight Domain-Aware Dynamic Router:
- An input image is scanned for face presence (via MTCNN / RetinaFace or lightweight landmark check).
- If Facial Portrait: Route to ViT-Priority (preserving 98% accuracy).
- If General Scene / Landscape: Route to UnivFD / ConvNeXt consensus.
- If Ambiguous / Borderline: Query full ensemble with uncertainty thresholding.