Spaces:
Running on Zero
Running on Zero
|
Download docs/PHASE6_GENERALIZATION_PLAN.md from Thiru1122/Multimodel_setup: direct link, hf CLI and curl.
- Browser
- Download file 9.68 kB
-
https://huggingface.co/spaces/Thiru1122/Multimodel_setup/resolve/main/docs/PHASE6_GENERALIZATION_PLAN.md
- Command line
-
hf download hf://spaces/Thiru1122/Multimodel_setup/docs/PHASE6_GENERALIZATION_PLAN.md
-
curl -L -o PHASE6_GENERALIZATION_PLAN.md https://huggingface.co/spaces/Thiru1122/Multimodel_setup/resolve/main/docs/PHASE6_GENERALIZATION_PLAN.md
9.68 kB
| # Phase 6 Generalization Plan: Independent Cross-Generator & Multi-Domain Evaluation | |
| **Project:** Cross-Model Evaluation of Deepfake Image Detection Under Diverse Image Generation and Manipulation Conditions | |
| **Date:** November 2026 | |
| **Status:** Approved Research & Execution Plan | |
| **Target:** Generalization beyond Facial GANs to Multi-Generator Scenes, Diffusion Models, and Local Manipulations | |
| --- | |
| ## 1. Context & Motivation | |
| Phase 4, Phase 5, and Phase 6 empirically confirmed the **Domain-Specialization Dilemma** in deepfake detection: | |
| 1. **ViT (`dima806`)** dominates facial GAN detection with **98.00% accuracy** on FFHQ vs. StyleGAN, but completely failed on diagnostic general scenes generated by modern diffusion systems (e.g., Google Imagen 3) and cut-and-paste manipulations. | |
| 2. **UnivFD (`CLIP ViT-L/14`)** and **ConvNeXt (`DRCT / RAID`)** performed poorly on facial StyleGAN benchmarks (~51% accuracy), yet were designed specifically for cross-generator artifacts (ProGAN, LDM, Diffusion). | |
| 3. Naive fusion (majority voting and mean probability) degrades facial accuracy by **-35.00 percentage points** due to correlated false-negative biases. | |
| To build an authentic, generalizable detection architecture, the evaluation benchmark must expand beyond faces to encompass: | |
| - **Diffusion Models** (Stable Diffusion, Midjourney, DALL-E, Imagen). | |
| - **Diverse Domains** (Landscapes, Urban Scenes, Objects, Textures). | |
| - **Local Manipulations** (Face-swapping, Inpainting, Splicing, Copy-Move). | |
| - **Pruned Uncompressed Baselines** (Camera-native RAW imagery). | |
| --- | |
| ## 2. Target External Datasets & Benchmark Specifications | |
| The evaluation suite will integrate 6 standardized external benchmarks across three functional axes: | |
| ``` | |
| [Cross-Generator Benchmarks] | |
| | | |
| +--------------------------------------+--------------------------------------+ | |
| | | | | |
| [Full-Frame Diffusion & GANs] [Facial Deepfake Video/Frames] [Local Splicing & RAW] | |
| - GenImage (NeurIPS 2023) - FaceForensics++ (ICCV 2019) - RAISE (Authentic RAW) | |
| - DiffusionForensics (AAAI 2024) - Celeb-DF v2 (CVPR 2020) - CASIA v2.0 (Splicing) | |
| ``` | |
| --- | |
| ### 2.1 Full-Frame Multi-Generator Benchmarks | |
| #### Dataset 1: GenImage (NeurIPS 2023) | |
| - **Primary Focus:** Broad-spectrum cross-generator evaluation across diverse general-scene categories. | |
| - **Reference:** Yang et al., *"GenImage: A Million-Scale Dataset for Detecting AI-Generated Image"*, NeurIPS 2023. | |
| - **Official Repository:** `https://github.com/GenImage-Dataset/GenImage` | |
| - **License:** Research / Academic Non-Commercial Use (subject to ImageNet Terms). | |
| - **Data Modality:** Full general scenes ($256 \times 256$ to $512 \times 512$ RGB images). | |
| - **Real Images:** Subsampled from ImageNet (ILSVRC 2012 validation/train sets). | |
| - **Generators Represented (8 Generators):** | |
| 1. *Stable Diffusion v1.4* (CompVis) | |
| 2. *Stable Diffusion v1.5* (RunwayML) | |
| 3. *Midjourney* (Commercial closed-source text-to-image) | |
| 4. *ADM* (Ablated Diffusion Model) | |
| 5. *Glide* (OpenAI text-guided diffusion) | |
| 6. *Wukong* (Multilingual text-to-image diffusion) | |
| 7. *VQDM* (Vector Quantized Diffusion Model) | |
| 8. *BigGAN* (Large-scale GAN) | |
| - **Strategic Role in Evaluation:** Tests whether UnivFD and ConvNeXt outperform ViT on general-scene diffusion images. | |
| #### Dataset 2: DiffusionForensics (AAAI 2024 / DIRE) | |
| - **Primary Focus:** Latent diffusion and pixel diffusion artifacts. | |
| - **Reference:** Ma et al., *"DiffusionForensics: Deepfake Detection for Diffusion Models"*, AAAI 2024. | |
| - **Official Repository:** `https://github.com/VCL-Lecun/DiffusionForensics` | |
| - **License:** MIT License / Open Academic Research. | |
| - **Data Modality:** General scenes and portraits ($256 \times 256$ and $512 \times 512$). | |
| - **Generators Represented:** | |
| - LDM (Latent Diffusion Models) | |
| - Stable Diffusion v2.1 | |
| - DALL-E 2 | |
| - Guided Diffusion | |
| - Midjourney v5 | |
| - **Strategic Role in Evaluation:** Probes high-frequency reconstruction errors and residual artifacts. | |
| --- | |
| ### 2.2 Advanced Facial Deepfake & Video Frame Benchmarks | |
| #### Dataset 3: FaceForensics++ (FF++) (ICCV 2019) | |
| - **Primary Focus:** Facial reenactment and swapping across standardized compression levels. | |
| - **Reference:** Rössler et al., *"FaceForensics++: Learning to Detect Manipulated Facial Images"*, ICCV 2019. | |
| - **Official Repository:** `https://github.com/ondyari/FaceForensics` | |
| - **License:** Academic Use Agreement (Non-Commercial). | |
| - **Data Modality:** High-resolution facial video frames (extracted at pristine and compressed rates). | |
| - **Manipulation Algorithms:** | |
| 1. *Deepfakes* (Autoencoder-based facial replacement) | |
| 2. *Face2Face* (Real-time facial reenactment) | |
| 3. *FaceSwap* (Graphics-based 3D landmark swapping) | |
| 4. *NeuralTextures* (Photometric reenactment) | |
| 5. *FaceShifter* (Two-stage facial swap with occlusions) | |
| - **Strategic Role in Evaluation:** Evaluates robustness under video compression (c0 raw, c23 high-quality, c40 low-quality). | |
| #### Dataset 4: Celeb-DF (v2) (CVPR 2020) | |
| - **Primary Focus:** Next-generation high-visual-quality deepfake face swapping. | |
| - **Reference:** Li et al., *"Celeb-DF: A Large-scale Challenging Dataset for Deepfake Detection"*, CVPR 2020. | |
| - **Official Repository:** `https://github.com/yuezunli/celeb-deepfakeforensic` | |
| - **License:** Non-Commercial Research License. | |
| - **Data Modality:** 5,639 high-definition YouTube celebrity interview frames. | |
| - **Strategic Role in Evaluation:** Mitigates common low-resolution boundary artifacts present in older benchmarks; tests detector performance against advanced boundary blending. | |
| --- | |
| ### 2.3 Local Tampering, Splicing & Pristine Uncompressed Baselines | |
| #### Dataset 5: RAISE (Raw Images for Authenticity Evaluation) | |
| - **Primary Focus:** Pristine, camera-sensor native ground truth for false-positive validation. | |
| - **Reference:** Pasquini et al., *"RAISE: A Raw Images Dataset for Digital Image Forensics"*, ACM MMSys 2015. | |
| - **Official Repository:** `http://raise.diei.unipv.it/` | |
| - **License:** Creative Commons Attribution-NonCommercial (CC BY-NC 4.0). | |
| - **Data Modality:** 8,156 uncompressed camera-native TIFF/DNG images with complete EXIF sensor parameters. | |
| - **Strategic Role in Evaluation:** Serves as the ultimate negative (REAL) calibration control to establish authentic false alarm rates when compression artifacts are completely absent. | |
| #### Dataset 6: CASIA Image Tampering Database (v2.0) | |
| - **Primary Focus:** Splicing, copy-move, and local image composition. | |
| - **Reference:** Dong et al., CASIA TIDE v2.0, Institute of Automation, Chinese Academy of Sciences. | |
| - **Official Repository:** `https://github.com/namtpham/casia2groundtruth` | |
| - **License:** Academic Research License. | |
| - **Data Modality:** 12,614 natural scene images with complex cut-and-paste manipulation boundaries. | |
| - **Strategic Role in Evaluation:** Evaluates whether detectors can identify localized tampered regions embedded within authentic natural backgrounds. | |
| --- | |
| ## 3. Sampling Strategy & Manifest Integration | |
| To respect host hardware constraints (8 GB RAM, CPU inference) while ensuring statistically rigorous representation: | |
| ### 3.1 Stratified Subsample Architecture | |
| For each external dataset, a frozen evaluation split of **100 REAL and 100 FAKE** images will be sampled: | |
| - **GenImage:** 50 SD-v1.5, 50 Midjourney vs. 100 ImageNet Real. | |
| - **DiffusionForensics:** 50 LDM, 50 DALL-E 2 vs. 100 LSUN Real. | |
| - **FaceForensics++:** 25 Deepfakes, 25 Face2Face, 25 FaceSwap, 25 NeuralTextures vs. 100 Authentic YouTube frames (c23 compression). | |
| - **Celeb-DF:** 100 Celeb-DF swap frames vs. 100 Celeb-Real frames. | |
| - **RAISE:** 100 pristine sensor captures (all REAL negative controls). | |
| - **CASIA v2.0:** 100 local spliced/tampered scenes vs. 100 authentic scenes. | |
| **Total Subsampled Evaluation Corpus:** 1,200 verified images across 6 datasets. | |
| ### 3.2 Automated Integrity Pipeline | |
| Prior to evaluation, all external downloads must pass through: | |
| 1. `Image.open().verify()` and RGB mode conversion. | |
| 2. Canonical resizing to standardized aspect ratios. | |
| 3. Cryptographic SHA-256 fingerprinting. | |
| 4. Schema validation into `benchmark/external_generalization_manifest.csv`. | |
| --- | |
| ## 4. Hypothesis & Multi-Model Routing Strategy | |
| Based on the empirical findings of Phase 6, we establish the following testable hypotheses: | |
| | Dataset Modality | Predicted Strongest Model | Predicted Weakest Model | Proposed Ensemble Strategy | | |
| | :--- | :--- | :--- | :--- | | |
| | **Facial GANs (FFHQ/StyleGAN)** | ViT (`dima806`) | ConvNeXt / UnivFD | **ViT-Priority Routing** | | |
| | **General-Scene Diffusion (GenImage)** | UnivFD / ConvNeXt | ViT (`dima806`) | **CLIP/ConvNeXt Consensus** | | |
| | **Video Deepfakes (FaceForensics++)** | ViT / ConvNeXt | UnivFD | **Confidence-Weighted Softmax** | | |
| | **Local Splicing (CASIA v2.0)** | ConvNeXt (DRCT) | ViT (`dima806`) | **Local Artifact Detection** | | |
| | **Pristine RAW (RAISE Controls)** | All (Target: Spec > 98%) | None | **Conservative Real Thresholding** | | |
| ### Next Step: Domain-Aware Dynamic Router | |
| Instead of static voting, the multi-model engine will implement a lightweight **Domain-Aware Dynamic Router**: | |
| 1. An input image is scanned for face presence (via MTCNN / RetinaFace or lightweight landmark check). | |
| 2. If **Facial Portrait:** Route to **ViT-Priority** (preserving 98% accuracy). | |
| 3. If **General Scene / Landscape:** Route to **UnivFD / ConvNeXt** consensus. | |
| 4. If **Ambiguous / Borderline:** Query full ensemble with uncertainty thresholding. | |