Multimodel_setup / README.md
Thiru1122's picture
docs: add concise README with architecture flowcharts and detector overview
ebaaf8b
|
Raw History Blame Contribute Delete
5.64 kB

A newer version of the Gradio SDK is available: 6.32.0

Upgrade
metadata
title: Multimodel Setup
emoji: πŸ¦€
colorFrom: pink
colorTo: red
sdk: gradio
sdk_version: 5.20.0
python_version: '3.12'
app_file: app.py
pinned: false

Multi-Model Deepfake Detection & Cross-Generator Evaluation

A multi-model deepfake detection framework that combines specialized computer vision architectures to detect AI-generated and manipulated images across diverse generator families (GANs, Diffusion, Midjourney, StyleGAN, ADM).


πŸ’‘ The Core Idea

Single deepfake detectors suffer from generator overfitting:

  • A model trained on facial GANs (like StyleGAN) often fails on diffusion models (like Midjourney).
  • A model trained on general diffusion artifacts might miss fine-grained facial manipulations.

Our Solution: Instead of relying on a single fallible detector, we run multiple complementary vision models with distinct inductive biases and combine their outputs using deterministic decision fusion.

flowchart TD
    IMG["Input Image"] --> P1["ViT Preprocessor (224x224 Resize)"]
    IMG --> P2["UnivFD Preprocessor (Native CenterCrop + CLIP Norm)"]
    IMG --> P3["ConvNeXt Preprocessor (Native CenterCrop + ImageNet Norm)"]

    P1 --> M1["ViT Detector<br/>(Facial & GAN Artifacts)"]
    P2 --> M2["UnivFD<br/>(CLIP ViT-L/14 + Linear Probe)"]
    P3 --> M3["ConvNeXt DRCT<br/>(Diffusion Contrastive Base)"]

    M1 -->|Fake Prob / Vote| FUSION{"Decision Fusion Engine<br/>(src/decision_fusion.py)"}
    M2 -->|Fake Prob / Vote| FUSION
    M3 -->|Fake Prob / Vote| FUSION

    FUSION --> DECISION["Final Output:<br/>β€’ Prediction: REAL / FAKE<br/>β€’ Fused Confidence Score<br/>β€’ Per-Model Explanation"]

πŸ” The Base Detectors

Model Architecture Specialization / Training Key Strength
ViT Vision Transformer (dima806) Facial deepfakes & GAN faces High sensitivity on facial boundaries and warping
UnivFD CLIP ViT-L/14 + Linear Classifier Multi-generator frequency & semantic cues Universal zero-shot artifact sensitivity
ConvNeXt DRCT ConvNeXt-Base (CVPR 2024) Diffusion Reconstruction Contrastive Training Outstanding generalization to diffusion (100% on Midjourney)

βš™οΈ How It Works: Inference & Fusion Flow

sequenceDiagram
    autonumber
    actor User
    participant App as Web UI / API (app.py)
    participant Reg as Detector Registry
    participant Models as Base Detectors (ViT, UnivFD, ConvNeXt)
    participant Fusion as Decision Fusion

    User->>App: Upload Image
    App->>Reg: Request Predictions
    par Run Detectors in Parallel / Safe Sequential
        Reg->>Models: ViT Inference
        Reg->>Models: UnivFD Inference
        Reg->>Models: ConvNeXt DRCT Inference
    end
    Models-->>Reg: Individual Probabilities & Votes
    Reg->>Fusion: Submit {ViT: p1, UnivFD: p2, ConvNeXt: p3}
    Note over Fusion: Apply Strategy (Majority Vote / Mean Prob / Priority)
    Fusion-->>App: Unified Fused Verdict + Confidence Breakdown
    App-->>User: Display Results, Charts & Confidence

Fusion Strategies

  1. Majority Voting: Collects discrete votes (REAL vs FAKE). If a tie occurs, falls back to probability thresholding.
  2. Mean Fake Probability: Computes the unweighted average of calibrated fake probabilities across all active detectors.
  3. Domain Priority: Selects primary detector based on input context (e.g. facial vs non-facial diffusion synthesis).
  4. Graceful Degradation: If any detector fails or checkpoint is absent, the engine falls back gracefully without crashing.

πŸ“Š Evaluation & Parity Verification

Our framework was evaluated against both facial GAN benchmarks and external diffusion datasets (Midjourney v5, ADM, ImageNet nature photos):

flowchart LR
    A["Test Datasets"] --> B["Phase 6: FFHQ vs StyleGAN (Facial)"]
    A --> C["Phase 7: Midjourney + ADM vs ImageNet (Diffusion)"]
    
    B --> D["ViT: 99.5% Acc<br/>ConvNeXt: 82.5% Acc"]
    C --> E["ConvNeXt DRCT: 77.0% Acc (100% Midjourney Recall)<br/>UnivFD: 57.5% Acc<br/>ViT: 52.0% Acc"]
  • Reference Parity: All adapters match official paper implementations (CVPR 2023 UniversalFakeDetect and CVPR 2024 DRCT) with numerical differences $< 10^{-6}$.
  • Comprehensive Regression Tests: Automated test suite with 25 unit and contract tests verifying preprocessing shapes, normalization, and fusion edge cases.

πŸš€ Quickstart

1. Installation

git clone https://github.com/Thiru-Selvam-06/Deepfake-Detection.git
cd Deepfake-Detection
pip install -r requirements.txt

2. Run the Interactive Web UI

python app.py

Open your browser at http://localhost:7860.

3. Run Automated Tests

pytest tests/ -v

πŸ“ Repository Structure

β”œβ”€β”€ app.py                   # Gradio Web Interface
β”œβ”€β”€ src/
β”‚   β”œβ”€β”€ convnext_adapter.py  # DRCT Native Crop Preprocessing & Inference
β”‚   β”œβ”€β”€ univfd_adapter.py    # UniversalFakeDetect CLIP Preprocessing & Inference
β”‚   β”œβ”€β”€ vit_adapter.py       # Vision Transformer Adapter
β”‚   β”œβ”€β”€ decision_fusion.py   # Multi-Model Voting & Probability Fusion Logic
β”‚   └── detector_registry.py # Unified Lifecycle & Model Registry
β”œβ”€β”€ benchmark/               # Labeled test manifests and evaluation results
β”œβ”€β”€ tests/                   # Regression and unit tests for adapters and fusion
β”œβ”€β”€ docs/                    # Detailed scientific evaluation reports
└── models/                  # Pretrained model weights and checkpoints (Git LFS)