Spaces:
Running on Zero
Running on Zero
File size: 5,640 Bytes
ebaaf8b | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 | ---
title: Multimodel Setup
emoji: π¦
colorFrom: pink
colorTo: red
sdk: gradio
sdk_version: 5.20.0
python_version: '3.12'
app_file: app.py
pinned: false
---
# Multi-Model Deepfake Detection & Cross-Generator Evaluation
A multi-model deepfake detection framework that combines specialized computer vision architectures to detect AI-generated and manipulated images across diverse generator families (GANs, Diffusion, Midjourney, StyleGAN, ADM).
---
## π‘ The Core Idea
Single deepfake detectors suffer from **generator overfitting**:
- A model trained on facial GANs (like StyleGAN) often fails on diffusion models (like Midjourney).
- A model trained on general diffusion artifacts might miss fine-grained facial manipulations.
**Our Solution:**
Instead of relying on a single fallible detector, we run multiple complementary vision models with distinct inductive biases and combine their outputs using **deterministic decision fusion**.
```mermaid
flowchart TD
IMG["Input Image"] --> P1["ViT Preprocessor (224x224 Resize)"]
IMG --> P2["UnivFD Preprocessor (Native CenterCrop + CLIP Norm)"]
IMG --> P3["ConvNeXt Preprocessor (Native CenterCrop + ImageNet Norm)"]
P1 --> M1["ViT Detector<br/>(Facial & GAN Artifacts)"]
P2 --> M2["UnivFD<br/>(CLIP ViT-L/14 + Linear Probe)"]
P3 --> M3["ConvNeXt DRCT<br/>(Diffusion Contrastive Base)"]
M1 -->|Fake Prob / Vote| FUSION{"Decision Fusion Engine<br/>(src/decision_fusion.py)"}
M2 -->|Fake Prob / Vote| FUSION
M3 -->|Fake Prob / Vote| FUSION
FUSION --> DECISION["Final Output:<br/>β’ Prediction: REAL / FAKE<br/>β’ Fused Confidence Score<br/>β’ Per-Model Explanation"]
```
---
## π The Base Detectors
| Model | Architecture | Specialization / Training | Key Strength |
|---|---|---|---|
| **ViT** | Vision Transformer (`dima806`) | Facial deepfakes & GAN faces | High sensitivity on facial boundaries and warping |
| **UnivFD** | CLIP ViT-L/14 + Linear Classifier | Multi-generator frequency & semantic cues | Universal zero-shot artifact sensitivity |
| **ConvNeXt DRCT** | ConvNeXt-Base (CVPR 2024) | Diffusion Reconstruction Contrastive Training | Outstanding generalization to diffusion (100% on Midjourney) |
---
## βοΈ How It Works: Inference & Fusion Flow
```mermaid
sequenceDiagram
autonumber
actor User
participant App as Web UI / API (app.py)
participant Reg as Detector Registry
participant Models as Base Detectors (ViT, UnivFD, ConvNeXt)
participant Fusion as Decision Fusion
User->>App: Upload Image
App->>Reg: Request Predictions
par Run Detectors in Parallel / Safe Sequential
Reg->>Models: ViT Inference
Reg->>Models: UnivFD Inference
Reg->>Models: ConvNeXt DRCT Inference
end
Models-->>Reg: Individual Probabilities & Votes
Reg->>Fusion: Submit {ViT: p1, UnivFD: p2, ConvNeXt: p3}
Note over Fusion: Apply Strategy (Majority Vote / Mean Prob / Priority)
Fusion-->>App: Unified Fused Verdict + Confidence Breakdown
App-->>User: Display Results, Charts & Confidence
```
### Fusion Strategies
1. **Majority Voting:** Collects discrete votes (`REAL` vs `FAKE`). If a tie occurs, falls back to probability thresholding.
2. **Mean Fake Probability:** Computes the unweighted average of calibrated fake probabilities across all active detectors.
3. **Domain Priority:** Selects primary detector based on input context (e.g. facial vs non-facial diffusion synthesis).
4. **Graceful Degradation:** If any detector fails or checkpoint is absent, the engine falls back gracefully without crashing.
---
## π Evaluation & Parity Verification
Our framework was evaluated against both facial GAN benchmarks and external diffusion datasets (Midjourney v5, ADM, ImageNet nature photos):
```mermaid
flowchart LR
A["Test Datasets"] --> B["Phase 6: FFHQ vs StyleGAN (Facial)"]
A --> C["Phase 7: Midjourney + ADM vs ImageNet (Diffusion)"]
B --> D["ViT: 99.5% Acc<br/>ConvNeXt: 82.5% Acc"]
C --> E["ConvNeXt DRCT: 77.0% Acc (100% Midjourney Recall)<br/>UnivFD: 57.5% Acc<br/>ViT: 52.0% Acc"]
```
- **Reference Parity:** All adapters match official paper implementations (CVPR 2023 UniversalFakeDetect and CVPR 2024 DRCT) with numerical differences $< 10^{-6}$.
- **Comprehensive Regression Tests:** Automated test suite with 25 unit and contract tests verifying preprocessing shapes, normalization, and fusion edge cases.
---
## π Quickstart
### 1. Installation
```bash
git clone https://github.com/Thiru-Selvam-06/Deepfake-Detection.git
cd Deepfake-Detection
pip install -r requirements.txt
```
### 2. Run the Interactive Web UI
```bash
python app.py
```
Open your browser at `http://localhost:7860`.
### 3. Run Automated Tests
```bash
pytest tests/ -v
```
---
## π Repository Structure
```text
βββ app.py # Gradio Web Interface
βββ src/
β βββ convnext_adapter.py # DRCT Native Crop Preprocessing & Inference
β βββ univfd_adapter.py # UniversalFakeDetect CLIP Preprocessing & Inference
β βββ vit_adapter.py # Vision Transformer Adapter
β βββ decision_fusion.py # Multi-Model Voting & Probability Fusion Logic
β βββ detector_registry.py # Unified Lifecycle & Model Registry
βββ benchmark/ # Labeled test manifests and evaluation results
βββ tests/ # Regression and unit tests for adapters and fusion
βββ docs/ # Detailed scientific evaluation reports
βββ models/ # Pretrained model weights and checkpoints (Git LFS)
```
|