File size: 5,640 Bytes
ebaaf8b
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
---
title: Multimodel Setup
emoji: πŸ¦€
colorFrom: pink
colorTo: red
sdk: gradio
sdk_version: 5.20.0
python_version: '3.12'
app_file: app.py
pinned: false
---

# Multi-Model Deepfake Detection & Cross-Generator Evaluation

A multi-model deepfake detection framework that combines specialized computer vision architectures to detect AI-generated and manipulated images across diverse generator families (GANs, Diffusion, Midjourney, StyleGAN, ADM).

---

## πŸ’‘ The Core Idea

Single deepfake detectors suffer from **generator overfitting**:
- A model trained on facial GANs (like StyleGAN) often fails on diffusion models (like Midjourney).
- A model trained on general diffusion artifacts might miss fine-grained facial manipulations.

**Our Solution:** 
Instead of relying on a single fallible detector, we run multiple complementary vision models with distinct inductive biases and combine their outputs using **deterministic decision fusion**.

```mermaid
flowchart TD
    IMG["Input Image"] --> P1["ViT Preprocessor (224x224 Resize)"]
    IMG --> P2["UnivFD Preprocessor (Native CenterCrop + CLIP Norm)"]
    IMG --> P3["ConvNeXt Preprocessor (Native CenterCrop + ImageNet Norm)"]

    P1 --> M1["ViT Detector<br/>(Facial & GAN Artifacts)"]
    P2 --> M2["UnivFD<br/>(CLIP ViT-L/14 + Linear Probe)"]
    P3 --> M3["ConvNeXt DRCT<br/>(Diffusion Contrastive Base)"]

    M1 -->|Fake Prob / Vote| FUSION{"Decision Fusion Engine<br/>(src/decision_fusion.py)"}
    M2 -->|Fake Prob / Vote| FUSION
    M3 -->|Fake Prob / Vote| FUSION

    FUSION --> DECISION["Final Output:<br/>β€’ Prediction: REAL / FAKE<br/>β€’ Fused Confidence Score<br/>β€’ Per-Model Explanation"]
```

---

## πŸ” The Base Detectors

| Model | Architecture | Specialization / Training | Key Strength |
|---|---|---|---|
| **ViT** | Vision Transformer (`dima806`) | Facial deepfakes & GAN faces | High sensitivity on facial boundaries and warping |
| **UnivFD** | CLIP ViT-L/14 + Linear Classifier | Multi-generator frequency & semantic cues | Universal zero-shot artifact sensitivity |
| **ConvNeXt DRCT** | ConvNeXt-Base (CVPR 2024) | Diffusion Reconstruction Contrastive Training | Outstanding generalization to diffusion (100% on Midjourney) |

---

## βš™οΈ How It Works: Inference & Fusion Flow

```mermaid
sequenceDiagram
    autonumber
    actor User
    participant App as Web UI / API (app.py)
    participant Reg as Detector Registry
    participant Models as Base Detectors (ViT, UnivFD, ConvNeXt)
    participant Fusion as Decision Fusion

    User->>App: Upload Image
    App->>Reg: Request Predictions
    par Run Detectors in Parallel / Safe Sequential
        Reg->>Models: ViT Inference
        Reg->>Models: UnivFD Inference
        Reg->>Models: ConvNeXt DRCT Inference
    end
    Models-->>Reg: Individual Probabilities & Votes
    Reg->>Fusion: Submit {ViT: p1, UnivFD: p2, ConvNeXt: p3}
    Note over Fusion: Apply Strategy (Majority Vote / Mean Prob / Priority)
    Fusion-->>App: Unified Fused Verdict + Confidence Breakdown
    App-->>User: Display Results, Charts & Confidence
```

### Fusion Strategies
1. **Majority Voting:** Collects discrete votes (`REAL` vs `FAKE`). If a tie occurs, falls back to probability thresholding.
2. **Mean Fake Probability:** Computes the unweighted average of calibrated fake probabilities across all active detectors.
3. **Domain Priority:** Selects primary detector based on input context (e.g. facial vs non-facial diffusion synthesis).
4. **Graceful Degradation:** If any detector fails or checkpoint is absent, the engine falls back gracefully without crashing.

---

## πŸ“Š Evaluation & Parity Verification

Our framework was evaluated against both facial GAN benchmarks and external diffusion datasets (Midjourney v5, ADM, ImageNet nature photos):

```mermaid
flowchart LR
    A["Test Datasets"] --> B["Phase 6: FFHQ vs StyleGAN (Facial)"]
    A --> C["Phase 7: Midjourney + ADM vs ImageNet (Diffusion)"]
    
    B --> D["ViT: 99.5% Acc<br/>ConvNeXt: 82.5% Acc"]
    C --> E["ConvNeXt DRCT: 77.0% Acc (100% Midjourney Recall)<br/>UnivFD: 57.5% Acc<br/>ViT: 52.0% Acc"]
```

- **Reference Parity:** All adapters match official paper implementations (CVPR 2023 UniversalFakeDetect and CVPR 2024 DRCT) with numerical differences $< 10^{-6}$.
- **Comprehensive Regression Tests:** Automated test suite with 25 unit and contract tests verifying preprocessing shapes, normalization, and fusion edge cases.

---

## πŸš€ Quickstart

### 1. Installation
```bash
git clone https://github.com/Thiru-Selvam-06/Deepfake-Detection.git
cd Deepfake-Detection
pip install -r requirements.txt
```

### 2. Run the Interactive Web UI
```bash
python app.py
```
Open your browser at `http://localhost:7860`.

### 3. Run Automated Tests
```bash
pytest tests/ -v
```

---

## πŸ“ Repository Structure

```text
β”œβ”€β”€ app.py                   # Gradio Web Interface
β”œβ”€β”€ src/
β”‚   β”œβ”€β”€ convnext_adapter.py  # DRCT Native Crop Preprocessing & Inference
β”‚   β”œβ”€β”€ univfd_adapter.py    # UniversalFakeDetect CLIP Preprocessing & Inference
β”‚   β”œβ”€β”€ vit_adapter.py       # Vision Transformer Adapter
β”‚   β”œβ”€β”€ decision_fusion.py   # Multi-Model Voting & Probability Fusion Logic
β”‚   └── detector_registry.py # Unified Lifecycle & Model Registry
β”œβ”€β”€ benchmark/               # Labeled test manifests and evaluation results
β”œβ”€β”€ tests/                   # Regression and unit tests for adapters and fusion
β”œβ”€β”€ docs/                    # Detailed scientific evaluation reports
└── models/                  # Pretrained model weights and checkpoints (Git LFS)
```