Spaces:
Running on Zero
Running on Zero
docs: add concise README with architecture flowcharts and detector overview
Browse files
README.md
CHANGED
|
@@ -1,13 +1,144 @@
|
|
| 1 |
-
---
|
| 2 |
-
title: Multimodel Setup
|
| 3 |
-
emoji: π¦
|
| 4 |
-
colorFrom: pink
|
| 5 |
-
colorTo: red
|
| 6 |
-
sdk: gradio
|
| 7 |
-
sdk_version: 5.20.0
|
| 8 |
-
python_version: '3.12'
|
| 9 |
-
app_file: app.py
|
| 10 |
-
pinned: false
|
| 11 |
-
---
|
| 12 |
-
|
| 13 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
title: Multimodel Setup
|
| 3 |
+
emoji: π¦
|
| 4 |
+
colorFrom: pink
|
| 5 |
+
colorTo: red
|
| 6 |
+
sdk: gradio
|
| 7 |
+
sdk_version: 5.20.0
|
| 8 |
+
python_version: '3.12'
|
| 9 |
+
app_file: app.py
|
| 10 |
+
pinned: false
|
| 11 |
+
---
|
| 12 |
+
|
| 13 |
+
# Multi-Model Deepfake Detection & Cross-Generator Evaluation
|
| 14 |
+
|
| 15 |
+
A multi-model deepfake detection framework that combines specialized computer vision architectures to detect AI-generated and manipulated images across diverse generator families (GANs, Diffusion, Midjourney, StyleGAN, ADM).
|
| 16 |
+
|
| 17 |
+
---
|
| 18 |
+
|
| 19 |
+
## π‘ The Core Idea
|
| 20 |
+
|
| 21 |
+
Single deepfake detectors suffer from **generator overfitting**:
|
| 22 |
+
- A model trained on facial GANs (like StyleGAN) often fails on diffusion models (like Midjourney).
|
| 23 |
+
- A model trained on general diffusion artifacts might miss fine-grained facial manipulations.
|
| 24 |
+
|
| 25 |
+
**Our Solution:**
|
| 26 |
+
Instead of relying on a single fallible detector, we run multiple complementary vision models with distinct inductive biases and combine their outputs using **deterministic decision fusion**.
|
| 27 |
+
|
| 28 |
+
```mermaid
|
| 29 |
+
flowchart TD
|
| 30 |
+
IMG["Input Image"] --> P1["ViT Preprocessor (224x224 Resize)"]
|
| 31 |
+
IMG --> P2["UnivFD Preprocessor (Native CenterCrop + CLIP Norm)"]
|
| 32 |
+
IMG --> P3["ConvNeXt Preprocessor (Native CenterCrop + ImageNet Norm)"]
|
| 33 |
+
|
| 34 |
+
P1 --> M1["ViT Detector<br/>(Facial & GAN Artifacts)"]
|
| 35 |
+
P2 --> M2["UnivFD<br/>(CLIP ViT-L/14 + Linear Probe)"]
|
| 36 |
+
P3 --> M3["ConvNeXt DRCT<br/>(Diffusion Contrastive Base)"]
|
| 37 |
+
|
| 38 |
+
M1 -->|Fake Prob / Vote| FUSION{"Decision Fusion Engine<br/>(src/decision_fusion.py)"}
|
| 39 |
+
M2 -->|Fake Prob / Vote| FUSION
|
| 40 |
+
M3 -->|Fake Prob / Vote| FUSION
|
| 41 |
+
|
| 42 |
+
FUSION --> DECISION["Final Output:<br/>β’ Prediction: REAL / FAKE<br/>β’ Fused Confidence Score<br/>β’ Per-Model Explanation"]
|
| 43 |
+
```
|
| 44 |
+
|
| 45 |
+
---
|
| 46 |
+
|
| 47 |
+
## π The Base Detectors
|
| 48 |
+
|
| 49 |
+
| Model | Architecture | Specialization / Training | Key Strength |
|
| 50 |
+
|---|---|---|---|
|
| 51 |
+
| **ViT** | Vision Transformer (`dima806`) | Facial deepfakes & GAN faces | High sensitivity on facial boundaries and warping |
|
| 52 |
+
| **UnivFD** | CLIP ViT-L/14 + Linear Classifier | Multi-generator frequency & semantic cues | Universal zero-shot artifact sensitivity |
|
| 53 |
+
| **ConvNeXt DRCT** | ConvNeXt-Base (CVPR 2024) | Diffusion Reconstruction Contrastive Training | Outstanding generalization to diffusion (100% on Midjourney) |
|
| 54 |
+
|
| 55 |
+
---
|
| 56 |
+
|
| 57 |
+
## βοΈ How It Works: Inference & Fusion Flow
|
| 58 |
+
|
| 59 |
+
```mermaid
|
| 60 |
+
sequenceDiagram
|
| 61 |
+
autonumber
|
| 62 |
+
actor User
|
| 63 |
+
participant App as Web UI / API (app.py)
|
| 64 |
+
participant Reg as Detector Registry
|
| 65 |
+
participant Models as Base Detectors (ViT, UnivFD, ConvNeXt)
|
| 66 |
+
participant Fusion as Decision Fusion
|
| 67 |
+
|
| 68 |
+
User->>App: Upload Image
|
| 69 |
+
App->>Reg: Request Predictions
|
| 70 |
+
par Run Detectors in Parallel / Safe Sequential
|
| 71 |
+
Reg->>Models: ViT Inference
|
| 72 |
+
Reg->>Models: UnivFD Inference
|
| 73 |
+
Reg->>Models: ConvNeXt DRCT Inference
|
| 74 |
+
end
|
| 75 |
+
Models-->>Reg: Individual Probabilities & Votes
|
| 76 |
+
Reg->>Fusion: Submit {ViT: p1, UnivFD: p2, ConvNeXt: p3}
|
| 77 |
+
Note over Fusion: Apply Strategy (Majority Vote / Mean Prob / Priority)
|
| 78 |
+
Fusion-->>App: Unified Fused Verdict + Confidence Breakdown
|
| 79 |
+
App-->>User: Display Results, Charts & Confidence
|
| 80 |
+
```
|
| 81 |
+
|
| 82 |
+
### Fusion Strategies
|
| 83 |
+
1. **Majority Voting:** Collects discrete votes (`REAL` vs `FAKE`). If a tie occurs, falls back to probability thresholding.
|
| 84 |
+
2. **Mean Fake Probability:** Computes the unweighted average of calibrated fake probabilities across all active detectors.
|
| 85 |
+
3. **Domain Priority:** Selects primary detector based on input context (e.g. facial vs non-facial diffusion synthesis).
|
| 86 |
+
4. **Graceful Degradation:** If any detector fails or checkpoint is absent, the engine falls back gracefully without crashing.
|
| 87 |
+
|
| 88 |
+
---
|
| 89 |
+
|
| 90 |
+
## π Evaluation & Parity Verification
|
| 91 |
+
|
| 92 |
+
Our framework was evaluated against both facial GAN benchmarks and external diffusion datasets (Midjourney v5, ADM, ImageNet nature photos):
|
| 93 |
+
|
| 94 |
+
```mermaid
|
| 95 |
+
flowchart LR
|
| 96 |
+
A["Test Datasets"] --> B["Phase 6: FFHQ vs StyleGAN (Facial)"]
|
| 97 |
+
A --> C["Phase 7: Midjourney + ADM vs ImageNet (Diffusion)"]
|
| 98 |
+
|
| 99 |
+
B --> D["ViT: 99.5% Acc<br/>ConvNeXt: 82.5% Acc"]
|
| 100 |
+
C --> E["ConvNeXt DRCT: 77.0% Acc (100% Midjourney Recall)<br/>UnivFD: 57.5% Acc<br/>ViT: 52.0% Acc"]
|
| 101 |
+
```
|
| 102 |
+
|
| 103 |
+
- **Reference Parity:** All adapters match official paper implementations (CVPR 2023 UniversalFakeDetect and CVPR 2024 DRCT) with numerical differences $< 10^{-6}$.
|
| 104 |
+
- **Comprehensive Regression Tests:** Automated test suite with 25 unit and contract tests verifying preprocessing shapes, normalization, and fusion edge cases.
|
| 105 |
+
|
| 106 |
+
---
|
| 107 |
+
|
| 108 |
+
## π Quickstart
|
| 109 |
+
|
| 110 |
+
### 1. Installation
|
| 111 |
+
```bash
|
| 112 |
+
git clone https://github.com/Thiru-Selvam-06/Deepfake-Detection.git
|
| 113 |
+
cd Deepfake-Detection
|
| 114 |
+
pip install -r requirements.txt
|
| 115 |
+
```
|
| 116 |
+
|
| 117 |
+
### 2. Run the Interactive Web UI
|
| 118 |
+
```bash
|
| 119 |
+
python app.py
|
| 120 |
+
```
|
| 121 |
+
Open your browser at `http://localhost:7860`.
|
| 122 |
+
|
| 123 |
+
### 3. Run Automated Tests
|
| 124 |
+
```bash
|
| 125 |
+
pytest tests/ -v
|
| 126 |
+
```
|
| 127 |
+
|
| 128 |
+
---
|
| 129 |
+
|
| 130 |
+
## π Repository Structure
|
| 131 |
+
|
| 132 |
+
```text
|
| 133 |
+
βββ app.py # Gradio Web Interface
|
| 134 |
+
βββ src/
|
| 135 |
+
β βββ convnext_adapter.py # DRCT Native Crop Preprocessing & Inference
|
| 136 |
+
β βββ univfd_adapter.py # UniversalFakeDetect CLIP Preprocessing & Inference
|
| 137 |
+
β βββ vit_adapter.py # Vision Transformer Adapter
|
| 138 |
+
β βββ decision_fusion.py # Multi-Model Voting & Probability Fusion Logic
|
| 139 |
+
β βββ detector_registry.py # Unified Lifecycle & Model Registry
|
| 140 |
+
βββ benchmark/ # Labeled test manifests and evaluation results
|
| 141 |
+
βββ tests/ # Regression and unit tests for adapters and fusion
|
| 142 |
+
βββ docs/ # Detailed scientific evaluation reports
|
| 143 |
+
βββ models/ # Pretrained model weights and checkpoints (Git LFS)
|
| 144 |
+
```
|