| --- |
| language: |
| - en |
| license: mit |
| tags: |
| - deepfake-detection |
| - video-deepfake |
| - face-forensics |
| - computer-vision |
| - gend |
| - pytorch |
| datasets: |
| - faceforensics |
| pipeline_tag: image-classification |
| base_model: openai/clip-vit-large-patch14 |
| --- |
| |
| # GenD_CLIP_L_14_FF |
|
|
| State-of-the-art Deepfake Detection model trained on **FaceForensics++ (FF++)** based on the **GenD** framework fine-tuned with **Sharpness-Aware Minimization (SAM)** and **Label Smoothing**. |
|
|
| This model uses **openai/clip-vit-large-patch14** as visual foundation backbone and fine-tunes only the **Layer Normalization parameters** (accounting for only ~0.03% of total parameters) while enforcing a **hyperspherical feature manifold** through L2-normalization and metric learning (Uniformity & Alignment losses). |
|
|
|
|
| ## 📊 Benchmark Results |
|
|
| | Metric | Score | |
| |:---|:---:| |
| | Video AUROC | **94.26%** | |
| | Video mAP | **93.30%** | |
| | Video Accuracy | **87.56%** | |
| | Video EER | **12.44%** | |
| | Frame AUROC | **90.57%** | |
| | Frame mAP | **89.14%** | |
| | Frame Accuracy | **82.91%** | |
|
|
| ## 📌 Model Details |
|
|
| - **Training Dataset:** `FaceForensics++ (FF++)` (`FF++`) |
| - **Visual Backbone:** `openai/clip-vit-large-patch14` |
| - **Classification Head:** `LinearNorm` |
| - **Optimizer:** `SAM-AdamW` (SAM $\rho=0.05$, adaptive=True) |
| - **Loss Formulation:** Cross-Entropy with Label Smoothing (0.1), Uniformity (0.5), Alignment (0.1) |
| - **Training Epochs:** `30` |
| - **Batch Size:** `96` |
| - **Precision:** `bf16-mixed` |
| - **Learning Rate:** `0.0003` |
|
|
| ## 🚀 Quickstart & Inference |
|
|
| ### 1. Using the Model in Python |
|
|
| ```python |
| import torch |
| from PIL import Image |
| from transformers import AutoModel |
| |
| # Load the model directly from Hugging Face Hub |
| model = AutoModel.from_pretrained("Ant-One/GenD_CLIP_L_14_FF", trust_remote_code=True) |
| model.eval() |
| |
| # Preprocess image crop (aligned face) |
| image = Image.open("path/to/face_crop.png").convert("RGB") |
| tensor = model.feature_extractor.preprocess(image).unsqueeze(0) |
| |
| # Run inference |
| with torch.no_grad(): |
| logits = model(tensor) |
| # Output class 0: Real, Output class 1: Fake |
| fake_prob = logits.softmax(dim=-1)[0, 1].item() |
| |
| print(f"Deepfake Probability: {fake_prob:.2%}") |
| ``` |
|
|
| ### 2. Using with `src.hf.modeling_gend` |
| |
| ```python |
| from src.hf.modeling_gend import GenD |
|
|
| model = GenD.from_pretrained("Ant-One/GenD_CLIP_L_14_FF") |
| model.eval() |
| ``` |
| |
| ## 🏗️ Architecture & Training Methodology |
| |
| The **GenD** method achieves superior cross-dataset generalization by avoiding catastrophic overfitting on manipulation-specific artifacts: |
| 1. **Training on FF++:** Trained on face crops from FaceForensics++ (FF++). |
| 2. **LayerNorm Tuning:** Keeps the visual transformer backbone frozen while updating only normalization scaling and bias terms. |
| 3. **Normalized Linear Head:** Normalizes feature vectors onto a hypersphere before linear projection. |
| 4. **SAM Optimization:** Smooths the loss landscape to find flat minima that resist out-of-distribution domain shifts. |
| |
