| --- |
| license: mit |
| language: |
| - en |
| metrics: |
| - accuracy |
| - f1 |
| - precision |
| - recall |
| base_model: |
| - google/vit-base-patch16-224-in21k |
| library_name: transformers |
| tags: |
| - deepfake detection |
| - fake-image detection |
| --- |
| # ViT Deepfake Detection Model |
|
|
| ## Model Description |
|
|
| This is a fine-tuned Vision Transformer (ViT) model for binary image classification to detect deepfake images. The model is based on `google/vit-base-patch16-224-in21k` and has been fine-tuned on the OpenForensics dataset to distinguish between real and fake (AI-generated/manipulated) images. |
|
|
| ## Model Details |
|
|
| - **Model Type:** Vision Transformer (ViT) for Image Classification |
| - **Base Model:** google/vit-base-patch16-224-in21k |
| - **Task:** Binary Image Classification (Real vs Fake Detection) |
| - **Language:** N/A (Computer Vision) |
| - **License:** Apache 2.0 |
|
|
| ## Intended Use |
|
|
| ### Primary Use Cases |
| - Detecting AI-generated or manipulated images |
| - Content moderation and verification |
| - Research in deepfake detection |
| - Media authenticity verification |
|
|
| ### Out-of-Scope Use |
| - This model should not be used as the sole method for making critical decisions about content authenticity |
| - Not intended for surveillance or privacy-invasive applications |
| - May not generalize well to deepfake techniques not present in the training data |
|
|
| ## Training Data |
|
|
| The model was trained on the **OpenForensics dataset** with the following distribution: |
|
|
| - **Training Set:** 16,000 images |
| - **Validation Set:** 2000 images |
| - **Test Set:** 2000 images |
|
|
| Images were preprocessed and transformed using ViTImageProcessor with standard normalization. |
|
|
| ## Training Procedure |
|
|
| ### Hyperparameters |
|
|
| ```python |
| Training Arguments: |
| - Batch Size: 24 per device |
| - Gradient Accumulation Steps: 1 |
| - Mixed Precision: FP16 |
| - Number of Epochs: 10 |
| - Learning Rate: 3e-5 |
| - Weight Decay: 0.02 |
| - Warmup Ratio: 0.08 |
| - LR Scheduler: Cosine |
| - Label Smoothing: 0.05 |
| - Optimizer: AdamW (default) |
| ``` |
|
|
| ### Training Hardware |
| - GPU: Tesla T4 |
| - Training Time: ~14 minutes for 10 epochs |
|
|
| ### Data Augmentation |
| Standard ViT preprocessing with normalization applied via `ViTImageProcessor`. |
|
|
| ## Performance |
|
|
| ### Validation Set Results (Best Epoch - Epoch 5) |
|
|
| | Metric | Score | |
| |--------|-------| |
| | Accuracy | 96.22% | |
| | F1 Score | 96.22% | |
| | Precision | 96.30% | |
| | Recall | 96.22% | |
|
|
| ### Test Set Results |
|
|
| | Metric | Score | |
| |--------|-------| |
| | Accuracy | **96.56%** | |
|
|
| ### Training Progress |
|
|
| The model showed consistent improvement across epochs: |
|
|
| | Epoch | Training Loss | Validation Loss | Accuracy | F1 Score | |
| |-------|---------------|-----------------|----------|----------| |
| | 1 | 0.2259 | 0.2567 | 92.89% | 92.88% | |
| | 2 | 0.2002 | 0.2360 | 93.44% | 93.43% | |
| | 3 | 0.1388 | 0.1925 | 96.11% | 96.11% | |
| | 4 | 0.1322 | 0.2161 | 95.67% | 95.67% | |
| | 5 | 0.1182 | 0.2208 | **96.22%** | **96.22%** | |
| | 6-10 | 0.1170-0.1171 | 0.2132-0.2142 | 95.67-95.78% | 95.67-95.78% | |
|
|
| ## Usage |
|
|
| ### Loading the Model |
|
|
| ```python |
| from transformers import ViTImageProcessor, ViTForImageClassification |
| from PIL import Image |
| import torch |
| |
| # Load model and processor |
| model = ViTForImageClassification.from_pretrained("YOUR_USERNAME/vit-deepfake-detector") |
| processor = ViTImageProcessor.from_pretrained("YOUR_USERNAME/vit-deepfake-detector") |
| |
| # Load and preprocess image |
| image = Image.open("path_to_image.jpg") |
| inputs = processor(images=image, return_tensors="pt") |
| |
| # Make prediction |
| with torch.no_grad(): |
| outputs = model(**inputs) |
| logits = outputs.logits |
| predicted_class = logits.argmax(-1).item() |
| |
| # Get label |
| labels = {0: "real", 1: "fake"} |
| print(f"Prediction: {labels[predicted_class]}") |
| |
| # Get confidence scores |
| probabilities = torch.nn.functional.softmax(logits, dim=-1) |
| confidence = probabilities[0][predicted_class].item() |
| print(f"Confidence: {confidence:.2%}") |
| ``` |
|
|
| ### Batch Prediction |
|
|
| ```python |
| from transformers import pipeline |
| |
| # Create classification pipeline |
| classifier = pipeline("image-classification", model="YOUR_USERNAME/vit-deepfake-detector") |
| |
| # Predict on single image |
| result = classifier("path_to_image.jpg") |
| print(result) |
| |
| # Predict on multiple images |
| images = ["image1.jpg", "image2.jpg", "image3.jpg"] |
| results = classifier(images) |
| for img, result in zip(images, results): |
| print(f"{img}: {result}") |
| ``` |
|
|
| ## Limitations and Biases |
|
|
| ### Known Limitations |
| - **Dataset Bias:** The model was trained on the OpenForensics dataset, which may not represent all types of deepfakes or manipulation techniques |
| - **Generalization:** Performance may degrade on deepfake generation methods not present in the training data |
| - **Adversarial Robustness:** The model has not been explicitly tested against adversarial attacks |
| - **Resolution Dependency:** Best performance on images around 224x224 pixels (ViT input size) |
|
|
| ### Potential Biases |
| - The model's performance may vary across different: |
| - Image sources and quality levels |
| - Demographic representations in images |
| - Types of manipulation techniques |
| - Content domains (faces, landscapes, objects, etc.) |
|
|
| ## Ethical Considerations |
|
|
| - This model should be used responsibly and not for harassment or privacy invasion |
| - Decisions based on this model should involve human oversight, especially in high-stakes scenarios |
| - Users should be aware that deepfake detection is an evolving field, and no model is perfect |
| - False positives and false negatives can have real-world consequences |
|
|
| ## Citation |
|
|
| If you use this model, please cite: |
|
|
| ```bibtex |
| @misc{vit-deepfake-detector, |
| author = {YOUR_NAME}, |
| title = {ViT Deepfake Detection Model}, |
| year = {2024}, |
| publisher = {HuggingFace}, |
| howpublished = {\url{https://huggingface.co/YOUR_USERNAME/vit-deepfake-detector}} |
| } |
| ``` |
| ## Author |
| - Dr. Lucy Liu |
| - Muhammad Hamza Sohail |
| - Ayaan Mohammed |
| - Shadab Karim |
| - kirti Dhir |
| ## **Disclaimer:** |
| his model is provided for research and educational purposes. Users are responsible for ensuring compliance with applicable laws and ethical guidelines when deploying this model. |
