Spaces:
Running on Zero
Download docs/UNIVFD_LIMITATIONS.md from Thiru1122/Multimodel_setup: direct link, hf CLI and curl.
- Browser
- Download file 7.3 kB
-
https://huggingface.co/spaces/Thiru1122/Multimodel_setup/resolve/main/docs/UNIVFD_LIMITATIONS.md
- Command line
-
hf download hf://spaces/Thiru1122/Multimodel_setup/docs/UNIVFD_LIMITATIONS.md
-
curl -L -o UNIVFD_LIMITATIONS.md https://huggingface.co/spaces/Thiru1122/Multimodel_setup/resolve/main/docs/UNIVFD_LIMITATIONS.md
A newer version of the Gradio SDK is available: 6.32.0
Capability and Limitation Report: UniversalFakeDetect
Repository: WisconsinAIVision/UniversalFakeDetect (CVPR 2023)
Paper Reference: Towards Universal Fake Image Detectors that Generalize Across Generative Models (Ojha et al., CVPR 2023)
Evaluation Target Space: https://huggingface.co/spaces/Thiru1122/Multimodel_setup
1. Verified Model Architecture & Checkpoint Type
- Backbone Architecture: OpenAI CLIP
ViT-L/14(Vision Transformer, 768 hidden feature dimension, $224 \times 224$ input resolution with patch size 14). The vision backbone weights (ViT-L-14.pt, ~890 MB) are frozen during training and inference. - Classification Head: Single linear layer (
nn.Linear(768, 1)). - Checkpoint File:
pretrained_weights/fc_weights.pth(4,083 bytes).- Structure:
collections.OrderedDictcontainingweightof shape[1, 768]andbiasof shape[1](scalar-0.51670). - Checkpoint Scope: Classifier head only (linear probe). It contains zero backbone parameters.
- Structure:
- Absence of Alternative Checkpoints: Exhaustive inspection of the repository confirms that no alternative
.pthcheckpoints exist in the repository. Whilemodels/__init__.pylists other model family names (e.g., ResNet-50, Swin, other ViT variants), no corresponding trained weights are supplied.
2. Preprocessing Options in Repository
Three distinct preprocessing configurations exist across the original repository code:
- Bundled CLIP Transform (
models/clip/clip.py):Resize(224, interpolation=BICUBIC) -> CenterCrop(224) -> ToTensor() -> Normalize(clip_mean, clip_std).- Used by default in
model.preprocessand utilized by the current application interface.
- Official Validation Script Transform (
validate.py):CenterCrop(224) -> ToTensor() -> Normalize(clip_mean, clip_std).- Contains no resize step. On high-resolution inputs ($4000 \times 3000$), this evaluates only a central $224 \times 224$ crop (0.418% of the total pixel area), ignoring 99.58% of the image.
- Training & Validation Dataset Transform (
data/datasets.py):Resize(256, interpolation=BICUBIC) -> CenterCrop(224) -> ToTensor() -> Normalize(clip_mean, clip_std).- Assumes training images from the Wang2020 dataset, which were natively low-resolution ($256 \times 256$).
3. Decision Logic, Label Mapping, and Thresholding
- Label Mapping:
0 = REAL(Negative class)1 = FAKE(Positive class)- Training loss objective:
nn.BCEWithLogitsLoss().
- Logit Interpretation:
- Output tensor is a single raw scalar logit: $\text{logit} \in (-\infty, +\infty)$.
- Fake probability: $P(\text{fake}) = \sigma(\text{logit}) = \frac{1}{1 + e^{-\text{logit}}}$.
- Real probability: $P(\text{real}) = 1.0 - P(\text{fake})$.
- Official Threshold:
- Default threshold is $0.50$ probability (corresponding to raw logit $0.00$).
- If $P(\text{fake}) \ge 0.50$, verdict is FAKE; if $P(\text{fake}) < 0.50$, verdict is REAL.
4. Official Evaluation Scope in CVPR 2023 Paper
The official repository was designed and evaluated on specific historical datasets (dataset_paths.py):
- Training Dataset: ProGAN (Wang et al., 2020), consisting of $256 \times 256$ images across 20 object categories.
- Evaluation Benchmarks (2022 era):
- GAN Models: ProGAN, StyleGAN, BigGAN, CycleGAN, StarGAN, GauGAN, DeepFake.
- Early Diffusion Models: Guided Diffusion, Latent Diffusion (LDM-100, LDM-200), Glide (2021), DALL-E mini.
5. Diagnostic Test Results on Evaluated Test Images
The model was evaluated across 5 standardized test cases using scripts/test_univfd_adapter.py:
| Test Input | Input Category | Ground-Truth Nature | Prediction | Raw Logit | Fake Probability | Confidence | Status |
|---|---|---|---|---|---|---|---|
real_phone_photo.png |
Valid Photo | Real camera capture ($4000 \times 3000$) | REAL | -10.9329 |
$0.0018%$ | $100.00%$ | Accordant |
ai_cyberpunk_city_1789660863192.jpg |
Valid AI Image | Google Imagen 3 diffusion ($1024 \times 1024$) | REAL | -7.0961 |
$0.0828%$ | $99.92%$ | Out-of-Domain Observation |
manipulated_image.png |
Valid Composite | Digital splicing composite ($800 \times 800$) | REAL | -5.1081 |
$0.6011%$ | $99.40%$ | Out-of-Domain Observation |
corrupted_image.png |
Corrupted File | Truncated 20-byte file | ERROR | N/A | N/A | N/A | Cleanly Rejected |
_temp_invalid.txt |
Non-Image File | Plain text document | ERROR | N/A | N/A | N/A | Cleanly Rejected |
6. Distinguishing Integration Success from Prediction Correctness
A fundamental distinction must be maintained between system integration and detector accuracy:
- Integration Success:
- The model loads correctly from weights.
- Exact official preprocessing is executed.
- Memory, device allocation, and execution operate without runtime exceptions.
- The output logits match the original author's repository implementation exactly down to 4 decimal places.
- Prediction Correctness:
- Whether the predicted binary label matches the actual empirical ground truth of the image.
- Integration success does not imply prediction correctness.
7. Model Limitations and Generalization Bounds
The following technical factors explain why modern AI-generated and manipulated images produced REAL classifications:
- Distribution Shift (Observation):
- The linear probe was trained exclusively on 2018-era ProGAN artifacts. ProGAN exhibits recognizable spectral and spatial frequency anomalies resulting from transpose convolutions and upsampling.
- Modern 2024+ diffusion models (such as Google Imagen 3) utilize advanced continuous-time flow matching and high-capacity latent decoders that eliminate early GAN frequency signatures.
- Representation Space Geometry:
- UniversalFakeDetect relies on frozen CLIP ViT-L/14 semantic embeddings. CLIP was pretrained on 400M natural image-text pairs.
- Modern photorealistic diffusion images project into the natural semantic subspace of CLIP.
- In this subspace, the linear probe dot product produces strongly negative projections (combined with the negative bias of
-0.5167), resulting in negative logits ($-7.0961$) that place the sample deep in the REAL category.
- High Confidence $\neq$ Authenticity Verification:
- The model outputting $99.92%$ confidence for REAL on an AI image is a mathematical consequence of the sigmoid saturation on negative logits ($\sigma(-7.0961) = 0.000828 \implies 1 - 0.000828 = 0.999172$).
- It reflects the complete absence of ProGAN-like artifacts, not verification of photographic authenticity.
8. Summary of Mandatory Constraints
- UniversalFakeDetect must never be presented as an authenticity verifier.
- Smoke tests and pipeline validation tests must never be cited as evidence of detection accuracy.
- Reliable evaluation of modern generative content requires specialized multi-model ensemble architectures and calibrated benchmarks.