Spaces:
Running on Zero
Running on Zero
Add standardized detector adapter and evaluation documentation
Browse files- docs/MULTI_MODEL_EVALUATION_PLAN.md +113 -0
- docs/UNIVFD_LIMITATIONS.md +112 -0
- scripts/adapter_test_results.json +122 -0
- scripts/test_univfd_adapter.py +161 -0
- src/__init__.py +1 -0
- src/detector_schema.py +78 -0
- src/univfd_adapter.py +231 -0
docs/MULTI_MODEL_EVALUATION_PLAN.md
ADDED
|
@@ -0,0 +1,113 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Multi-Model Evaluation and Comparison Plan
|
| 2 |
+
|
| 3 |
+
**Target Space**: [`https://huggingface.co/spaces/Thiru1122/Multimodel_setup`](https://huggingface.co/spaces/Thiru1122/Multimodel_setup)
|
| 4 |
+
**Status**: Architectural Planning Document (Preparation Phase — No Agreement Layer Implemented Yet)
|
| 5 |
+
|
| 6 |
+
---
|
| 7 |
+
|
| 8 |
+
## 1. Overview & Objective
|
| 9 |
+
|
| 10 |
+
No individual deepfake detection architecture generalizes universally across all generative algorithms (GANs, autoregressive transformers, latent diffusion, flow-matching) and manipulation techniques (inpainting, face-swapping, splicing).
|
| 11 |
+
|
| 12 |
+
The objective of this framework is to establish a rigorous, standardized, multi-model evaluation harness that allows multiple independent detectors to analyze identical input assets under consistent protocols, producing standardized diagnostic outputs and agreement metrics without modifying individual model behaviors.
|
| 13 |
+
|
| 14 |
+
---
|
| 15 |
+
|
| 16 |
+
## 2. Common Input Format
|
| 17 |
+
|
| 18 |
+
To ensure parity across all candidate models, all inputs entering the evaluation harness must conform to the following pipeline:
|
| 19 |
+
|
| 20 |
+
1. **Primary Input Types**:
|
| 21 |
+
- `PIL.Image.Image` object, OR
|
| 22 |
+
- Absolute file path string pointing to a local image file.
|
| 23 |
+
2. **Standard Ingress Transformation**:
|
| 24 |
+
- Validation against corruption and truncated headers using `PIL.Image.open()`.
|
| 25 |
+
- Explicit conversion to 3-channel RGB mode (`.convert("RGB")`).
|
| 26 |
+
- Preservation of original image aspect ratio and native resolution metadata prior to model-specific preprocessing.
|
| 27 |
+
3. **Model-Specific Downstream Adapters**:
|
| 28 |
+
- Each detector adapter is responsible for applying its own exact author-defined preprocessing (e.g., specific resizing interpolation, cropping dimensions, normalization means and standard deviations).
|
| 29 |
+
|
| 30 |
+
---
|
| 31 |
+
|
| 32 |
+
## 3. Common Result Schema
|
| 33 |
+
|
| 34 |
+
All candidate detectors must implement an adapter adhering strictly to the `DetectorResult` schema defined in [`src/detector_schema.py`](file:///c:/Users/user/Desktop/multi%20model%20connection/src/detector_schema.py):
|
| 35 |
+
|
| 36 |
+
```python
|
| 37 |
+
@dataclass
|
| 38 |
+
class DetectorResult:
|
| 39 |
+
detector_name: str # Unique identifier of detector (e.g. "UniversalFakeDetect")
|
| 40 |
+
model_name: str # Backbone and checkpoint descriptor
|
| 41 |
+
prediction: str # Strictly one of: "REAL", "FAKE", "UNCERTAIN", "ERROR"
|
| 42 |
+
fake_probability: float # Normalized probability in [0.0, 1.0] that input is synthetic
|
| 43 |
+
real_probability: float # Normalized probability in [0.0, 1.0] that input is real
|
| 44 |
+
raw_logit: float # Unbounded pre-activation output logit
|
| 45 |
+
confidence: float # Confidence score in [0.0, 1.0]
|
| 46 |
+
threshold: float # Operational decision boundary (e.g., 0.50)
|
| 47 |
+
device: str # Execution hardware ("cuda", "cpu")
|
| 48 |
+
preprocessing: str # Summary description of transform pipeline
|
| 49 |
+
inference_time_ms: float # Wall-clock execution time in milliseconds
|
| 50 |
+
error: Optional[str] # Descriptive error trace if prediction is "ERROR"
|
| 51 |
+
warning: Optional[str] # Mandatory authenticity limitation notice
|
| 52 |
+
```
|
| 53 |
+
|
| 54 |
+
Allowed prediction values are strictly bounded to:
|
| 55 |
+
- `REAL`: Detector classified input as natural photograph.
|
| 56 |
+
- `FAKE`: Detector classified input as synthetic / manipulated.
|
| 57 |
+
- `UNCERTAIN`: Prediction cannot be reliably assigned (e.g., logit falls in dead-band or model disagreement).
|
| 58 |
+
- `ERROR`: Input was invalid, corrupted, or pipeline encountered an unrecoverable exception.
|
| 59 |
+
|
| 60 |
+
---
|
| 61 |
+
|
| 62 |
+
## 4. Common Error Handling Protocol
|
| 63 |
+
|
| 64 |
+
1. **Rejection at Ingress**:
|
| 65 |
+
- Corrupted files, non-image formats, and truncated byte streams must produce `prediction: "ERROR"` with a non-null `error` field.
|
| 66 |
+
2. **No Silent Error Fallback**:
|
| 67 |
+
- Under no circumstances may an unhandled exception or decode failure default silently to `REAL` or `FAKE`.
|
| 68 |
+
3. **Hardware & OOM Isolation**:
|
| 69 |
+
- If an individual detector exhausts GPU memory or times out, its adapter must return an `ERROR` result without crashing the enclosing process or aborting sibling detector evaluations.
|
| 70 |
+
|
| 71 |
+
---
|
| 72 |
+
|
| 73 |
+
## 5. Per-Model Reporting & Aggregation
|
| 74 |
+
|
| 75 |
+
When multiple detectors evaluate a single input, the evaluation harness must report:
|
| 76 |
+
|
| 77 |
+
1. **Individual Model Records**:
|
| 78 |
+
- Tabular display of each detector's raw logit, calibrated fake probability, assigned binary label, and latency.
|
| 79 |
+
2. **Consensus & Disagreement**:
|
| 80 |
+
- Quantitative concordance count (e.g., `2/3 detectors predict FAKE`).
|
| 81 |
+
- Mean and variance of predicted fake probabilities across models.
|
| 82 |
+
3. **The `UNCERTAIN` State**:
|
| 83 |
+
- When constituent models produce conflicting predictions (e.g., Detector A outputs `REAL` with 99% confidence, while Detector B outputs `FAKE` with 95% confidence), the composite verdict must be explicitly designated as **`UNCERTAIN`**.
|
| 84 |
+
- The system must not arbitrate conflicting predictions by simple average if the underlying feature distributions are uncalibrated.
|
| 85 |
+
|
| 86 |
+
---
|
| 87 |
+
|
| 88 |
+
## 6. Rigorous Evaluation Protocol & Benchmark Requirements
|
| 89 |
+
|
| 90 |
+
Future evaluations across multiple models must adhere to the following experimental standards:
|
| 91 |
+
|
| 92 |
+
1. **Curated Ground-Truth Benchmark**:
|
| 93 |
+
- Evaluation must be conducted on a balanced, labeled benchmark dataset containing verified real photographs, known GAN images (ProGAN, StyleGAN), modern diffusion images (Midjourney, Stable Diffusion, Imagen, Flux), and localized image manipulations.
|
| 94 |
+
2. **Separate Reporting of Error Types**:
|
| 95 |
+
- Metrics must report False-Real (False Negative / Missed AI) and False-Fake (False Positive / Flagged Real) error rates independently.
|
| 96 |
+
- Aggregate accuracy alone is misleading on imbalanced or out-of-distribution test sets.
|
| 97 |
+
3. **Calibration Requirements**:
|
| 98 |
+
- Raw logits from distinct architectures cannot be compared directly without Platt scaling or temperature calibration on a shared validation set.
|
| 99 |
+
|
| 100 |
+
---
|
| 101 |
+
|
| 102 |
+
## 7. Fundamental Disclaimer on Model Agreement
|
| 103 |
+
|
| 104 |
+
> [!IMPORTANT]
|
| 105 |
+
> **Consensus $\neq$ Authenticity Verification:**
|
| 106 |
+
> If all detectors in a multi-model ensemble classify an image as `REAL`, this indicates only that none of the models detected the specific synthetic patterns for which they were trained. It **does not verify** that the image is an authentic photograph. Generative models developed after a detector's training cutoff may bypass all detectors simultaneously.
|
| 107 |
+
|
| 108 |
+
---
|
| 109 |
+
|
| 110 |
+
## 8. Implementation Boundary Notice
|
| 111 |
+
|
| 112 |
+
**Current State**: Only UniversalFakeDetect and its standardized adapter (`UnivFDAdapter`) are active.
|
| 113 |
+
**Strict Boundary**: The multi-model agreement layer, additional detector repositories (e.g., RAID), and ensemble fusion logic are **not implemented** in this phase.
|
docs/UNIVFD_LIMITATIONS.md
ADDED
|
@@ -0,0 +1,112 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Capability and Limitation Report: UniversalFakeDetect
|
| 2 |
+
|
| 3 |
+
**Repository**: [`WisconsinAIVision/UniversalFakeDetect`](https://github.com/WisconsinAIVision/UniversalFakeDetect) (CVPR 2023)
|
| 4 |
+
**Paper Reference**: *Towards Universal Fake Image Detectors that Generalize Across Generative Models* (Ojha et al., CVPR 2023)
|
| 5 |
+
**Evaluation Target Space**: [`https://huggingface.co/spaces/Thiru1122/Multimodel_setup`](https://huggingface.co/spaces/Thiru1122/Multimodel_setup)
|
| 6 |
+
|
| 7 |
+
---
|
| 8 |
+
|
| 9 |
+
## 1. Verified Model Architecture & Checkpoint Type
|
| 10 |
+
|
| 11 |
+
- **Backbone Architecture**: OpenAI CLIP `ViT-L/14` (Vision Transformer, 768 hidden feature dimension, $224 \times 224$ input resolution with patch size 14). The vision backbone weights (`ViT-L-14.pt`, ~890 MB) are frozen during training and inference.
|
| 12 |
+
- **Classification Head**: Single linear layer (`nn.Linear(768, 1)`).
|
| 13 |
+
- **Checkpoint File**: `pretrained_weights/fc_weights.pth` (4,083 bytes).
|
| 14 |
+
- Structure: `collections.OrderedDict` containing `weight` of shape `[1, 768]` and `bias` of shape `[1]` (scalar `-0.51670`).
|
| 15 |
+
- Checkpoint Scope: **Classifier head only (linear probe)**. It contains zero backbone parameters.
|
| 16 |
+
- **Absence of Alternative Checkpoints**: Exhaustive inspection of the repository confirms that **no alternative `.pth` checkpoints** exist in the repository. While `models/__init__.py` lists other model family names (e.g., ResNet-50, Swin, other ViT variants), no corresponding trained weights are supplied.
|
| 17 |
+
|
| 18 |
+
---
|
| 19 |
+
|
| 20 |
+
## 2. Preprocessing Options in Repository
|
| 21 |
+
|
| 22 |
+
Three distinct preprocessing configurations exist across the original repository code:
|
| 23 |
+
|
| 24 |
+
1. **Bundled CLIP Transform (`models/clip/clip.py`)**:
|
| 25 |
+
- `Resize(224, interpolation=BICUBIC) -> CenterCrop(224) -> ToTensor() -> Normalize(clip_mean, clip_std)`.
|
| 26 |
+
- Used by default in `model.preprocess` and utilized by the current application interface.
|
| 27 |
+
2. **Official Validation Script Transform (`validate.py`)**:
|
| 28 |
+
- `CenterCrop(224) -> ToTensor() -> Normalize(clip_mean, clip_std)`.
|
| 29 |
+
- Contains **no resize step**. On high-resolution inputs ($4000 \times 3000$), this evaluates only a central $224 \times 224$ crop (0.418% of the total pixel area), ignoring 99.58% of the image.
|
| 30 |
+
3. **Training & Validation Dataset Transform (`data/datasets.py`)**:
|
| 31 |
+
- `Resize(256, interpolation=BICUBIC) -> CenterCrop(224) -> ToTensor() -> Normalize(clip_mean, clip_std)`.
|
| 32 |
+
- Assumes training images from the Wang2020 dataset, which were natively low-resolution ($256 \times 256$).
|
| 33 |
+
|
| 34 |
+
---
|
| 35 |
+
|
| 36 |
+
## 3. Decision Logic, Label Mapping, and Thresholding
|
| 37 |
+
|
| 38 |
+
- **Label Mapping**:
|
| 39 |
+
- `0 = REAL` (Negative class)
|
| 40 |
+
- `1 = FAKE` (Positive class)
|
| 41 |
+
- Training loss objective: `nn.BCEWithLogitsLoss()`.
|
| 42 |
+
- **Logit Interpretation**:
|
| 43 |
+
- Output tensor is a single raw scalar logit: $\text{logit} \in (-\infty, +\infty)$.
|
| 44 |
+
- Fake probability: $P(\text{fake}) = \sigma(\text{logit}) = \frac{1}{1 + e^{-\text{logit}}}$.
|
| 45 |
+
- Real probability: $P(\text{real}) = 1.0 - P(\text{fake})$.
|
| 46 |
+
- **Official Threshold**:
|
| 47 |
+
- Default threshold is **$0.50$ probability** (corresponding to raw logit $0.00$).
|
| 48 |
+
- If $P(\text{fake}) \ge 0.50$, verdict is **FAKE**; if $P(\text{fake}) < 0.50$, verdict is **REAL**.
|
| 49 |
+
|
| 50 |
+
---
|
| 51 |
+
|
| 52 |
+
## 4. Official Evaluation Scope in CVPR 2023 Paper
|
| 53 |
+
|
| 54 |
+
The official repository was designed and evaluated on specific historical datasets (`dataset_paths.py`):
|
| 55 |
+
1. **Training Dataset**: ProGAN (Wang et al., 2020), consisting of $256 \times 256$ images across 20 object categories.
|
| 56 |
+
2. **Evaluation Benchmarks (2022 era)**:
|
| 57 |
+
- GAN Models: ProGAN, StyleGAN, BigGAN, CycleGAN, StarGAN, GauGAN, DeepFake.
|
| 58 |
+
- Early Diffusion Models: Guided Diffusion, Latent Diffusion (LDM-100, LDM-200), Glide (2021), DALL-E mini.
|
| 59 |
+
|
| 60 |
+
---
|
| 61 |
+
|
| 62 |
+
## 5. Diagnostic Test Results on Evaluated Test Images
|
| 63 |
+
|
| 64 |
+
The model was evaluated across 5 standardized test cases using `scripts/test_univfd_adapter.py`:
|
| 65 |
+
|
| 66 |
+
| Test Input | Input Category | Ground-Truth Nature | Prediction | Raw Logit | Fake Probability | Confidence | Status |
|
| 67 |
+
| :--- | :--- | :--- | :---: | :---: | :---: | :---: | :---: |
|
| 68 |
+
| `real_phone_photo.png` | Valid Photo | Real camera capture ($4000 \times 3000$) | **REAL** | `-10.9329` | $0.0018\%$ | $100.00\%$ | Accordant |
|
| 69 |
+
| `ai_cyberpunk_city_1789660863192.jpg` | Valid AI Image | Google Imagen 3 diffusion ($1024 \times 1024$) | **REAL** | `-7.0961` | $0.0828\%$ | $99.92\%$ | Out-of-Domain Observation |
|
| 70 |
+
| `manipulated_image.png` | Valid Composite | Digital splicing composite ($800 \times 800$) | **REAL** | `-5.1081` | $0.6011\%$ | $99.40\%$ | Out-of-Domain Observation |
|
| 71 |
+
| `corrupted_image.png` | Corrupted File | Truncated 20-byte file | **ERROR** | *N/A* | *N/A* | *N/A* | Cleanly Rejected |
|
| 72 |
+
| `_temp_invalid.txt` | Non-Image File | Plain text document | **ERROR** | *N/A* | *N/A* | *N/A* | Cleanly Rejected |
|
| 73 |
+
|
| 74 |
+
---
|
| 75 |
+
|
| 76 |
+
## 6. Distinguishing Integration Success from Prediction Correctness
|
| 77 |
+
|
| 78 |
+
A fundamental distinction must be maintained between system integration and detector accuracy:
|
| 79 |
+
|
| 80 |
+
1. **Integration Success**:
|
| 81 |
+
- The model loads correctly from weights.
|
| 82 |
+
- Exact official preprocessing is executed.
|
| 83 |
+
- Memory, device allocation, and execution operate without runtime exceptions.
|
| 84 |
+
- The output logits match the original author's repository implementation exactly down to 4 decimal places.
|
| 85 |
+
2. **Prediction Correctness**:
|
| 86 |
+
- Whether the predicted binary label matches the actual empirical ground truth of the image.
|
| 87 |
+
- Integration success **does not imply** prediction correctness.
|
| 88 |
+
|
| 89 |
+
---
|
| 90 |
+
|
| 91 |
+
## 7. Model Limitations and Generalization Bounds
|
| 92 |
+
|
| 93 |
+
The following technical factors explain why modern AI-generated and manipulated images produced **REAL** classifications:
|
| 94 |
+
|
| 95 |
+
1. **Distribution Shift (Observation)**:
|
| 96 |
+
- The linear probe was trained exclusively on 2018-era ProGAN artifacts. ProGAN exhibits recognizable spectral and spatial frequency anomalies resulting from transpose convolutions and upsampling.
|
| 97 |
+
- Modern 2024+ diffusion models (such as Google Imagen 3) utilize advanced continuous-time flow matching and high-capacity latent decoders that eliminate early GAN frequency signatures.
|
| 98 |
+
2. **Representation Space Geometry**:
|
| 99 |
+
- UniversalFakeDetect relies on frozen CLIP ViT-L/14 semantic embeddings. CLIP was pretrained on 400M natural image-text pairs.
|
| 100 |
+
- Modern photorealistic diffusion images project into the natural semantic subspace of CLIP.
|
| 101 |
+
- In this subspace, the linear probe dot product produces strongly negative projections (combined with the negative bias of `-0.5167`), resulting in negative logits ($-7.0961$) that place the sample deep in the REAL category.
|
| 102 |
+
3. **High Confidence $\neq$ Authenticity Verification**:
|
| 103 |
+
- The model outputting $99.92\%$ confidence for REAL on an AI image is a mathematical consequence of the sigmoid saturation on negative logits ($\sigma(-7.0961) = 0.000828 \implies 1 - 0.000828 = 0.999172$).
|
| 104 |
+
- It reflects the complete absence of ProGAN-like artifacts, **not** verification of photographic authenticity.
|
| 105 |
+
|
| 106 |
+
---
|
| 107 |
+
|
| 108 |
+
## 8. Summary of Mandatory Constraints
|
| 109 |
+
|
| 110 |
+
- UniversalFakeDetect must **never** be presented as an authenticity verifier.
|
| 111 |
+
- Smoke tests and pipeline validation tests must **never** be cited as evidence of detection accuracy.
|
| 112 |
+
- Reliable evaluation of modern generative content requires specialized multi-model ensemble architectures and calibrated benchmarks.
|
scripts/adapter_test_results.json
ADDED
|
@@ -0,0 +1,122 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
[
|
| 2 |
+
{
|
| 3 |
+
"test_id": 1,
|
| 4 |
+
"name": "Known real photograph (smartphone capture)",
|
| 5 |
+
"category": "VALID_REAL_IMAGE",
|
| 6 |
+
"ground_truth_label": "REAL",
|
| 7 |
+
"expected_outcome": "SUCCESSFUL_INFERENCE",
|
| 8 |
+
"operational_status": "SUCCESSFUL_INFERENCE",
|
| 9 |
+
"domain_assessment": "ACCORDANT (Prediction matches ground truth label)",
|
| 10 |
+
"result": {
|
| 11 |
+
"detector_name": "UniversalFakeDetect",
|
| 12 |
+
"model_name": "CLIP:ViT-L/14 + fc_weights.pth",
|
| 13 |
+
"prediction": "REAL",
|
| 14 |
+
"fake_probability": 1.8e-05,
|
| 15 |
+
"real_probability": 0.999982,
|
| 16 |
+
"raw_logit": -10.932908,
|
| 17 |
+
"confidence": 0.999982,
|
| 18 |
+
"threshold": 0.5,
|
| 19 |
+
"device": "cpu",
|
| 20 |
+
"preprocessing": "Resize(224, bicubic) -> CenterCrop(224) -> ToTensor() -> Normalize(clip)",
|
| 21 |
+
"inference_time_ms": 16069.53,
|
| 22 |
+
"error": null,
|
| 23 |
+
"warning": "Identifies learned patterns associated with synthetic or manipulated images. Does not guarantee image authenticity."
|
| 24 |
+
}
|
| 25 |
+
},
|
| 26 |
+
{
|
| 27 |
+
"test_id": 2,
|
| 28 |
+
"name": "Known AI-generated image (Google Imagen 3)",
|
| 29 |
+
"category": "VALID_AI_IMAGE",
|
| 30 |
+
"ground_truth_label": "FAKE",
|
| 31 |
+
"expected_outcome": "SUCCESSFUL_INFERENCE",
|
| 32 |
+
"operational_status": "SUCCESSFUL_INFERENCE",
|
| 33 |
+
"domain_assessment": "MODEL_LIMITATION_OBSERVED (Known ProGAN-trained classifier boundary: output REAL on FAKE input)",
|
| 34 |
+
"result": {
|
| 35 |
+
"detector_name": "UniversalFakeDetect",
|
| 36 |
+
"model_name": "CLIP:ViT-L/14 + fc_weights.pth",
|
| 37 |
+
"prediction": "REAL",
|
| 38 |
+
"fake_probability": 0.000828,
|
| 39 |
+
"real_probability": 0.999172,
|
| 40 |
+
"raw_logit": -7.096089,
|
| 41 |
+
"confidence": 0.999172,
|
| 42 |
+
"threshold": 0.5,
|
| 43 |
+
"device": "cpu",
|
| 44 |
+
"preprocessing": "Resize(224, bicubic) -> CenterCrop(224) -> ToTensor() -> Normalize(clip)",
|
| 45 |
+
"inference_time_ms": 621.43,
|
| 46 |
+
"error": null,
|
| 47 |
+
"warning": "Identifies learned patterns associated with synthetic or manipulated images. Does not guarantee image authenticity."
|
| 48 |
+
}
|
| 49 |
+
},
|
| 50 |
+
{
|
| 51 |
+
"test_id": 3,
|
| 52 |
+
"name": "Known manipulated image (digital composite)",
|
| 53 |
+
"category": "VALID_MANIPULATED_IMAGE",
|
| 54 |
+
"ground_truth_label": "MANIPULATED",
|
| 55 |
+
"expected_outcome": "SUCCESSFUL_INFERENCE",
|
| 56 |
+
"operational_status": "SUCCESSFUL_INFERENCE",
|
| 57 |
+
"domain_assessment": "MODEL_LIMITATION_OBSERVED (Known ProGAN-trained classifier boundary: output REAL on MANIPULATED input)",
|
| 58 |
+
"result": {
|
| 59 |
+
"detector_name": "UniversalFakeDetect",
|
| 60 |
+
"model_name": "CLIP:ViT-L/14 + fc_weights.pth",
|
| 61 |
+
"prediction": "REAL",
|
| 62 |
+
"fake_probability": 0.006011,
|
| 63 |
+
"real_probability": 0.993989,
|
| 64 |
+
"raw_logit": -5.108142,
|
| 65 |
+
"confidence": 0.993989,
|
| 66 |
+
"threshold": 0.5,
|
| 67 |
+
"device": "cpu",
|
| 68 |
+
"preprocessing": "Resize(224, bicubic) -> CenterCrop(224) -> ToTensor() -> Normalize(clip)",
|
| 69 |
+
"inference_time_ms": 633.63,
|
| 70 |
+
"error": null,
|
| 71 |
+
"warning": "Identifies learned patterns associated with synthetic or manipulated images. Does not guarantee image authenticity."
|
| 72 |
+
}
|
| 73 |
+
},
|
| 74 |
+
{
|
| 75 |
+
"test_id": 4,
|
| 76 |
+
"name": "Corrupted image file (truncated header)",
|
| 77 |
+
"category": "CORRUPTED_FILE",
|
| 78 |
+
"ground_truth_label": "CORRUPTED",
|
| 79 |
+
"expected_outcome": "INPUT_REJECTION",
|
| 80 |
+
"operational_status": "INPUT_REJECTION_VERIFIED",
|
| 81 |
+
"domain_assessment": "N/A (Input correctly rejected before model evaluation)",
|
| 82 |
+
"result": {
|
| 83 |
+
"detector_name": "UniversalFakeDetect",
|
| 84 |
+
"model_name": "CLIP:ViT-L/14 + fc_weights.pth",
|
| 85 |
+
"prediction": "ERROR",
|
| 86 |
+
"fake_probability": null,
|
| 87 |
+
"real_probability": null,
|
| 88 |
+
"raw_logit": null,
|
| 89 |
+
"confidence": null,
|
| 90 |
+
"threshold": null,
|
| 91 |
+
"device": "cpu",
|
| 92 |
+
"preprocessing": "Resize(224, bicubic) -> CenterCrop(224) -> ToTensor() -> Normalize(clip)",
|
| 93 |
+
"inference_time_ms": null,
|
| 94 |
+
"error": "Invalid or corrupted image: cannot identify image file 'C:\\\\Users\\\\user\\\\.gemini\\\\antigravity-ide\\\\brain\\\\3b54ff09-c09f-4fe6-9276-f4cca9fbc6d7\\\\scratch\\\\corrupted_image.png'",
|
| 95 |
+
"warning": "Identifies learned patterns associated with synthetic or manipulated images. Does not guarantee image authenticity."
|
| 96 |
+
}
|
| 97 |
+
},
|
| 98 |
+
{
|
| 99 |
+
"test_id": 5,
|
| 100 |
+
"name": "Unsupported file format (plain text document)",
|
| 101 |
+
"category": "UNSUPPORTED_INPUT",
|
| 102 |
+
"ground_truth_label": "INVALID_TYPE",
|
| 103 |
+
"expected_outcome": "INPUT_REJECTION",
|
| 104 |
+
"operational_status": "INPUT_REJECTION_VERIFIED",
|
| 105 |
+
"domain_assessment": "N/A (Input correctly rejected before model evaluation)",
|
| 106 |
+
"result": {
|
| 107 |
+
"detector_name": "UniversalFakeDetect",
|
| 108 |
+
"model_name": "CLIP:ViT-L/14 + fc_weights.pth",
|
| 109 |
+
"prediction": "ERROR",
|
| 110 |
+
"fake_probability": null,
|
| 111 |
+
"real_probability": null,
|
| 112 |
+
"raw_logit": null,
|
| 113 |
+
"confidence": null,
|
| 114 |
+
"threshold": null,
|
| 115 |
+
"device": "cpu",
|
| 116 |
+
"preprocessing": "Resize(224, bicubic) -> CenterCrop(224) -> ToTensor() -> Normalize(clip)",
|
| 117 |
+
"inference_time_ms": null,
|
| 118 |
+
"error": "Invalid or corrupted image: cannot identify image file 'C:\\\\Users\\\\user\\\\Desktop\\\\multi model connection\\\\scripts\\\\_temp_invalid.txt'",
|
| 119 |
+
"warning": "Identifies learned patterns associated with synthetic or manipulated images. Does not guarantee image authenticity."
|
| 120 |
+
}
|
| 121 |
+
}
|
| 122 |
+
]
|
scripts/test_univfd_adapter.py
ADDED
|
@@ -0,0 +1,161 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Diagnostic Test Suite for UnivFDAdapter
|
| 3 |
+
Validates adapter integration, schema adherence, error handling, and input rejection.
|
| 4 |
+
NOTE: This script verifies operational pipeline behavior. It is NOT an accuracy evaluation
|
| 5 |
+
and does NOT verify image authenticity.
|
| 6 |
+
"""
|
| 7 |
+
|
| 8 |
+
import os
|
| 9 |
+
import sys
|
| 10 |
+
import json
|
| 11 |
+
from PIL import Image
|
| 12 |
+
|
| 13 |
+
# Add project root to sys.path
|
| 14 |
+
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
| 15 |
+
if BASE_DIR not in sys.path:
|
| 16 |
+
sys.path.insert(0, BASE_DIR)
|
| 17 |
+
|
| 18 |
+
from src.univfd_adapter import UnivFDAdapter
|
| 19 |
+
from src.detector_schema import PREDICTION_REAL, PREDICTION_FAKE, PREDICTION_ERROR
|
| 20 |
+
|
| 21 |
+
ASSET_DIR = r"C:\Users\user\.gemini\antigravity-ide\brain\3b54ff09-c09f-4fe6-9276-f4cca9fbc6d7"
|
| 22 |
+
|
| 23 |
+
def main():
|
| 24 |
+
print("=" * 75)
|
| 25 |
+
print("UNIVERSALFAKEDETECT ADAPTER DIAGNOSTIC TEST SUITE")
|
| 26 |
+
print("Verifies adapter schema, error handling, and inference integrity.")
|
| 27 |
+
print("DISCLAIMER: This test validates integration mechanics, not model accuracy.")
|
| 28 |
+
print("=" * 75 + "\n")
|
| 29 |
+
|
| 30 |
+
adapter = UnivFDAdapter(lazy_load=True)
|
| 31 |
+
|
| 32 |
+
# Prepare non-image text file for test case 5
|
| 33 |
+
invalid_txt_path = os.path.join(BASE_DIR, "scripts", "_temp_invalid.txt")
|
| 34 |
+
with open(invalid_txt_path, "w", encoding="utf-8") as f:
|
| 35 |
+
f.write("This is a plain text file, not an image format.")
|
| 36 |
+
|
| 37 |
+
test_cases = [
|
| 38 |
+
{
|
| 39 |
+
"id": 1,
|
| 40 |
+
"category": "VALID_REAL_IMAGE",
|
| 41 |
+
"name": "Known real photograph (smartphone capture)",
|
| 42 |
+
"input": os.path.join(ASSET_DIR, "scratch", "real_phone_photo.png"),
|
| 43 |
+
"ground_truth_label": "REAL",
|
| 44 |
+
"expected_outcome": "SUCCESSFUL_INFERENCE",
|
| 45 |
+
"nature": "Natural photographic capture from smartphone camera.",
|
| 46 |
+
},
|
| 47 |
+
{
|
| 48 |
+
"id": 2,
|
| 49 |
+
"category": "VALID_AI_IMAGE",
|
| 50 |
+
"name": "Known AI-generated image (Google Imagen 3)",
|
| 51 |
+
"input": os.path.join(ASSET_DIR, "ai_cyberpunk_city_1789660863192.jpg"),
|
| 52 |
+
"ground_truth_label": "FAKE",
|
| 53 |
+
"expected_outcome": "SUCCESSFUL_INFERENCE",
|
| 54 |
+
"nature": "Modern text-to-image diffusion generation (out-of-distribution for ProGAN-trained head).",
|
| 55 |
+
},
|
| 56 |
+
{
|
| 57 |
+
"id": 3,
|
| 58 |
+
"category": "VALID_MANIPULATED_IMAGE",
|
| 59 |
+
"name": "Known manipulated image (digital composite)",
|
| 60 |
+
"input": os.path.join(ASSET_DIR, "scratch", "manipulated_image.png"),
|
| 61 |
+
"ground_truth_label": "MANIPULATED",
|
| 62 |
+
"expected_outcome": "SUCCESSFUL_INFERENCE",
|
| 63 |
+
"nature": "Digitally spliced composite of photographic and synthetic elements.",
|
| 64 |
+
},
|
| 65 |
+
{
|
| 66 |
+
"id": 4,
|
| 67 |
+
"category": "CORRUPTED_FILE",
|
| 68 |
+
"name": "Corrupted image file (truncated header)",
|
| 69 |
+
"input": os.path.join(ASSET_DIR, "scratch", "corrupted_image.png"),
|
| 70 |
+
"ground_truth_label": "CORRUPTED",
|
| 71 |
+
"expected_outcome": "INPUT_REJECTION",
|
| 72 |
+
"nature": "Invalid 20-byte file that cannot be decoded by image decoders.",
|
| 73 |
+
},
|
| 74 |
+
{
|
| 75 |
+
"id": 5,
|
| 76 |
+
"category": "UNSUPPORTED_INPUT",
|
| 77 |
+
"name": "Unsupported file format (plain text document)",
|
| 78 |
+
"input": invalid_txt_path,
|
| 79 |
+
"ground_truth_label": "INVALID_TYPE",
|
| 80 |
+
"expected_outcome": "INPUT_REJECTION",
|
| 81 |
+
"nature": "Non-image text file submitted to image inference pipeline.",
|
| 82 |
+
},
|
| 83 |
+
]
|
| 84 |
+
|
| 85 |
+
all_results = []
|
| 86 |
+
|
| 87 |
+
for tc in test_cases:
|
| 88 |
+
print("-" * 75)
|
| 89 |
+
print(f"TEST {tc['id']}: {tc['name']}")
|
| 90 |
+
print(f" Input Source: {tc['input']}")
|
| 91 |
+
print(f" Category: {tc['category']}")
|
| 92 |
+
print(f" Ground-Truth Label: {tc['ground_truth_label']}")
|
| 93 |
+
print(f" Expected Outcome: {tc['expected_outcome']}")
|
| 94 |
+
print(f" Test Nature: {tc['nature']}")
|
| 95 |
+
|
| 96 |
+
result = adapter.predict(tc["input"])
|
| 97 |
+
res_dict = result.to_dict()
|
| 98 |
+
|
| 99 |
+
# Operational status assessment
|
| 100 |
+
is_error = result.prediction == PREDICTION_ERROR
|
| 101 |
+
if tc["expected_outcome"] == "INPUT_REJECTION":
|
| 102 |
+
status_classification = "INPUT_REJECTION_VERIFIED" if is_error else "FAILURE (Did not reject)"
|
| 103 |
+
else:
|
| 104 |
+
status_classification = "SUCCESSFUL_INFERENCE" if not is_error else "EXECUTION_FAILURE"
|
| 105 |
+
|
| 106 |
+
# Correctness vs Model Limitation assessment
|
| 107 |
+
if not is_error:
|
| 108 |
+
if result.prediction == tc["ground_truth_label"]:
|
| 109 |
+
domain_assessment = "ACCORDANT (Prediction matches ground truth label)"
|
| 110 |
+
else:
|
| 111 |
+
domain_assessment = (
|
| 112 |
+
"MODEL_LIMITATION_OBSERVED (Known ProGAN-trained classifier boundary: "
|
| 113 |
+
f"output {result.prediction} on {tc['ground_truth_label']} input)"
|
| 114 |
+
)
|
| 115 |
+
else:
|
| 116 |
+
domain_assessment = "N/A (Input correctly rejected before model evaluation)"
|
| 117 |
+
|
| 118 |
+
print(f" Prediction: {result.prediction}")
|
| 119 |
+
print(f" Raw Model Logit: {f'{result.raw_logit:+.5f}' if result.raw_logit is not None else 'N/A'}")
|
| 120 |
+
print(f" Fake Probability: {f'{result.fake_probability:.6f}' if result.fake_probability is not None else 'N/A'}")
|
| 121 |
+
print(f" Confidence: {f'{result.confidence*100:.2f}%' if result.confidence is not None else 'N/A'}")
|
| 122 |
+
print(f" Threshold: {result.threshold}")
|
| 123 |
+
print(f" Compute Device: {result.device}")
|
| 124 |
+
print(f" Inference Latency: {f'{result.inference_time_ms:.2f} ms' if result.inference_time_ms is not None else 'N/A'}")
|
| 125 |
+
print(f" Error Field: {result.error if result.error else 'None'}")
|
| 126 |
+
print(f" Operational Status: {status_classification}")
|
| 127 |
+
print(f" Domain Assessment: {domain_assessment}")
|
| 128 |
+
|
| 129 |
+
all_results.append({
|
| 130 |
+
"test_id": tc["id"],
|
| 131 |
+
"name": tc["name"],
|
| 132 |
+
"category": tc["category"],
|
| 133 |
+
"ground_truth_label": tc["ground_truth_label"],
|
| 134 |
+
"expected_outcome": tc["expected_outcome"],
|
| 135 |
+
"operational_status": status_classification,
|
| 136 |
+
"domain_assessment": domain_assessment,
|
| 137 |
+
"result": res_dict
|
| 138 |
+
})
|
| 139 |
+
print()
|
| 140 |
+
|
| 141 |
+
# Clean up temporary test file
|
| 142 |
+
if os.path.exists(invalid_txt_path):
|
| 143 |
+
try:
|
| 144 |
+
os.remove(invalid_txt_path)
|
| 145 |
+
except Exception:
|
| 146 |
+
pass
|
| 147 |
+
|
| 148 |
+
print("=" * 75)
|
| 149 |
+
print("TEST SUITE SUMMARY")
|
| 150 |
+
print("=" * 75)
|
| 151 |
+
for r in all_results:
|
| 152 |
+
res = r["result"]
|
| 153 |
+
print(f"Test {r['test_id']}: {r['name'][:30]:<30} | Pred: {res['prediction']:<6} | Logit: {str(res['raw_logit']):<9} | Op: {r['operational_status']}")
|
| 154 |
+
|
| 155 |
+
summary_file = os.path.join(BASE_DIR, "scripts", "adapter_test_results.json")
|
| 156 |
+
with open(summary_file, "w", encoding="utf-8") as f:
|
| 157 |
+
json.dump(all_results, f, indent=2)
|
| 158 |
+
print(f"\nSaved structured test results to: {summary_file}")
|
| 159 |
+
|
| 160 |
+
if __name__ == "__main__":
|
| 161 |
+
main()
|
src/__init__.py
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
"""Source package for multi-model deepfake detector evaluation framework."""
|
src/detector_schema.py
ADDED
|
@@ -0,0 +1,78 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Standardized Detector Result Schema
|
| 3 |
+
Defines a consistent output schema for deepfake detectors to enable standardized logging,
|
| 4 |
+
diagnostics, and future multi-model comparison without modifying underlying detector behaviors.
|
| 5 |
+
"""
|
| 6 |
+
|
| 7 |
+
from dataclasses import dataclass, asdict
|
| 8 |
+
from typing import Optional, Dict, Any, Set
|
| 9 |
+
|
| 10 |
+
PREDICTION_REAL = "REAL"
|
| 11 |
+
PREDICTION_FAKE = "FAKE"
|
| 12 |
+
PREDICTION_UNCERTAIN = "UNCERTAIN"
|
| 13 |
+
PREDICTION_ERROR = "ERROR"
|
| 14 |
+
|
| 15 |
+
VALID_PREDICTIONS: Set[str] = {
|
| 16 |
+
PREDICTION_REAL,
|
| 17 |
+
PREDICTION_FAKE,
|
| 18 |
+
PREDICTION_UNCERTAIN,
|
| 19 |
+
PREDICTION_ERROR,
|
| 20 |
+
}
|
| 21 |
+
|
| 22 |
+
|
| 23 |
+
@dataclass
|
| 24 |
+
class DetectorResult:
|
| 25 |
+
"""
|
| 26 |
+
Standardized result data structure for detector outputs.
|
| 27 |
+
|
| 28 |
+
Allowed prediction values:
|
| 29 |
+
- 'REAL'
|
| 30 |
+
- 'FAKE'
|
| 31 |
+
- 'UNCERTAIN'
|
| 32 |
+
- 'ERROR'
|
| 33 |
+
"""
|
| 34 |
+
detector_name: str
|
| 35 |
+
model_name: str
|
| 36 |
+
prediction: str
|
| 37 |
+
fake_probability: Optional[float] = None
|
| 38 |
+
real_probability: Optional[float] = None
|
| 39 |
+
raw_logit: Optional[float] = None
|
| 40 |
+
confidence: Optional[float] = None
|
| 41 |
+
threshold: Optional[float] = None
|
| 42 |
+
device: Optional[str] = None
|
| 43 |
+
preprocessing: Optional[str] = None
|
| 44 |
+
inference_time_ms: Optional[float] = None
|
| 45 |
+
error: Optional[str] = None
|
| 46 |
+
warning: Optional[str] = None
|
| 47 |
+
|
| 48 |
+
def __post_init__(self):
|
| 49 |
+
if self.prediction not in VALID_PREDICTIONS:
|
| 50 |
+
raise ValueError(
|
| 51 |
+
f"Invalid prediction value '{self.prediction}'. "
|
| 52 |
+
f"Must be strictly one of: {sorted(list(VALID_PREDICTIONS))}"
|
| 53 |
+
)
|
| 54 |
+
|
| 55 |
+
def to_dict(self) -> Dict[str, Any]:
|
| 56 |
+
"""Convert result to a standard serializable dictionary."""
|
| 57 |
+
return asdict(self)
|
| 58 |
+
|
| 59 |
+
@classmethod
|
| 60 |
+
def create_error(
|
| 61 |
+
cls,
|
| 62 |
+
detector_name: str,
|
| 63 |
+
model_name: str,
|
| 64 |
+
error_message: str,
|
| 65 |
+
warning: Optional[str] = None,
|
| 66 |
+
device: Optional[str] = None,
|
| 67 |
+
preprocessing: Optional[str] = None,
|
| 68 |
+
) -> "DetectorResult":
|
| 69 |
+
"""Factory helper to construct a well-formed error result."""
|
| 70 |
+
return cls(
|
| 71 |
+
detector_name=detector_name,
|
| 72 |
+
model_name=model_name,
|
| 73 |
+
prediction=PREDICTION_ERROR,
|
| 74 |
+
error=str(error_message),
|
| 75 |
+
warning=warning,
|
| 76 |
+
device=device,
|
| 77 |
+
preprocessing=preprocessing,
|
| 78 |
+
)
|
src/univfd_adapter.py
ADDED
|
@@ -0,0 +1,231 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
UniversalFakeDetect Adapter Module
|
| 3 |
+
Provides a clean, standardized wrapper around the CVPR 2023 UniversalFakeDetect model.
|
| 4 |
+
Follows the DetectorResult schema and isolates model loading and inference logic.
|
| 5 |
+
"""
|
| 6 |
+
|
| 7 |
+
import os
|
| 8 |
+
import sys
|
| 9 |
+
import time
|
| 10 |
+
from typing import Optional, Union, Any
|
| 11 |
+
from PIL import Image
|
| 12 |
+
import torch
|
| 13 |
+
|
| 14 |
+
from src.detector_schema import (
|
| 15 |
+
DetectorResult,
|
| 16 |
+
PREDICTION_REAL,
|
| 17 |
+
PREDICTION_FAKE,
|
| 18 |
+
PREDICTION_ERROR,
|
| 19 |
+
)
|
| 20 |
+
|
| 21 |
+
# Compatibility shim for Python 3.12+ environments where pkg_resources is deprecated/absent
|
| 22 |
+
try:
|
| 23 |
+
import pkg_resources
|
| 24 |
+
except ImportError:
|
| 25 |
+
try:
|
| 26 |
+
import packaging
|
| 27 |
+
import types
|
| 28 |
+
_pr = types.ModuleType("pkg_resources")
|
| 29 |
+
_pr.packaging = packaging
|
| 30 |
+
sys.modules["pkg_resources"] = _pr
|
| 31 |
+
except Exception:
|
| 32 |
+
pass
|
| 33 |
+
|
| 34 |
+
# Ensure UniversalFakeDetect repository is importable
|
| 35 |
+
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
| 36 |
+
UFD_DIR = os.path.join(BASE_DIR, "UniversalFakeDetect")
|
| 37 |
+
if UFD_DIR not in sys.path:
|
| 38 |
+
sys.path.insert(0, UFD_DIR)
|
| 39 |
+
|
| 40 |
+
|
| 41 |
+
class UnivFDAdapter:
|
| 42 |
+
"""
|
| 43 |
+
Adapter for UniversalFakeDetect (CLIP:ViT-L/14 + Linear Probe).
|
| 44 |
+
|
| 45 |
+
Adheres strictly to the existing pipeline:
|
| 46 |
+
- Architecture: CLIP:ViT-L/14
|
| 47 |
+
- Weights: UniversalFakeDetect/pretrained_weights/fc_weights.pth
|
| 48 |
+
- Preprocessing: Resize(224, bicubic) -> CenterCrop(224) -> ToTensor() -> Normalize(clip)
|
| 49 |
+
- Decision Threshold: 0.50 (logit 0.00)
|
| 50 |
+
- Label Mapping: 0 = REAL, 1 = FAKE
|
| 51 |
+
"""
|
| 52 |
+
|
| 53 |
+
DETECTOR_NAME = "UniversalFakeDetect"
|
| 54 |
+
MODEL_NAME = "CLIP:ViT-L/14 + fc_weights.pth"
|
| 55 |
+
PREPROCESSING_NAME = "Resize(224, bicubic) -> CenterCrop(224) -> ToTensor() -> Normalize(clip)"
|
| 56 |
+
DECISION_THRESHOLD = 0.50
|
| 57 |
+
LIMITATION_WARNING = (
|
| 58 |
+
"Identifies learned patterns associated with synthetic or manipulated images. "
|
| 59 |
+
"Does not guarantee image authenticity."
|
| 60 |
+
)
|
| 61 |
+
|
| 62 |
+
def __init__(self, device: Optional[Union[str, torch.device]] = None, lazy_load: bool = True):
|
| 63 |
+
self.device_requested = device
|
| 64 |
+
self.model = None
|
| 65 |
+
self.preprocess = None
|
| 66 |
+
self.device = None
|
| 67 |
+
self.load_error = None
|
| 68 |
+
self.load_time_sec = None
|
| 69 |
+
|
| 70 |
+
if not lazy_load:
|
| 71 |
+
self.load_model()
|
| 72 |
+
|
| 73 |
+
def load_model(self) -> bool:
|
| 74 |
+
"""Loads model backbone and classifier weights once if not already initialized."""
|
| 75 |
+
if self.model is not None:
|
| 76 |
+
return True
|
| 77 |
+
|
| 78 |
+
start_t = time.perf_counter()
|
| 79 |
+
try:
|
| 80 |
+
from models import get_model
|
| 81 |
+
|
| 82 |
+
# Determine compute device
|
| 83 |
+
if self.device_requested is not None:
|
| 84 |
+
self.device = torch.device(self.device_requested)
|
| 85 |
+
else:
|
| 86 |
+
self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
|
| 87 |
+
|
| 88 |
+
# Initialize CLIP backbone
|
| 89 |
+
model = get_model("CLIP:ViT-L/14")
|
| 90 |
+
|
| 91 |
+
# Load linear probe checkpoint
|
| 92 |
+
ckpt_path = os.path.join(UFD_DIR, "pretrained_weights", "fc_weights.pth")
|
| 93 |
+
if not os.path.exists(ckpt_path):
|
| 94 |
+
raise FileNotFoundError(f"Classifier checkpoint not found at: {ckpt_path}")
|
| 95 |
+
|
| 96 |
+
state_dict = torch.load(ckpt_path, map_location="cpu")
|
| 97 |
+
model.fc.load_state_dict(state_dict)
|
| 98 |
+
|
| 99 |
+
model.to(self.device)
|
| 100 |
+
model.eval()
|
| 101 |
+
|
| 102 |
+
self.model = model
|
| 103 |
+
self.preprocess = model.preprocess
|
| 104 |
+
self.load_time_sec = time.perf_counter() - start_t
|
| 105 |
+
return True
|
| 106 |
+
|
| 107 |
+
except Exception as e:
|
| 108 |
+
self.load_error = str(e)
|
| 109 |
+
self.model = None
|
| 110 |
+
self.preprocess = None
|
| 111 |
+
return False
|
| 112 |
+
|
| 113 |
+
def predict(self, image_input: Any) -> DetectorResult:
|
| 114 |
+
"""
|
| 115 |
+
Execute single-image inference and return a standardized DetectorResult.
|
| 116 |
+
|
| 117 |
+
Accepts:
|
| 118 |
+
- PIL Image
|
| 119 |
+
- File path string
|
| 120 |
+
- Numpy image array (or any object convertible to PIL Image)
|
| 121 |
+
"""
|
| 122 |
+
start_time = time.perf_counter()
|
| 123 |
+
|
| 124 |
+
# 1. Validation for empty input
|
| 125 |
+
if image_input is None:
|
| 126 |
+
return DetectorResult.create_error(
|
| 127 |
+
detector_name=self.DETECTOR_NAME,
|
| 128 |
+
model_name=self.MODEL_NAME,
|
| 129 |
+
error_message="No image provided (input was None).",
|
| 130 |
+
warning=self.LIMITATION_WARNING,
|
| 131 |
+
device=str(self.device) if self.device else None,
|
| 132 |
+
preprocessing=self.PREPROCESSING_NAME,
|
| 133 |
+
)
|
| 134 |
+
|
| 135 |
+
# 2. Safe image loading and RGB conversion
|
| 136 |
+
try:
|
| 137 |
+
if isinstance(image_input, str):
|
| 138 |
+
if not os.path.exists(image_input):
|
| 139 |
+
return DetectorResult.create_error(
|
| 140 |
+
detector_name=self.DETECTOR_NAME,
|
| 141 |
+
model_name=self.MODEL_NAME,
|
| 142 |
+
error_message=f"Image file does not exist: {image_input}",
|
| 143 |
+
warning=self.LIMITATION_WARNING,
|
| 144 |
+
device=str(self.device) if self.device else None,
|
| 145 |
+
preprocessing=self.PREPROCESSING_NAME,
|
| 146 |
+
)
|
| 147 |
+
with Image.open(image_input) as opened_img:
|
| 148 |
+
img_rgb = opened_img.convert("RGB")
|
| 149 |
+
elif isinstance(image_input, Image.Image):
|
| 150 |
+
img_rgb = image_input.convert("RGB")
|
| 151 |
+
elif hasattr(image_input, "__array__"):
|
| 152 |
+
img_rgb = Image.fromarray(image_input).convert("RGB")
|
| 153 |
+
else:
|
| 154 |
+
return DetectorResult.create_error(
|
| 155 |
+
detector_name=self.DETECTOR_NAME,
|
| 156 |
+
model_name=self.MODEL_NAME,
|
| 157 |
+
error_message=f"Unsupported image input type: {type(image_input).__name__}",
|
| 158 |
+
warning=self.LIMITATION_WARNING,
|
| 159 |
+
device=str(self.device) if self.device else None,
|
| 160 |
+
preprocessing=self.PREPROCESSING_NAME,
|
| 161 |
+
)
|
| 162 |
+
except Exception as img_err:
|
| 163 |
+
return DetectorResult.create_error(
|
| 164 |
+
detector_name=self.DETECTOR_NAME,
|
| 165 |
+
model_name=self.MODEL_NAME,
|
| 166 |
+
error_message=f"Invalid or corrupted image: {str(img_err)}",
|
| 167 |
+
warning=self.LIMITATION_WARNING,
|
| 168 |
+
device=str(self.device) if self.device else None,
|
| 169 |
+
preprocessing=self.PREPROCESSING_NAME,
|
| 170 |
+
)
|
| 171 |
+
|
| 172 |
+
# 3. Ensure model is loaded
|
| 173 |
+
if not self.load_model():
|
| 174 |
+
return DetectorResult.create_error(
|
| 175 |
+
detector_name=self.DETECTOR_NAME,
|
| 176 |
+
model_name=self.MODEL_NAME,
|
| 177 |
+
error_message=f"Model loading failed: {self.load_error}",
|
| 178 |
+
warning=self.LIMITATION_WARNING,
|
| 179 |
+
device=str(self.device) if self.device else None,
|
| 180 |
+
preprocessing=self.PREPROCESSING_NAME,
|
| 181 |
+
)
|
| 182 |
+
|
| 183 |
+
# 4. Inference execution under torch.inference_mode()
|
| 184 |
+
try:
|
| 185 |
+
# Check dynamic device placement (ZeroGPU or CPU fallback)
|
| 186 |
+
current_device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
|
| 187 |
+
if next(self.model.parameters()).device != current_device:
|
| 188 |
+
self.model.to(current_device)
|
| 189 |
+
self.device = current_device
|
| 190 |
+
|
| 191 |
+
input_tensor = self.preprocess(img_rgb).unsqueeze(0).to(self.device)
|
| 192 |
+
|
| 193 |
+
with torch.inference_mode():
|
| 194 |
+
features = self.model.model.encode_image(input_tensor)
|
| 195 |
+
logit_tensor = self.model.fc(features.float())
|
| 196 |
+
raw_logit = logit_tensor.item()
|
| 197 |
+
prob_fake = torch.sigmoid(logit_tensor).item()
|
| 198 |
+
|
| 199 |
+
prob_real = 1.0 - prob_fake
|
| 200 |
+
latency_ms = (time.perf_counter() - start_time) * 1000
|
| 201 |
+
|
| 202 |
+
# Output decision and confidence
|
| 203 |
+
is_fake = prob_fake >= self.DECISION_THRESHOLD
|
| 204 |
+
prediction = PREDICTION_FAKE if is_fake else PREDICTION_REAL
|
| 205 |
+
confidence = prob_fake if is_fake else prob_real
|
| 206 |
+
|
| 207 |
+
return DetectorResult(
|
| 208 |
+
detector_name=self.DETECTOR_NAME,
|
| 209 |
+
model_name=self.MODEL_NAME,
|
| 210 |
+
prediction=prediction,
|
| 211 |
+
fake_probability=round(prob_fake, 6),
|
| 212 |
+
real_probability=round(prob_real, 6),
|
| 213 |
+
raw_logit=round(raw_logit, 6),
|
| 214 |
+
confidence=round(confidence, 6),
|
| 215 |
+
threshold=self.DECISION_THRESHOLD,
|
| 216 |
+
device=str(self.device),
|
| 217 |
+
preprocessing=self.PREPROCESSING_NAME,
|
| 218 |
+
inference_time_ms=round(latency_ms, 2),
|
| 219 |
+
error=None,
|
| 220 |
+
warning=self.LIMITATION_WARNING,
|
| 221 |
+
)
|
| 222 |
+
|
| 223 |
+
except Exception as inf_err:
|
| 224 |
+
return DetectorResult.create_error(
|
| 225 |
+
detector_name=self.DETECTOR_NAME,
|
| 226 |
+
model_name=self.MODEL_NAME,
|
| 227 |
+
error_message=f"Inference execution failed: {str(inf_err)}",
|
| 228 |
+
warning=self.LIMITATION_WARNING,
|
| 229 |
+
device=str(self.device) if self.device else None,
|
| 230 |
+
preprocessing=self.PREPROCESSING_NAME,
|
| 231 |
+
)
|