Multimodel_setup / docs /MULTI_MODEL_EVALUATION_PLAN.md
Thiru1122's picture
Add standardized detector adapter and evaluation documentation
6f9e017
|
Raw History Blame Contribute Delete
6.59 kB
# Multi-Model Evaluation and Comparison Plan
**Target Space**: [`https://huggingface.co/spaces/Thiru1122/Multimodel_setup`](https://huggingface.co/spaces/Thiru1122/Multimodel_setup)
**Status**: Architectural Planning Document (Preparation Phase — No Agreement Layer Implemented Yet)
---
## 1. Overview & Objective
No individual deepfake detection architecture generalizes universally across all generative algorithms (GANs, autoregressive transformers, latent diffusion, flow-matching) and manipulation techniques (inpainting, face-swapping, splicing).
The objective of this framework is to establish a rigorous, standardized, multi-model evaluation harness that allows multiple independent detectors to analyze identical input assets under consistent protocols, producing standardized diagnostic outputs and agreement metrics without modifying individual model behaviors.
---
## 2. Common Input Format
To ensure parity across all candidate models, all inputs entering the evaluation harness must conform to the following pipeline:
1. **Primary Input Types**:
- `PIL.Image.Image` object, OR
- Absolute file path string pointing to a local image file.
2. **Standard Ingress Transformation**:
- Validation against corruption and truncated headers using `PIL.Image.open()`.
- Explicit conversion to 3-channel RGB mode (`.convert("RGB")`).
- Preservation of original image aspect ratio and native resolution metadata prior to model-specific preprocessing.
3. **Model-Specific Downstream Adapters**:
- Each detector adapter is responsible for applying its own exact author-defined preprocessing (e.g., specific resizing interpolation, cropping dimensions, normalization means and standard deviations).
---
## 3. Common Result Schema
All candidate detectors must implement an adapter adhering strictly to the `DetectorResult` schema defined in [`src/detector_schema.py`](file:///c:/Users/user/Desktop/multi%20model%20connection/src/detector_schema.py):
```python
@dataclass
class DetectorResult:
detector_name: str # Unique identifier of detector (e.g. "UniversalFakeDetect")
model_name: str # Backbone and checkpoint descriptor
prediction: str # Strictly one of: "REAL", "FAKE", "UNCERTAIN", "ERROR"
fake_probability: float # Normalized probability in [0.0, 1.0] that input is synthetic
real_probability: float # Normalized probability in [0.0, 1.0] that input is real
raw_logit: float # Unbounded pre-activation output logit
confidence: float # Confidence score in [0.0, 1.0]
threshold: float # Operational decision boundary (e.g., 0.50)
device: str # Execution hardware ("cuda", "cpu")
preprocessing: str # Summary description of transform pipeline
inference_time_ms: float # Wall-clock execution time in milliseconds
error: Optional[str] # Descriptive error trace if prediction is "ERROR"
warning: Optional[str] # Mandatory authenticity limitation notice
```
Allowed prediction values are strictly bounded to:
- `REAL`: Detector classified input as natural photograph.
- `FAKE`: Detector classified input as synthetic / manipulated.
- `UNCERTAIN`: Prediction cannot be reliably assigned (e.g., logit falls in dead-band or model disagreement).
- `ERROR`: Input was invalid, corrupted, or pipeline encountered an unrecoverable exception.
---
## 4. Common Error Handling Protocol
1. **Rejection at Ingress**:
- Corrupted files, non-image formats, and truncated byte streams must produce `prediction: "ERROR"` with a non-null `error` field.
2. **No Silent Error Fallback**:
- Under no circumstances may an unhandled exception or decode failure default silently to `REAL` or `FAKE`.
3. **Hardware & OOM Isolation**:
- If an individual detector exhausts GPU memory or times out, its adapter must return an `ERROR` result without crashing the enclosing process or aborting sibling detector evaluations.
---
## 5. Per-Model Reporting & Aggregation
When multiple detectors evaluate a single input, the evaluation harness must report:
1. **Individual Model Records**:
- Tabular display of each detector's raw logit, calibrated fake probability, assigned binary label, and latency.
2. **Consensus & Disagreement**:
- Quantitative concordance count (e.g., `2/3 detectors predict FAKE`).
- Mean and variance of predicted fake probabilities across models.
3. **The `UNCERTAIN` State**:
- When constituent models produce conflicting predictions (e.g., Detector A outputs `REAL` with 99% confidence, while Detector B outputs `FAKE` with 95% confidence), the composite verdict must be explicitly designated as **`UNCERTAIN`**.
- The system must not arbitrate conflicting predictions by simple average if the underlying feature distributions are uncalibrated.
---
## 6. Rigorous Evaluation Protocol & Benchmark Requirements
Future evaluations across multiple models must adhere to the following experimental standards:
1. **Curated Ground-Truth Benchmark**:
- Evaluation must be conducted on a balanced, labeled benchmark dataset containing verified real photographs, known GAN images (ProGAN, StyleGAN), modern diffusion images (Midjourney, Stable Diffusion, Imagen, Flux), and localized image manipulations.
2. **Separate Reporting of Error Types**:
- Metrics must report False-Real (False Negative / Missed AI) and False-Fake (False Positive / Flagged Real) error rates independently.
- Aggregate accuracy alone is misleading on imbalanced or out-of-distribution test sets.
3. **Calibration Requirements**:
- Raw logits from distinct architectures cannot be compared directly without Platt scaling or temperature calibration on a shared validation set.
---
## 7. Fundamental Disclaimer on Model Agreement
> [!IMPORTANT]
> **Consensus $\neq$ Authenticity Verification:**
> If all detectors in a multi-model ensemble classify an image as `REAL`, this indicates only that none of the models detected the specific synthetic patterns for which they were trained. It **does not verify** that the image is an authentic photograph. Generative models developed after a detector's training cutoff may bypass all detectors simultaneously.
---
## 8. Implementation Boundary Notice
**Current State**: Only UniversalFakeDetect and its standardized adapter (`UnivFDAdapter`) are active.
**Strict Boundary**: The multi-model agreement layer, additional detector repositories (e.g., RAID), and ensemble fusion logic are **not implemented** in this phase.