Spaces:
Sleeping
Sleeping
|
Download docs/MULTI_MODEL_EVALUATION_PLAN.md from Thiru1122/Multimodel_setup: direct link, hf CLI and curl.
- Browser
- Download file 6.59 kB
-
https://huggingface.co/spaces/Thiru1122/Multimodel_setup/resolve/main/docs/MULTI_MODEL_EVALUATION_PLAN.md
- Command line
-
hf download hf://spaces/Thiru1122/Multimodel_setup/docs/MULTI_MODEL_EVALUATION_PLAN.md
-
curl -L -o MULTI_MODEL_EVALUATION_PLAN.md https://huggingface.co/spaces/Thiru1122/Multimodel_setup/resolve/main/docs/MULTI_MODEL_EVALUATION_PLAN.md
6.59 kB
| # Multi-Model Evaluation and Comparison Plan | |
| **Target Space**: [`https://huggingface.co/spaces/Thiru1122/Multimodel_setup`](https://huggingface.co/spaces/Thiru1122/Multimodel_setup) | |
| **Status**: Architectural Planning Document (Preparation Phase — No Agreement Layer Implemented Yet) | |
| --- | |
| ## 1. Overview & Objective | |
| No individual deepfake detection architecture generalizes universally across all generative algorithms (GANs, autoregressive transformers, latent diffusion, flow-matching) and manipulation techniques (inpainting, face-swapping, splicing). | |
| The objective of this framework is to establish a rigorous, standardized, multi-model evaluation harness that allows multiple independent detectors to analyze identical input assets under consistent protocols, producing standardized diagnostic outputs and agreement metrics without modifying individual model behaviors. | |
| --- | |
| ## 2. Common Input Format | |
| To ensure parity across all candidate models, all inputs entering the evaluation harness must conform to the following pipeline: | |
| 1. **Primary Input Types**: | |
| - `PIL.Image.Image` object, OR | |
| - Absolute file path string pointing to a local image file. | |
| 2. **Standard Ingress Transformation**: | |
| - Validation against corruption and truncated headers using `PIL.Image.open()`. | |
| - Explicit conversion to 3-channel RGB mode (`.convert("RGB")`). | |
| - Preservation of original image aspect ratio and native resolution metadata prior to model-specific preprocessing. | |
| 3. **Model-Specific Downstream Adapters**: | |
| - Each detector adapter is responsible for applying its own exact author-defined preprocessing (e.g., specific resizing interpolation, cropping dimensions, normalization means and standard deviations). | |
| --- | |
| ## 3. Common Result Schema | |
| All candidate detectors must implement an adapter adhering strictly to the `DetectorResult` schema defined in [`src/detector_schema.py`](file:///c:/Users/user/Desktop/multi%20model%20connection/src/detector_schema.py): | |
| ```python | |
| @dataclass | |
| class DetectorResult: | |
| detector_name: str # Unique identifier of detector (e.g. "UniversalFakeDetect") | |
| model_name: str # Backbone and checkpoint descriptor | |
| prediction: str # Strictly one of: "REAL", "FAKE", "UNCERTAIN", "ERROR" | |
| fake_probability: float # Normalized probability in [0.0, 1.0] that input is synthetic | |
| real_probability: float # Normalized probability in [0.0, 1.0] that input is real | |
| raw_logit: float # Unbounded pre-activation output logit | |
| confidence: float # Confidence score in [0.0, 1.0] | |
| threshold: float # Operational decision boundary (e.g., 0.50) | |
| device: str # Execution hardware ("cuda", "cpu") | |
| preprocessing: str # Summary description of transform pipeline | |
| inference_time_ms: float # Wall-clock execution time in milliseconds | |
| error: Optional[str] # Descriptive error trace if prediction is "ERROR" | |
| warning: Optional[str] # Mandatory authenticity limitation notice | |
| ``` | |
| Allowed prediction values are strictly bounded to: | |
| - `REAL`: Detector classified input as natural photograph. | |
| - `FAKE`: Detector classified input as synthetic / manipulated. | |
| - `UNCERTAIN`: Prediction cannot be reliably assigned (e.g., logit falls in dead-band or model disagreement). | |
| - `ERROR`: Input was invalid, corrupted, or pipeline encountered an unrecoverable exception. | |
| --- | |
| ## 4. Common Error Handling Protocol | |
| 1. **Rejection at Ingress**: | |
| - Corrupted files, non-image formats, and truncated byte streams must produce `prediction: "ERROR"` with a non-null `error` field. | |
| 2. **No Silent Error Fallback**: | |
| - Under no circumstances may an unhandled exception or decode failure default silently to `REAL` or `FAKE`. | |
| 3. **Hardware & OOM Isolation**: | |
| - If an individual detector exhausts GPU memory or times out, its adapter must return an `ERROR` result without crashing the enclosing process or aborting sibling detector evaluations. | |
| --- | |
| ## 5. Per-Model Reporting & Aggregation | |
| When multiple detectors evaluate a single input, the evaluation harness must report: | |
| 1. **Individual Model Records**: | |
| - Tabular display of each detector's raw logit, calibrated fake probability, assigned binary label, and latency. | |
| 2. **Consensus & Disagreement**: | |
| - Quantitative concordance count (e.g., `2/3 detectors predict FAKE`). | |
| - Mean and variance of predicted fake probabilities across models. | |
| 3. **The `UNCERTAIN` State**: | |
| - When constituent models produce conflicting predictions (e.g., Detector A outputs `REAL` with 99% confidence, while Detector B outputs `FAKE` with 95% confidence), the composite verdict must be explicitly designated as **`UNCERTAIN`**. | |
| - The system must not arbitrate conflicting predictions by simple average if the underlying feature distributions are uncalibrated. | |
| --- | |
| ## 6. Rigorous Evaluation Protocol & Benchmark Requirements | |
| Future evaluations across multiple models must adhere to the following experimental standards: | |
| 1. **Curated Ground-Truth Benchmark**: | |
| - Evaluation must be conducted on a balanced, labeled benchmark dataset containing verified real photographs, known GAN images (ProGAN, StyleGAN), modern diffusion images (Midjourney, Stable Diffusion, Imagen, Flux), and localized image manipulations. | |
| 2. **Separate Reporting of Error Types**: | |
| - Metrics must report False-Real (False Negative / Missed AI) and False-Fake (False Positive / Flagged Real) error rates independently. | |
| - Aggregate accuracy alone is misleading on imbalanced or out-of-distribution test sets. | |
| 3. **Calibration Requirements**: | |
| - Raw logits from distinct architectures cannot be compared directly without Platt scaling or temperature calibration on a shared validation set. | |
| --- | |
| ## 7. Fundamental Disclaimer on Model Agreement | |
| > [!IMPORTANT] | |
| > **Consensus $\neq$ Authenticity Verification:** | |
| > If all detectors in a multi-model ensemble classify an image as `REAL`, this indicates only that none of the models detected the specific synthetic patterns for which they were trained. It **does not verify** that the image is an authentic photograph. Generative models developed after a detector's training cutoff may bypass all detectors simultaneously. | |
| --- | |
| ## 8. Implementation Boundary Notice | |
| **Current State**: Only UniversalFakeDetect and its standardized adapter (`UnivFDAdapter`) are active. | |
| **Strict Boundary**: The multi-model agreement layer, additional detector repositories (e.g., RAID), and ensemble fusion logic are **not implemented** in this phase. | |