Spaces:
Running on Zero
Download docs/PHASE6_INDEPENDENT_EVALUATION_PLAN.md from Thiru1122/Multimodel_setup: direct link, hf CLI and curl.
- Browser
- Download file 27.5 kB
-
https://huggingface.co/spaces/Thiru1122/Multimodel_setup/resolve/main/docs/PHASE6_INDEPENDENT_EVALUATION_PLAN.md
- Command line
-
hf download hf://spaces/Thiru1122/Multimodel_setup/docs/PHASE6_INDEPENDENT_EVALUATION_PLAN.md
-
curl -L -o PHASE6_INDEPENDENT_EVALUATION_PLAN.md https://huggingface.co/spaces/Thiru1122/Multimodel_setup/resolve/main/docs/PHASE6_INDEPENDENT_EVALUATION_PLAN.md
A newer version of the Gradio SDK is available: 6.32.0
Phase 6: Independent Cross-Generator Evaluation β Audit and Research Plan
Project Root: C:\Users\user\Desktop\multi model connection
Document: docs/PHASE6_INDEPENDENT_EVALUATION_PLAN.md
Phase: Phase 6 β Independent Cross-Generator Evaluation: Audit and Research Plan
Status: Plan Only (Strictly No Code Modification, No Dataset Downloads, No Package Installation)
Date: October 2026
1. Executive Summary
This document presents a scientifically defensible research and evaluation plan for Phase 6 of the project: Cross-Model Evaluation of Deepfake Image Detection Under Diverse Image Generation and Manipulation Conditions.
1.1 Context and Problem Statement
Phases 1 through 5 of this project audited the repository, resolved a critical Windows virtual memory exhaustion error (os error 1455) via subprocess isolation, curated and verified a 159-sample manifest from local assets, executed a reproducible baseline benchmark on 105 held-out samples, and empirically tested multi-model decision fusion on 50 validation samples.
The core empirical findings from Phases 4 and 5 revealed:
- Facial Domain Specialization: The Vision Transformer (
dima806/deepfake_vs_real_image_detection) achieves 100.00% accuracy ($100 / 100$) on the tested facial crops (Flickr-Faces-HQ authentic portraits vs. StyleGAN2 synthetic faces). - General Scene Blindspots: On modern latent diffusion (Google Imagen 3, $1024 \times 1024$) and localized digital composite splicing ($800 \times 800$), all three integrated detectors failed at default thresholds, misclassifying synthetic and manipulated scenes as
REAL($P_\text{fake} \le 34.24%$). - Failure of Naive Fusion: Democratic decision fusion (majority voting and mean-probability averaging) caused severe performance degradation (dropping accuracy from $100.00%$ to $74.00%$ and $76.00%$ respectively) because face-insensitive secondary models (UnivFD at $64.00%$, ConvNeXt/DRCT at $58.00%$) outvoted ViT on subtle synthetic faces. Only
vit_prioritypreserved ViT's facial accuracy while providing a fault-tolerant fallback chain. - Quarantine Status of Current Data: The existing
held_out_testpartition ($N=105$) has already been observed twice (Phase 4 baseline and Phase 5 frozen verification). It is no longer a blind test set.
1.2 Objective of Phase 6
To escape the narrow domain of FFHQ-versus-StyleGAN facial crops, this plan establishes a methodology to evaluate whether ViTAdapter, UnivFDAdapter, ConvNeXtAdapter, and vit_priority generalize to unseen generative architectures (diffusion faces, diffusion scenes, commercial generators, image splicing, and post-processing degradations) without introducing data leakage, test-set overfitting, or hardware exhaustion.
2. Verified Current Project State
An independent audit of repository code, manifests, and historical logs confirms the following technical baseline:
2.1 Integrated Model Roster & Checkpoint Audit
| Detector Key | Adapter Class | Architecture Family | Checkpoint File / Identifier | Checkpoint Size | Decision Threshold | Input Resolution & Preprocessing | Label Mapping | Training Domain & Source |
|---|---|---|---|---|---|---|---|---|
vit |
ViTAdapter |
Vision Transformer (ViT-Base/16) |
models/deepfake_vs_real_image_detection/model.safetensors |
343.22 MB | 0.50 | $3 \times 224 \times 224$, Bicubic, Mean/Std [0.5, 0.5, 0.5] |
0: Real, 1: Fake | Fine-tuned on Kaggle 140k cropped faces |
univfd |
UnivFDAdapter |
Vision-Language Probe (CLIP ViT-L/14) |
UniversalFakeDetect/pretrained_weights/fc_weights.pth |
4.08 KB (probe) + ~1.7 GB (CLIP) | 0.50 | $3 \times 224 \times 224$, Bicubic, CenterCrop, OpenAI CLIP Mean/Std | 0: Real, 1: Fake (Sigmoid on single logit) | Linear probe trained on 2018 ProGAN artifacts |
chen2024_convnext |
ConvNeXtDetectorAdapter |
ConvNeXt-Base | models/chen2024_convnext/16_acc0.9993.pth |
354.34 MB | 0.50 | $3 \times 224 \times 224$, ImageNet Mean/Std | 0: Real, 1: Fake | Trained on DRCT-2M with Stable Diffusion v2 reconstruction contrastive loss |
Verification Source: Checked via src/detector_registry.py, src/vit_adapter.py, src/univfd_adapter.py, and src/convnext_adapter.py.
2.2 Manifest Inventory & Sample Usage History
The active manifest benchmark/evaluation_manifest.csv contains 159 verified rows, partitioned as follows:
| Partition Name | Total Samples | Real | Fake | Control | Visual Domain | Generators Represented | Prior Usage History | Quarantine Status |
|---|---|---|---|---|---|---|---|---|
held_out_test |
105 | 51 | 52 | 2 | 100 Facial Crops 3 General Scenes 2 Corrupted/Format |
FFHQ (50) StyleGAN2 (50) Smartphone Photo (1) Imagen 3 Diffusion (1) Spliced Composite (1) |
Evaluated in Phase 4 (baseline) and Phase 5 (vit_priority) |
EXHAUSTED AS BLIND TEST; results already observed. |
development_val |
50 | 25 | 25 | 0 | 50 Facial Crops | FFHQ (25) StyleGAN2 (25) |
Evaluated in Phase 5 to compare 4 fusion strategies | EXHAUSTED AS VALIDATION; used to freeze fusion rules. |
operational_smoke |
4 | 1 | 2 | 1 | 3 General Scenes 1 Corrupted |
Natural Photo (1) Diffusion (1) Composite (1) |
CI contract & adapter regression checks | RESERVED strictly for smoke tests; excluded from metrics. |
Historical Exploratory Samples (in currently working): An additional 2,530 samples were touched in past experiments (Group A 0..49, Calibration 400..499, Holdout 500..699, random_250, extended_predictions). All 150 facial samples currently in the manifest originate from the pristine index range ($1500 \le \text{index} \le 1574$) to avoid overlap with those historical subsets.
2.3 Sample-Level Metadata & Provenance Audit
- Available Fields:
sample_id,split,domain,category,ground_truth,source_dataset,generation_or_manipulation_method,label_evidence,verification_status,file_path,relative_path,file_format,width,height,file_size_bytes,sha256. - Missing Sample-Level Metadata:
- Subject Identity: Not recorded for FFHQ faces (anonymous Flickr crawls).
- Generation Parameters: Prompts, seeds, guidance scales, and latent noise vectors are unrecorded for the StyleGAN and Imagen 3 samples.
- Camera Hardware EXIF: Stripped from all $256 \times 256$ facial crops during upstream Kaggle dataset packaging.
- Identity & Near-Duplicate Leakage Assessment:
- Byte-level deduplication confirmed zero exact duplicate hashes between splits.
- However, because the upstream Kaggle 140k dataset was partitioned randomly rather than clustered by facial identity embeddings, latent-space identity overlap or near-duplicate poses between FFHQ/StyleGAN faces cannot be mathematically ruled out in the existing manifest.
3. Limitations of the Existing Evaluation
The current evaluation framework suffers from four fundamental scientific limitations that Phase 6 must address:
flowchart TD
subgraph CurrentLimitations["Existing Evaluation Limitations"]
L1["1. Single-Generator Bias<br>100% of facial fakes originate from StyleGAN2.<br>Detectors may learn StyleGAN-specific artifacts rather than generalized synthesis."]
L2["2. Face-Scene Confound<br>Facial images are 256x256 crops.<br>General scenes are uncropped (1024x1024, 4000x3000).<br>Domain and resolution are entangled."]
L3["3. N=1 Scene Diagnostics<br>Only 1 Imagen 3 image, 1 spliced image, 1 photo.<br>Zero statistical power for scene detection (Wilson CI: [6.1%, 79.2%])."]
L4["4. Unblinded Test Set<br>held_out_test was observed in Phase 4 and Phase 5.<br>Iterating further on it risks unconscious test-set overfitting."]
end
Detailed Limitations Breakdown:
- Single-Generator Facial Monoculture: Every synthetic face in
held_out_test($N=50$) anddevelopment_val($N=25$) was synthesized by NVIDIA StyleGAN2. Zero samples represent modern latent diffusion faces (Stable Diffusion XL portraits, Midjourney portraits, Flux.1 faces), identity swapping (DeepFaceLab, SimSwap), or face morphing. - Domain-Resolution Entanglement: In the existing benchmark, all facial images are $256 \times 256$ pixels, while general scenes are $800 \times 800$, $1024 \times 1024$, or $4000 \times 3000$ pixels. A detector's failure on general scenes could stem from resolution downsampling, lack of facial alignment, or generative model differences, without the ability to isolate the root cause.
- Statistically Meaningless Scene Sample Size: With exactly $N=1$ sample per scene category, calculated scene accuracy ($33.33%$) carries an extreme 95% Wilson confidence interval of $[6.15%, 79.23%]$. It serves as a qualitative canary test, not a valid benchmark metric.
- Absence of Post-Processing & Perturbations: All current test images are uncompressed or high-quality originals. In-the-wild deepfakes undergo social media recompression (JPEG quality 30β75, WebP, WhatsApp downsampling), motion blur, and color grading.
4. External Dataset Comparison Table
To address these limitations, public deepfake and synthetic image datasets were surveyed against verified academic publications and official repositories:
| Dataset Name & Official Source | Image Modalities | Generative & Manipulation Methods | Scale & Label Structure | Licensing & Access Terms | Known Provenance & Leakage Considerations | Domain Novelty vs. Current Project | Hardware Compatibility (8GB RAM / 4GB VRAM) |
|---|---|---|---|---|---|---|---|
GenImagegithub.com/GenImage-Dataset/GenImageNeurIPS 2023 |
Real + Synthetic | 8 Generators: Stable Diffusion (v1.4, v1.5), Midjourney, ADM, GLIDE, Wukong, VQDM, BigGAN | 1,331,167 images (899k train, 432k val/test). 1:1 Real/Fake balance per generator. | CC BY-NC-SA 4.0 (Open academic research use; non-commercial). Direct download. | Real images sourced from ImageNet-1K; potential overlap with ConvNeXt pretraining backbone. | HIGH. Covers 7 unseen diffusion architectures + BigGAN across general visual scenes. | EXCELLENT. Standard JPEG/PNG format, evaluates in small sequential batches. |
DiffusionForensics (DIRE)github.com/ZhendongWang6/DIREICCV 2023 |
Real + Synthetic | Diffusion Models: ADM, DDPM, iDDPM, PNDM, LDM | ~100,000 images across LSUN-Bedroom, ImageNet, and CelebA-HQ | Open academic research use (Hosted on Baidu/RecDrive; code under MIT). | Real faces from CelebA-HQ; potential identity overlap with FFHQ subjects crawled from Flickr. | VERY HIGH. Provides diffusion-generated human faces (CelebA-HQ), directly challenging ViT's facial generalization. | GOOD. $256 \times 256$ and $512 \times 512$ resolutions load easily in memory. |
FaceForensics++ (FF++)github.com/ondyari/FaceForensicsICCV 2019 |
Real + Manipulated | 5 Facial Manipulations: Deepfakes, Face2Face, FaceSwap, NeuralTextures, FaceShifter | 1,000 original YouTube video sequences + manipulated counterparts; 3 compression levels (c0, c23, c40). | FaceForensics Terms of Use (Requires signed application form; non-commercial). | YouTube video origins; real identities are public video creators. | VERY HIGH. Introduces face-swapping and facial reenactment, completely absent from current data. | MODERATE. Video files require frame extraction ($224 \times 224$ crops) prior to sequential inference. |
Celeb-DF (v2)github.com/yuezunli/celeb-deepfakeforensicsCVPR 2020 |
Real + Manipulated | DeepFaceLab Face-Swapping (High-quality identity replacement with temporal blending) | 590 real celebrity YouTube videos + 5,639 high-grade DeepFake videos (~2M frames). | Celeb-DF Terms of Use (Application form required; research-only). | Celebrity subjects (actors, politicians); zero overlap with FFHQ Flickr users. | HIGH. Evaluates high-fidelity face-swapping designed to defeat edge artifacts. | MODERATE. Requires frame extraction; single-frame evaluation is lightweight. |
CASIA v2.0forensics.idealtest.orgChinaSIP 2013 |
Real + Manipulated | Digital Splicing & Copy-Move (Local spatial pasting, boundary post-processing) | 12,615 images (7,491 authentic, 5,124 tampered). JPEG, BMP, TIFF formats. | Academic non-commercial research use only; registration required. | Sourced from Corel dataset and personal digital photos. Zero generative AI; pure manual digital manipulation. | HIGH. Expands the $N=1$ composite probe into a statistically valid image splicing benchmark. | EXCELLENT. Variable resolutions ($384 \times 256$ to $800 \times 600$) easily handled on 4GB VRAM. |
RAISEmmlab.science.unitn.it/RAISEACM MMSys 2015 |
Authentic Real Only | Camera Sensor Hardware Capture (Nikon D90, D7000, D40 uncompressed RAW/TIFF) | 8,156 uncompressed RAW images (Subsets: RAISE-1k, RAISE-2k) with full EXIF camera parameters. | Open research use. Direct download from University of Trento. | Strictly authentic photographic capture with verified hardware provenance. | HIGH. Provides pristine camera hardware ground truth to rigorously test detector False Positive Rates. | GOOD. RAW files require conversion to uncompressed PNG/JPEG for adapter ingestion. |
AIGCDetectBenchmarkgithub.com/Ekko-zn/AIGCDetectBenchmarkCVPR 2024 |
Real + Synthetic | Multi-Generator Cross-Domain: ProGAN, BigGAN, StyleGAN, SDv1.4, SDv1.5, SDXL, Midjourney, DALL-E 2 | Standardized benchmark splits with diverse generators. | Open academic research use (Hugging Face mirror available). | Combines multiple upstream sources; requires per-source tracking. | VERY HIGH. Pre-aggregated cross-generator benchmark specifically created for detector comparison. | EXCELLENT. Standardized image dimensions matching CLIP/ViT inputs. |
5. Recommended Dataset Shortlist and Rationale
To maintain experimental tractability and prevent hardware exhaustion on the local workstation (8 GB RAM, RTX 3050 4 GB VRAM), a curated, balanced shortlist of three complementary datasets is recommended:
Proposed Independent External Benchmark (Target: 300 New Samples)
βββ Partition 1: Unseen Generative Faces (100 samples)
β βββ 50 Authentic Real Faces (CelebA-HQ / Celeb-DF Real)
β βββ 50 Diffusion Faces (DiffusionForensics CelebA-HQ: LDM & DDPM)
β
βββ Partition 2: Unseen Facial Manipulations (100 samples)
β βββ 50 Authentic Video Frames (FaceForensics++ Real c23)
β βββ 50 Face-Swapped Frames (FaceForensics++ Deepfakes & FaceShifter c23)
β
βββ Partition 3: Unseen General Scenes & Splicing (100 samples)
βββ 25 Authentic Camera Photographs (RAISE-1k uncompressed)
βββ 25 Authentic Scene Photos (ImageNet / MSCOCO)
βββ 25 Modern Diffusion Scenes (GenImage: Midjourney & SDXL)
βββ 25 Spliced / Manipulated Scenes (CASIA v2.0 splicing)
5.1 Shortlist Selection Rationale
- DiffusionForensics (CelebA-HQ Partition):
- Rationale: Isolates the Face vs. Generator confound. In Phase 4, ViT scored 100% on StyleGAN faces but failed on Imagen 3 scene diffusion. Testing ViT on diffusion-generated human faces will resolve whether ViT's strength is human facial geometry or StyleGAN-specific frequency artifacts.
- FaceForensics++ (c23 High-Quality Partition):
- Rationale: Tests identity-swapping and boundary manipulation. The current project has zero face-swap samples. Testing on FaceForensics++ c23 (compressed) reflects real-world deepfake distribution where high-frequency GAN artifacts are partially smoothed by H.264 video compression.
- GenImage (Midjourney & SDXL Subsets) + CASIA v2.0:
- Rationale: Solves the $N=1$ limitation of
test_scene_diffusion_001andtest_scene_manip_001. Provides statistically powered evaluation of modern diffusion architectures (Midjourney, SDXL) and digital splicing, testing whether ConvNeXt/DRCT or UnivFD outperform ViT on non-facial visual scenes.
- Rationale: Solves the $N=1$ limitation of
6. Proposed Split and Leakage-Prevention Protocol
To maintain scientific integrity and prevent unblinded optimization, Phase 6 must adhere to a strict three-tier data quarantine architecture:
flowchart TD
subgraph DataQuarantine["Three-Tier Data Quarantine Architecture"]
Tier1["Tier 1: Historical Data (FROZEN)<br>β’ development_val (50 samples)<br>β’ held_out_test (105 samples)<br>STATUS: ARCHIVED. Zero re-tuning permitted."]
Tier2["Tier 2: External Discovery & Verification<br>β’ Pre-registration of candidate URLs & licenses<br>β’ SHA-256 manifest registration<br>β’ Identity & duplicate filtering<br>STATUS: Read-only inspection."]
Tier3["Tier 3: Independent Evaluation Split (BLIND)<br>β’ 300 curated external samples<br>β’ 1:1 Class balance per category<br>β’ Evaluated ONCE under frozen code<br>STATUS: Zero parameter adjustment."]
end
Tier1 -.->|Rules Frozen| Tier2
Tier2 -->|Single-Pass Evaluation| Tier3
6.1 Leakage-Prevention Rules:
- Pre-Evaluation Code Freezing:
- Detector adapters (
src/vit_adapter.py,src/univfd_adapter.py,src/convnext_adapter.py) must be locked. - Decision thresholds (fixed at $0.50$) and preprocessing pipelines must NOT be adjusted based on external data previews.
- The fusion strategy (
vit_priority) selected in Phase 5 must be evaluated without modification.
- Detector adapters (
- Backbone Training Set Overlap Screening:
ConvNeXt/DRCTwas trained on MSCOCO and Stable Diffusion v1.4 reconstructions. External real images should prioritize sources outside MSCOCO (e.g., RAISE, CelebA-HQ).UnivFDwas trained on OpenAI CLIP ViT-L/14, which saw 400M web images. Real-image source attribution must be documented to acknowledge potential web pretraining overlap.ViTwas trained on Kaggle 140k (FFHQ + StyleGAN2). External faces from Celeb-DF and CelebA-HQ must be verified for identity separation.
- Near-Duplicate & Perceptual Hash Filtering:
- All newly ingested candidate images must undergo both SHA-256 exact matching and perceptual hashing (pHash) against all 190,335 images in
currently working/Datasetandbenchmark/images. - Any sample with Hamming distance $\le 4$ against existing training/test samples must be excluded.
- All newly ingested candidate images must undergo both SHA-256 exact matching and perceptual hashing (pHash) against all 190,335 images in
- Identity-Level Separation for Faces:
- Multiple video frames from the same video clip or identity in FaceForensics++ / Celeb-DF must NOT be split across evaluation groups. All frames from a single video identity must reside in the same evaluation batch.
7. Metrics and Statistical Analysis Plan
To prevent misleading claims arising from small subgroups or imbalanced categories, the evaluation protocol defines exact statistical requirements:
7.1 Quantitative Classification Metrics
For each detector and the frozen fusion engine, report:
- Sample Counts: Total evaluated ($N$), successfully evaluated, inference errors.
- Confusion Matrix: True Positives (TP), False Positives (FP), True Negatives (TN), False Negatives (FN).
- Core Metrics: $$\text{Accuracy} = \frac{\text{TP} + \text{TN}}{\text{TP} + \text{TN} + \text{FP} + \text{FN}}$$ $$\text{Balanced Accuracy} = \frac{1}{2} \left( \frac{\text{TP}}{\text{TP} + \text{FN}} + \frac{\text{TN}}{\text{TN} + \text{FP}} \right)$$ $$\text{Precision} = \frac{\text{TP}}{\text{TP} + \text{FP}}, \quad \text{Recall} = \frac{\text{TP}}{\text{TP} + \text{FN}}, \quad \text{Specificity} = \frac{\text{TN}}{\text{TN} + \text{FP}}$$ $$F_1 = \frac{2 \cdot \text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}}$$
7.2 Confidence Intervals and Sample Size Rules
- All classification metrics must include Wilson Score 95% Confidence Intervals: $$w = \frac{p + \frac{z^2}{2n} \pm z \sqrt{\frac{p(1-p)}{n} + \frac{z^2}{4n^2}}}{1 + \frac{z^2}{n}}$$ where $z = 1.96$ for a $95%$ confidence level.
- Reporting Threshold: Aggregate category metrics are permitted ONLY for subcategories with $N \ge 25$ samples. Categories with $N < 25$ must be reported as individual diagnostic probe listings.
7.3 Multi-Level Domain Stratification
Results must be partitioned and reported along three orthogonal dimensions:
- By Visual Domain:
Facial Portraits(aligned crops) vs.General Scenes(uncropped scenes, landscapes, objects).
- By Manipulation Family:
GAN Synthesis(StyleGAN, ProGAN, BigGAN).Diffusion Synthesis(Stable Diffusion, Midjourney, Imagen, LDM).Face Swapping(DeepFaceLab, FaceShifter).Digital Splicing(CASIA copy-move, composite paste).Authentic Photography(RAISE, ImageNet, CelebA-HQ).
- By Robustness / Perturbation Level:
Uncompressed / Rawvs.Compressed (c23 / JPEG Q=50)vs.Downsampled.
8. Hardware and Runtime Considerations
The evaluation execution must run safely on the local workstation:
- Operating System: Windows 11 Home (Build 26200).
- Host CPU: AMD Ryzen 5 5600H (6 cores, 12 threads).
- System Memory: 8.0 GB Physical RAM.
- GPU: NVIDIA GeForce RTX 3050 Laptop GPU (4.0 GB VRAM).
- Python Environment: Python 3.13.13, PyTorch 2.13.0+cpu, timm 1.0.29, transformers 5.17.0.
8.1 Memory-Safety Protocols:
- Subprocess Isolation: Each detector model must execute in a separate Python subprocess, exactly as implemented in
scripts/run_phase4_baseline_evaluation.py. The child process loads the model, evaluates its batch, flushes results to disk, and terminates, guaranteeing $100%$ virtual memory release back to Windows (os error 1455mitigation). - Sequential Queueing: Detectors must execute strictly in sequence: $$\text{Run ConvNeXt/DRCT} \longrightarrow \text{Process Exits} \longrightarrow \text{Run UnivFD} \longrightarrow \text{Process Exits} \longrightarrow \text{Run ViT} \longrightarrow \text{Process Exits}$$ Never load two models simultaneously into the same process space.
- Batch Sizing & Input Streaming: Images must be loaded as individual PIL image handles and closed immediately after preprocessing, avoiding large in-memory image tensor arrays.
- Expected Execution Latency:
- Based on Phase 4 benchmarks (315 inferences in 345.9s $\approx 1.1\text{s/sample}$ across all 3 models):
- A 300-sample external benchmark ($3 \times 300 = 900$ inferences) will execute in approximately 10 to 12 minutes on CPU without risk of memory overflow.
9. Risks, Limitations, and Unresolved Questions
9.1 Technical & Methodological Risks:
- Web Pretraining Contamination: CLIP (
UnivFD) was pretrained on 400M internet image-text pairs; ConvNeXt was pretrained on ImageNet-22k. Completely guaranteeing that an authentic external photograph was never seen during foundational pretraining is impossible for modern foundation backbones. - Access & Licensing Barriers: Datasets such as FaceForensics++ and Celeb-DF require institutional verification and signed request forms. If academic approval is delayed, alternative open-license benchmarks (e.g. GenImage on Hugging Face, RAISE direct download) must be prioritized.
- Upstream Alignment Mismatch:
ViTAdapterexpects pre-cropped facial bounding boxes. Evaluating full video frames from FaceForensics++ without an upstream face detection crop will cause resolution distortion. A standardized MTCNN or RetinaFace crop stage must be clearly documented if full frames are ingested.
9.2 Explicit Assumptions vs. Verified Facts:
| Item | Fact, Assumption, or Unresolved Question | Supporting Evidence / Source |
|---|---|---|
| ViT Face Specialization | VERIFIED FACT | 100% accuracy on 100 held-out facial samples (phase4_baseline_metrics.json). |
| Diffusion Scene Blindspot | VERIFIED FACT | All 3 models scored $P(\text{fake}) \le 34.24%$ on Imagen 3 diffusion scene (phase4_baseline_predictions.csv). |
| ViT Generalization to Diffusion Faces | UNRESOLVED QUESTION | Unknown whether ViT detects diffusion artifacts in faces or only StyleGAN texture artifacts. Requires Phase 6 testing. |
| FaceForensics++ Frame Extraction | ASSUMPTION | Assumed that extracting cropped face frames at $256 \times 256$ maintains compatibility with ViTAdapter input expectations. |
| Zero Prior Contamination of RAISE | VERIFIED FACT | RAISE images are raw Nikon camera sensor files never included in Kaggle 140k or DRCT-2M training sets. |
10. Staged Implementation Plan with Explicit Approval Gates
Implementation of Phase 6 must proceed through four strictly gated stages:
flowchart TD
G0["Current State: Audit & Research Plan<br>(docs/PHASE6_INDEPENDENT_EVALUATION_PLAN.md)"] --> Gate1{"APPROVAL GATE 1:<br>Approve Candidate Datasets & Scope"}
Gate1 --> Stage1["Stage 1: Sample Acquisition & Curation<br>β’ Acquire 300 curated samples (GenImage, DiffusionForensics, RAISE)<br>β’ Compute SHA-256 & pHash deduplication<br>β’ Build benchmark/external_manifest.csv"]
Stage1 --> Gate2{"APPROVAL GATE 2:<br>Approve External Manifest & Deduplication"}
Gate2 --> Stage2["Stage 2: Sequential Execution<br>β’ Execute scripts/evaluate_phase6_external.py<br>β’ Sequential subprocess isolation across ConvNeXt, UnivFD, ViT<br>β’ Apply frozen vit_priority fusion strategy"]
Stage2 --> Gate3{"APPROVAL GATE 3:<br>Approve Raw Predictions & Integrity"}
Gate3 --> Stage3["Stage 3: Statistical Synthesis & Reporting<br>β’ Compute domain-stratified metrics & Wilson 95% CIs<br>β’ Author docs/PHASE6_INDEPENDENT_EVALUATION_REPORT.md<br>β’ Deliver final cross-generator generalization assessment"]
Stage Summary:
- Gate 1 (Current Stop Point): User reviews and approves the candidate dataset selection, licensing terms, and sample sizing.
- Stage 1 (Acquisition & Manifesting): Download only the approved samples, execute perceptual hash deduplication, and compile
benchmark/external_manifest.csv. No inference. - Gate 2: User verifies manifest integrity, balance, and lack of training set overlap.
- Stage 2 (Inference Execution): Run sequential, subprocess-isolated evaluation across the 3 detectors and frozen
vit_priorityengine. Export raw logs tobenchmark/results/phase6_external_predictions.csv. - Gate 3: User inspects error logs and valid prediction counts.
- Stage 3 (Reporting): Synthesize metrics, calculate Wilson confidence intervals, and author
docs/PHASE6_INDEPENDENT_EVALUATION_REPORT.md.
Report completed in accordance with Phase 6 Strict Operating Rules. All existing code, weights, manifests, and Phase 4/5 deliverables remain unmodified. Execution is stopped awaiting user review and approval.