Spaces:
Running on Zero
Running on Zero
File size: 11,344 Bytes
151f388 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 | # Phase 7 — External AI-Generated Image Generalization Evaluation
## Stage 1 Audit & Proposed Dataset Plan
**Document Path:** [`docs/PHASE7_DATASET_AUDIT.md`](file:///c:/Users/user/Desktop/multi%20model%20connection/docs/PHASE7_DATASET_AUDIT.md)
**Date:** November 2026
**Status:** Stage 1 Audit Completed — Pending User Review & Approval
**Operating Mode:** Read-Only Audit (Zero model weights modified, zero inferences initiated, zero downloads started)
---
## 1. Executive Summary
This audit establishes the empirical foundation for **Phase 7: External AI-Generated Image Generalization Evaluation**.
In Phase 6, our evaluation benchmark focused on **facial portraits** (FFHQ real faces vs. StyleGAN synthetic faces), where the facial ViT detector achieved 98.00% accuracy while ConvNeXt and UnivFD achieved ~51% accuracy (due to feature blindness to StyleGAN manifold structures).
Phase 7 investigates whether our pretrained detectors can distinguish authentic photographs from AI-generated images produced by modern generator architectures (e.g., latent diffusion, pixel diffusion, and large-scale GANs) across **general object and scene categories**.
In strict accordance with the **Mandatory Stage 1** directive, no files have been downloaded and no inference runs have been triggered. This report audits model training provenance, assesses authoritative dataset sources and host storage constraints, verifies class distributions, and outlines a rigorous plan for Stage 2.
---
## 2. Model Training Provenance & Generator Family Overlap
We audited the training lineages of all three integrated detectors against source checkpoints, configuration files, and published literature:
```
+-------------------------------------------------------------------------------------------------------------------------+
| DETECTOR TRAINING PROVENANCE & DOMAIN MAP |
+--------------------------+------------------------------+------------------------------+--------------------------------+
| Attribute | ViT (dima806) | UnivFD (Ojha et al., 2023) | ConvNeXt / DRCT (Chen et al.) |
+--------------------------+------------------------------+------------------------------+--------------------------------+
| Architecture | ViT-Base/16 (85.8M params) | CLIP ViT-L/14 + Linear Probe | ConvNeXt-Base + Linear Head |
| Foundation Pretraining | ImageNet-21k (Google, 2020) | WebImageText 400M (CLIP 2020)| ImageNet-22k (timm convnext_b) |
| Fine-Tuning Task | Facial Real vs Fake | Cross-domain Synthetic Det. | Contrastive Diffusion Rec. |
| Real Images in Training | FFHQ (Flickr human faces) | LSUN (20 object categories) | MS-COCO train2017 (objects) |
| Fake Images in Training | StyleGAN / StyleGAN2 faces | ProGAN (20 LSUN categories) | SD v1.4, SD-Inpaint, SD v2 rec |
| Checkpoint Path | models/.../model.safetensors | UniversalFakeDetect/... | models/.../16_acc0.9993.pth |
| Input Dimensions | 3 x 224 x 224 | 3 x 224 x 224 | 3 x 224 x 224 |
| Seen: StyleGAN? | YES (In-distribution) | NO | NO |
| Seen: ProGAN? | NO | YES (In-distribution) | NO |
| Seen: Stable Diffusion? | NO | NO | YES (v1.4 & v2 reconstruction) |
| Seen: Midjourney? | NO (Unseen) | NO (Unseen) | NO (Unseen) |
| Seen: ADM? | NO (Unseen) | NO (Unseen) | NO (Unseen) |
| Seen: BigGAN? | NO (Unseen) | NO (Unseen in probe) | NO (Unseen in training) |
+--------------------------+------------------------------+------------------------------+--------------------------------+
```
### Detailed Lineage Findings:
1. **ViT (`dima806/deepfake_vs_real_image_detection`):**
- Fine-tuned exclusively on Kaggle 140k faces (FFHQ vs. StyleGAN2).
- **Has never seen diffusion models or general-scene AI generators.**
- In general-scene domains, ViT is operating completely out-of-distribution.
2. **UnivFD (`CLIP ViT-L/14` + Linear Probe):**
- Linear probe trained exclusively on ProGAN across 20 LSUN scene categories.
- Backbone visual encoder (OpenAI CLIP ViT-L/14) was trained on 400M web pairs collected prior to mid-2020.
- **Has never seen diffusion models (Midjourney, ADM) or StyleGAN during linear probe training.**
3. **ConvNeXt / DRCT (`16_acc0.9993.pth`):**
- Trained contrastively on MS-COCO train2017 real photographs vs. **Stable Diffusion v1.4, SD Inpainting, and SD v2 reconstruction**.
- **Stable Diffusion is in-distribution for ConvNeXt.**
- **Has never seen Midjourney, ADM, or BigGAN in its training data.**
- In published DRCT benchmarks ([`temp_DRCT_reference/README.md`](file:///c:/Users/user/Desktop/multi%20model%20connection/temp_DRCT_reference/README.md)), Chen et al. demonstrated that ConvNeXt achieved high accuracy on unseen diffusion models but dropped to 59.81% on BigGAN.
---
## 3. Host Storage, Dependencies & External Dataset Sources
### 3.1 Host Environment
- **Operating System:** Windows 10/11 Enterprise (64-bit AMD64).
- **Available Storage:** **92.46 GB free** on drive `C:`.
- **Installed Dependencies:** `torch` (2.13.0+cpu), `torchvision`, `transformers` (5.17.0), `timm` (1.0.29), `pyarrow` (24.0.0), `huggingface_hub` (1.31.0), `PIL` (12.2.0), `pandas` (3.0.2), `scikit-learn` (1.9.0).
- *No additional package installations are required.*
### 3.2 Evaluation of Authoritative Dataset Sources
1. **GenImage Official Google Drive (`1jGt10bwTbhEZuGXLyvrCuxOI0cBqQ1FS`):**
- Packaged as multi-part split zip archives: Midjourney spans `imagenet_midjourney.z01` to `.z50` (>100 GB); ADM spans `imagenet_ai_0508_adm.z01` to `.z12` (>50 GB).
- Downloading multi-part archives would exhaust local disk storage (92.46 GB free) and is **technically unviable**.
2. **DiffusionDB (`poloclub/diffusiondb`):**
- Contains millions of SD-generated images with text prompts, but **zero authentic photographs**. Pairing with external photo collections introduces uncontrollable confounding variables (camera sensor bias, compression artifacts, semantic mismatch). Unsuitable as a balanced benchmark.
3. **Curated GenImage Validation Split (`TheKernel01/Tiny-GenImage` on Hugging Face):**
- Preserves official NeurIPS 2023 GenImage validation partition under **CC BY-NC-SA 4.0**.
- Packaged as 4 modular Parquet shards of ~413 MB each.
- Remote Parquet columnar queries confirmed that **Shard 0 (`validation-00000-of-00004.parquet`, 433.18 MB)** contains:
* **875 Real photographs** (ImageNet-1k `val/nature`)
* **125 Midjourney** samples
* **125 ADM** samples
* **125 BigGAN** samples
* **125 GLIDE** samples
* **125 SD v1.5** samples (Note: SD v1.4 is absent in validation)
* **125 VQDM** samples
* **125 Wukong** samples
- **Verdict:** Downloading Shard 0 alone requires only **~413 MB** of temporary download space and **~20 MB** of extracted image space (<0.5 GB total), which is completely safe for our 92.46 GB free storage.
---
## 4. Proposed Stage 2 Dataset Plan ($N = 200$)
### 4.1 Target Composition
- **Total Images:** Exactly 200 images.
- **Authentic Class (`REAL`):** **100 authentic photographs** from ImageNet-1k validation set (`val/nature`).
- **AI-Generated Class (`FAKE`):** **100 AI-generated images**, balanced across two candidate generator families:
- **Generator A (50 samples):** **Midjourney** (commercial closed-source latent diffusion).
- **Generator B (50 samples):** **ADM (Ablated Diffusion Model)** (academic pixel-space guided diffusion).
*(Alternative Generator B: **BigGAN** if GAN vs. Diffusion comparison is preferred).*
### 4.2 Semantic Matching & Data Hygiene
- **Category Alignment:** Real images are ImageNet-1k validation natural objects; fake images were generated using ImageNet-1k synset text prompts (`"a photo of {class_name}"`).
- **Pristine Preservation:** Source image bytes will be extracted directly from Parquet records without resizing, cropping, or re-encoding. Each detector's adapter will apply its native preprocessing in-memory.
- **Collision Screening:** Every sample's SHA-256 hash will be verified against all historical benchmark manifests (Phase 4, Phase 5, Phase 6, random 250, and master Kaggle duplicate tables) to guarantee zero overlap.
- **Frozen Manifest:** `benchmark/external_generalization_manifest.csv` with 15 schema columns including `sample_id`, `relative_path`, `label`, `generator_family`, `sha256`, dimensions, and licensing.
---
## 5. Domain Inversion Hypothesis & Decision Fusion Forecast
In Phase 6 (facial portraits), ViT achieved 98.00% while ConvNeXt and UnivFD achieved ~51%, causing democratic voting to degrade accuracy by -35.00%.
In Phase 7 (general ImageNet objects/scenes), empirical conditions are **completely inverted**:
- **ViT (`dima806`):** Expected to struggle or become uncertain because it lacks general-scene detection capabilities and was fine-tuned purely on facial features.
- **ConvNeXt (`chen2024`):** Expected to lead detection because it was trained contrastively on diffusion reconstruction artifacts.
- **UnivFD (`CLIP ViT-L/14`):** Expected to demonstrate improved cross-generator transfer compared to facial StyleGAN.
- **Ensemble Dynamics:** Unlike Phase 6, democratic voting (Majority Vote, Mean Fake Probability) is expected to **outperform ViT-Priority**, because ConvNeXt and UnivFD should form a correct consensus to outvote an erroneous ViT.
---
## 6. Key Decisions Submitted for User Approval
Before proceeding to Stage 2 downloads and extraction, user confirmation is requested on the following design choices:
1. **Generator Family Pair Selection (100 fake images = $2 \times 50$):**
- **Recommended (Option 1):** **Midjourney (50) + ADM (50)**.
* Evaluates true cross-generator zero-shot generalization on two generator families **completely unseen in training to all three detectors** (commercial latent diffusion vs. academic pixel diffusion).
- **Alternative (Option 2):** **Midjourney (50) + BigGAN (50)**.
* Evaluates commercial diffusion against modern large-scale GAN (both unseen in detector training).
- **Alternative (Option 3):** **Midjourney (50) + Stable Diffusion v1.5 (50)**.
* Evaluates one unseen generator (Midjourney) and one near-in-distribution generator for ConvNeXt (SD v1.5).
2. **Dataset Ingestion Source:**
- **Recommended:** Ingest from Shard 0 (`validation-00000-of-00004.parquet`, ~413 MB) of `TheKernel01/Tiny-GenImage` on Hugging Face Hub. (Consumes <0.5 GB total space; avoids >100 GB multi-part Google Drive download).
3. **Scientific Claim Boundary:**
- Option 1/2 scientifically evaluates **unseen cross-generator generalization across architectures absent from all three training sets**.
- Option 3 tests **generalization beyond our existing facial benchmark**, noting that SD v1.5 is within ConvNeXt's training family.
|