Spaces:
Running on Zero
Running on Zero
|
Download MODEL_STATUS_AND_PROJECT_PROGRESSION.md from Thiru1122/Multimodel_setup: direct link, hf CLI and curl.
- Browser
- Download file 52.4 kB
-
https://huggingface.co/spaces/Thiru1122/Multimodel_setup/resolve/main/MODEL_STATUS_AND_PROJECT_PROGRESSION.md
- Command line
-
hf download hf://spaces/Thiru1122/Multimodel_setup/MODEL_STATUS_AND_PROJECT_PROGRESSION.md
-
curl -L -o MODEL_STATUS_AND_PROJECT_PROGRESSION.md https://huggingface.co/spaces/Thiru1122/Multimodel_setup/resolve/main/MODEL_STATUS_AND_PROJECT_PROGRESSION.md
52.4 kB
| # Model Status Audit and Project Progression Report | |
| **Project Title:** Cross-Model Evaluation of Deepfake Image Detection Under Diverse Image Generation and Manipulation Conditions | |
| **Audit Date:** 2026-10-30 | |
| **Audit Scope:** Read-only inspection of Desktop workspaces (`multi model connection`, `currently working`, `Deepfake-Detection-main`, `file 1`, `PG final year Project`), model caches, source code, adapters, test suites, and benchmark outputs. | |
| **Auditor Mode:** Safe diagnostic audit — zero modifications to source code, checkpoints, configurations, or packages. | |
| --- | |
| ## 1. Executive Summary | |
| A comprehensive, non-destructive audit of all project workspaces on the Desktop was conducted to uncover every detector model, establish exact provenance and physical condition, evaluate implementation completeness, and reconstruct historical project progression. | |
| ### Key Audit Findings | |
| 1. **Total Distinct Detectors Discovered:** Five (5) distinct model architectures across eight (8) physical weight checkpoints. | |
| 2. **`df_model.pt` Reality:** `df_model.pt` is **NOT** present in `multi model connection`. It resides in `currently working\model\df_model.pt` (and older repositories `file 1` and `Deepfake-Detection-main`). Crucially, `df_model.pt` is **not an image detector**; it is a **temporal video deepfake detector** (ResNeXt-50-32x4d + LSTM) trained on 20-frame FaceForensics++ video sequences. When evaluated on static images, the LSTM evaluates at timestep $t=1$ instead of $t=20$, collapsing accuracy to ~36%. It was deliberately set aside during the transition to image-based multi-model detection. | |
| 3. **Primary Image Detector Suite in `multi model connection`:** Three (3) image detector models are integrated into a standardized, decoupled adapter layer (`src/`): | |
| - **`chen2024_convnext` (ConvNeXt-Base):** Diffusion Reconstruction Contrastive Training (DRCT, ICML 2024 Spotlight / RAID). Checkpoint: `16_acc0.9993.pth` (354.6 MB). Fully operational adapter, 13/13 diagnostic checks passed. | |
| - **`vit` (ViT-Base/16):** Fine-tuned Vision Transformer (`dima806/deepfake_vs_real_image_detection`). Checkpoint: `model.safetensors` (343.2 MB). Highly accurate on facial manipulations (99.30% measured on 1,000 images in `currently working`), fully operational adapter. | |
| - **`univfd` (CLIP ViT-L/14 + Linear Probe):** UniversalFakeDetect (CVPR 2023). Checkpoint: `fc_weights.pth` (4.1 KB) + cached CLIP backbone `ViT-L-14.pt` (932.8 MB). Fully operational adapter, verified against official reference implementations. | |
| 4. **Additional Discovered Model in `currently working`:** `CommunityForensics-DeepfakeDet-ViT` (CVPR 2025, ViT-Small 384x384). Checkpoints: `model.safetensors` (87.3 MB) and legacy weights. Evaluated in `currently working` where it achieved 52.0% on face crops (indicating general scene training does not transfer directly to isolated faces). Not yet ported to `multi model connection`. | |
| 5. **Folder Completeness Comparison:** | |
| - `C:\Users\user\Desktop\multi model connection` contains the **most advanced, clean, and extensible multi-model architecture** for the research objective ("Cross-Model Evaluation of Deepfake Image Detection"). It implements a formal dataclass schema (`DetectorResult`), dynamic registry with lazy loading (`detector_registry.py`), isolated adapters for all three core image detectors, and diagnostic benchmark scripts. | |
| - `C:\Users\user\Desktop\currently working` contains the **most comprehensive empirical evaluation datasets and results** (10,905 labeled Kaggle face images, 1,000-sample balanced benchmarks, robustness ablations, video pipeline, and MTCNN face detection), but is architecturally a monolithic Flask/Gradio web deployment. | |
| 6. **Current Primary Blocker:** In `multi model connection`, when `run_detector_benchmark.py` runs all three detectors in a single Python process on CPU, the sequential retention of large model weights (CLIP ViT-L/14 + ConvNeXt-Base + ViT-Base) exhausts the Windows virtual memory paging file (`os error 1455`), crashing ViT inference. Additionally, no decision-fusion layer exists yet (currently only observational agreement tracking is implemented), and the local benchmark dataset contains only 5 diagnostic smoke-test samples. | |
| --- | |
| ## 2. Complete Model Inventory | |
| | Model Identifier | Architecture | Checkpoint Path | Size (Bytes) | SHA-256 Hash | Provenance & Source | Intended Task | | |
| | :--- | :--- | :--- | :--- | :--- | :--- | :--- | | |
| | **`df_model.pt`** | ResNeXt-50-32x4d + LSTM(2048, 2048) + Linear(2048, 2) | `c:\Users\user\Desktop\currently working\model\df_model.pt` *(also in `file 1` and `Deepfake-Detection-main`; pointer in `PG final year Project`)* | 226,597,567 (~216.1 MB) | `B893DE920A78A23F23F56F5EDADFD21D0217DC0C5BF15691EA6EE13CB9AFCD3C` | Pretrained on FaceForensics++ video sequences (PyTorch state_dict). Derived from original video deepfake project. | Sequential Video Deepfake Detection (20-frame temporal sequence) | | |
| | **`chen2024_convnext`** | ConvNeXt-Base (1024 embedding) + Linear(1024, 2) | `c:\Users\user\Desktop\multi model connection\models\chen2024_convnext\16_acc0.9993.pth` | 354,605,453 (~338.2 MB) | `F3638355676C0C07BC0B0E1082EBA698314971ED4771D0B55B207E946A2D74E0` | ICML 2024 Spotlight: *DRCT* (Chen et al., BokingChen) trained on DRCT-2M with SDv2 reconstruction; mirrored in RAID benchmark (`aimagelab/RAID_ckpt`). | Image Deepfake Detection (Diffusion & Synthetic Media) | | |
| | **`vit` (dima806)** | `ViTForImageClassification` (`google/vit-base-patch16-224`) | `c:\Users\user\Desktop\multi model connection\models\deepfake_vs_real_image_detection\model.safetensors` *(identical copy in `currently working`)* | 343,223,968 (~327.3 MB) | `B38E7D3DE2BE44B224CB4CFFE881A0C4B75F7283C5D4A2017FBADD58BD41FDEF` | Fine-tuned from Google ViT-Base on Kaggle 140k Real & Fake Faces dataset (`dima806/deepfake_vs_real_image_detection`). SafeTensors. | Static Image Facial Deepfake Detection | | |
| | **`univfd` (Probe)** | Linear Layer (`nn.Linear(768, 1)`) | `c:\Users\user\Desktop\multi model connection\UniversalFakeDetect\pretrained_weights\fc_weights.pth` | 4,083 (~4.0 KB) | `477100745713BCC957BEB2B40859536859B6483FD6301B3B9293151B194C7847` | CVPR 2023: *UniversalFakeDetect* (Ojha et al., WisconsinAIVision). Trained on ProGAN artifacts. | General Synthetic Image Detection (Linear Probe Head) | | |
| | **`univfd` (Backbone)** | OpenAI CLIP `ViT-L/14` (Visual Transformer, frozen) | `C:\Users\user\.cache\clip\ViT-L-14.pt` | 932,768,134 (~889.6 MB) | `B8CCA3FD41AE0C99BA7E8951ADF17D267CDB84CD88BE6F7C2E0ECA1737A03836` | Official OpenAI CLIP ViT-L/14 release, auto-downloaded by `clip.load()` in `UniversalFakeDetect`. | Feature Representation Backbone for UnivFD | | |
| | **`CommunityForensics ViT` (Fixed)** | `ViTForImageClassification` (`timm/vit_small_patch16_384`, ViT-Small) | `c:\Users\user\Desktop\currently working\deepfake-detector-model-v1-main\models\CommunityForensics-DeepfakeDet-ViT\model.safetensors` *(and `model_fixed.safetensors`)* | 87,270,764 (~83.2 MB) | `275BA982236DDD6AFDDF7131F8133E89F537574B964CF8FA5825B4956D741692` | CVPR 2025: *Community Forensics* (buildborderless/OpenSight). Trained on 2.7M samples across 4,803 generators. | Multi-Generator Scene Deepfake Detection | | |
| | **`CommunityForensics ViT` (Legacy)** | Pre-fix ViT configuration checkpoint | `c:\Users\user\Desktop\currently working\deepfake-detector-model-v1-main\models\CommunityForensics-DeepfakeDet-ViT\model_legacy.safetensors` | 143,969,256 (~137.3 MB) | `91A3C4D1B7D453B4B74E377C8E079662A6CC2C97F8CA448DAEDDA8864EC54605` | Pre-July 2026 legacy release from CommunityForensics (superseded by fixed checkpoint). | Archived/Historical Checkpoint | | |
| --- | |
| ## 3. Model Status Matrix | |
| Each discovered model is evaluated against the 8 required lifecycle stages. Statuses are assigned based on strict empirical evidence. | |
| | Status Stage | `df_model.pt` | `chen2024_convnext` | `vit` (dima806) | `univfd` | `CommunityForensics` | | |
| | :--- | :---: | :---: | :---: | :---: | :---: | | |
| | **1. Discovered** | **VERIFIED** | **VERIFIED** | **VERIFIED** | **VERIFIED** | **VERIFIED** | | |
| | **2. Identified** | **VERIFIED** | **VERIFIED** | **VERIFIED** | **VERIFIED** | **VERIFIED** | | |
| | **3. Loadable** | **VERIFIED** | **VERIFIED** | **VERIFIED** | **VERIFIED** | **VERIFIED** | | |
| | **4. Integrated** | **PARTIAL** *(Video only in `currently working`; NOT in `multi model connection`)* | **VERIFIED** *(in `multi model connection`)* | **VERIFIED** *(in both folders)* | **VERIFIED** *(in `multi model connection`)* | **NOT VERIFIED** *(Only in `currently working` eval script)* | | |
| | **5. Smoke-tested** | **VERIFIED** *(Video passed; static image failed)* | **VERIFIED** *(13/13 checks passed in `scripts/test_convnext_adapter.py`)* | **VERIFIED** *(5/5 checks passed in `scripts/test_vit_adapter.py`)* | **VERIFIED** *(5/5 checks passed in `scripts/test_univfd_adapter.py`)* | **VERIFIED** *(in `currently working` test suite)* | | |
| | **6. Inference-validated** | **VERIFIED** *(as video model; INVALIDATED as image detector)* | **VERIFIED** *(Validated against DRCT reference code)* | **VERIFIED** *(Config label mapping & ViTProcessor verified)* | **VERIFIED** *(Validated against `validate.py` & `datasets.py`)* | **VERIFIED** *(384x384 sigmoid output verified)* | | |
| | **7. Benchmark-evaluated**| **VERIFIED** *(4 videos passed; 100 static images failed ~36%)* | **PARTIAL** *(5 diagnostic smoke images in MMC; NOT on full DRCT-2M)* | **VERIFIED** *(1,000 images in CW: 99.30%; 5 images in MMC)* | **PARTIAL** *(5 diagnostic smoke images in MMC; NOT on full Wang2020)* | **VERIFIED** *(100 images in CW: 52.00%)* | | |
| | **8. Fusion-ready** | **NOT VERIFIED** *(Modality mismatch: 5D video tensor)* | **VERIFIED** *(Standard `DetectorResult`, calibrated logits)* | **PARTIAL** *(RAM paging error under joint execution; face-crop bias)* | **VERIFIED** *(Standard `DetectorResult`, calibrated logits)* | **NOT VERIFIED** *(No adapter in `multi model connection`)* | | |
| --- | |
| ## 4. Evidence Supporting Completed Stages | |
| ### Model 1: `df_model.pt` | |
| - **Identity & Provenance:** Diagnostic report `model_accuracy_diagnosis.txt` (lines 33-88) and `CURRENT_STATE_AUDIT.md` confirm it is a ResNeXt-50-32x4d spatial backbone combined with a 1-layer unidirectional LSTM (latent dimension 2048, hidden dimension 2048, no bias) and a linear classification head (2048 $\to$ 2). | |
| - **Video Modality Evidence:** In `currently working/server.py` (lines 375-463) and `video_inference.py`, input is extracted into 20 sequential frames, resized to $112 \times 112$, normalized with ImageNet statistics, and stacked into a 5-dimensional tensor `[1, 20, 3, 112, 112]`. | |
| - **Static Image Collapse:** When fed a single static image, sequence length is 1 (`[1, 1, 3, 112, 112]`). The LSTM hidden state is extracted at timestep $t=-1$, which evaluates initial state $h_1$ instead of accumulated temporal state $h_{20}$. Evaluated on 100 images from Kaggle 140k faces, it produced 36% accuracy (worse than random guessing). | |
| - **Physical Locations:** Exactly identical 226,597,567-byte binary present at `currently working\model\df_model.pt`, `Deepfake-Detection-main\Deepfake-Detection-main\model\df_model.pt`, and `file 1\model\df_model.pt`. Git LFS pointer at `PG final year Project`. Completely absent from `multi model connection`. | |
| ### Model 2: `chen2024_convnext` | |
| - **Identity & Provenance:** Checkpoint `16_acc0.9993.pth` is verified against `temp_DRCT_reference/README.md` (lines 72, 92). It represents the official ICML 2024 Spotlight checkpoint for `convnext_base_in22k` trained on DRCT-2M with Stable Diffusion v2 reconstruction (`sdv2/convnext_base_in22k_224_drct_amp_crop/16_acc0.9993.pth`). It was also mirrored in the RAID benchmark checkpoint repository (`aimagelab/RAID_ckpt`). | |
| - **Architecture & Loading:** Built with `timm.create_model("convnext_base", pretrained=False, num_classes=1024)` + `nn.Linear(1024, 2)`. Loads cleanly with `strict=True` and 0 missing or unexpected parameters. | |
| - **Preprocessing:** Verified in `src/convnext_adapter.py`: `transforms.Resize(224, bicubic) -> transforms.CenterCrop(224) -> transforms.ToTensor() -> transforms.Normalize(ImageNet mean/std)`. | |
| - **Operational Diagnostic Results:** In `scripts/convnext_adapter_test_results.json`, all 13 diagnostic tests passed: | |
| - Lazy loading: verified (zero parameter instantiation before first call). | |
| - Detector isolation: verified. | |
| - Valid real image (`real.jpg`): `Prediction=REAL`, fake probability `0.004387`, logit `-5.4248`. | |
| - Valid AI image (`ai_generated.jpg`): `Prediction=REAL`, fake probability `0.073853`, logit `-2.5290`. | |
| - Valid manipulated image (`manipulated.jpg`): `Prediction=REAL`, fake probability `0.142673`, logit `-1.7936`. | |
| - Invalid inputs: Corrupted file, plain text file, non-existent file, and `None` input all properly rejected with `Prediction=ERROR`. | |
| - Probability bounds $[0, 1]$, sum to $1.0 \pm 10^{-5}$, and confidence alignment all verified. | |
| ### Model 3: `vit` (`dima806/deepfake_vs_real_image_detection`) | |
| - **Identity & Checkpoint:** Hugging Face model `dima806/deepfake_vs_real_image_detection`, based on `ViTForImageClassification` (`google/vit-base-patch16-224`). Checkpoint `model.safetensors` (343.2 MB) matches SHA-256 `B38E7D3D...` in both project directories. | |
| - **Preprocessing & Label Mapping:** `AutoImageProcessor` applies Resize 224x224, Rescale 1/255, and Normalize mean/std `[0.5, 0.5, 0.5]`. Adapter dynamically inspects `config.id2label` to map label indices (`0: Real`, `1: Fake`). | |
| - **Benchmark Evaluation in `currently working`:** | |
| - Initial 100-sample test: 99.00% accuracy (50/50 fake recall, 49/50 real recall). | |
| - 1,000-sample large-scale balanced evaluation (`extended_validation_report.md`): **99.30% accuracy** (500/500 fake recall, 493/500 real recall), ROC-AUC **0.9999**, PR-AUC **0.9999**, Brier score **0.0058**. | |
| - Robustness testing: Maintained $\ge 96\%$ accuracy under JPEG compression (Q=30, Q=50), resizing down to $64 \times 64$, Gaussian blur, brightness adjustments, and contrast perturbations. | |
| - **Performance in `multi model connection`:** Passed all operational checks in `scripts/vit_adapter_test_results.json`. On general/non-facial diffusion (`ai_001`), produced fake probability 34.24% (assigned higher suspicion than UnivFD or ConvNeXt, though still below the 0.50 threshold due to lack of face geometry). | |
| ### Model 4: `univfd` (UniversalFakeDetect) | |
| - **Identity & Checkpoint:** CVPR 2023 UniversalFakeDetect (Ojha et al.). Checkpoint `UniversalFakeDetect/pretrained_weights/fc_weights.pth` contains only linear probe weights `[1, 768]` and scalar bias `-0.5167`. Frozen visual backbone weights cached at `C:\Users\user\.cache\clip\ViT-L-14.pt` (932.8 MB). | |
| - **Preprocessing Pipeline Alignment:** `scripts/compare_official_inference.py` cross-examined three repository implementations: | |
| 1. `validate.py`: Direct `CenterCrop(224)` without resizing (evaluates only 0.4% of high-res image area). | |
| 2. `datasets.py`: `Resize(256, bicubic) -> CenterCrop(224)`. | |
| 3. `app.py` / `src/univfd_adapter.py`: `model.preprocess` (`Resize(224, bicubic) -> CenterCrop(224)`), which provides full-scene coverage and matches CLIP standards. | |
| - **Operational Diagnostic Results:** Passed all test cases in `scripts/adapter_test_results.json`. | |
| - **Known Domain Boundary:** `docs/UNIVFD_LIMITATIONS.md` establishes that because UnivFD was trained on 2018 ProGAN artifacts, it is blind to modern diffusion generation (predicts REAL on Google Imagen 3 diffusion with 99.9% confidence) and localized splicing (predicts REAL on composite images with 99.4% confidence). | |
| ### Model 5: `CommunityForensics-DeepfakeDet-ViT` | |
| - **Identity & Checkpoint:** CVPR 2025 "Community Forensics" (OpenSight). Built on `timm/vit_small_patch16_384.augreg_in21k_ft_in1k`, single-class sigmoid output (`num_labels: 1`), input size $384 \times 384$. Local directory contains verified fixed weights `model.safetensors` (87.3 MB, SHA-256 `275BA982...`). | |
| - **Benchmark in `currently working`:** Evaluated on 100 face images in `evaluate_community_forensics.py`: achieved **52.00% accuracy** (8.00% fake recall, 96.00% real recall). This proves that models trained on general scene generators (Midjourney/SD landscapes) fail to detect facial manipulations without facial fine-tuning. | |
| --- | |
| ## 5. Timeline of Project Progression | |
| ``` | |
| +---------------------------------------------------------------------------------------------------------+ | |
| | PHASE 1: Single-Model Video Baseline (December 2025 - Early 2026) | | |
| | - Repositories: 'file 1', 'Deepfake-Detection-main', 'PG final year Project' | | |
| | - Core Model: df_model.pt (ResNeXt-50 + LSTM) | | |
| | - Architecture: Flask web server (server.py) + React frontend | | |
| | - Key Event: 100-image static evaluation yields unexpected 36% accuracy | | |
| +---------------------------------------------------------------------------------------------------------+ | |
| | | |
| v | |
| +---------------------------------------------------------------------------------------------------------+ | |
| | PHASE 2: Deep Architectural Diagnosis & Modality Separation (September 2026) | | |
| | - Workspace: 'currently working' | | |
| | - model_accuracy_diagnosis.txt proves df_model.pt is a 20-frame temporal video model, not image model | | |
| | - Discovered dlib face-cropping bug that mutilated already-cropped face datasets | | |
| | - Split application into dual-pipeline: Video (df_model.pt) vs Static Image (dima806 ViT) | | |
| | - Evaluated ViT (dima806) on 1,000 Kaggle face images: achieved 99.30% accuracy, ROC-AUC 0.9999 | | |
| | - Evaluated CommunityForensics-DeepfakeDet-ViT (52.0% accuracy) and PrithivMLmods SigLIP (39.0% acc) | | |
| | - Attempted Hugging Face ZeroGPU Gradio deployment (branch hf-gradio-deployment) | | |
| +---------------------------------------------------------------------------------------------------------+ | |
| | | |
| v | |
| +---------------------------------------------------------------------------------------------------------+ | |
| | PHASE 3: New Multi-Model Cross-Evaluation Framework (Mid-to-Late September 2026) | | |
| | - Workspace: 'multi model connection' | | |
| | - Focused on: Cross-Model Evaluation of Deepfake Image Detection Under Diverse Generation Conditions | | |
| | - Integrated UniversalFakeDetect (CLIP ViT-L/14 + linear probe) in app.py and Hugging Face Space | | |
| | - Documented UnivFD limitations on modern diffusion (Imagen 3) and composite manipulations | | |
| | - Built standardized dataclass schema (src/detector_schema.py) defining DetectorResult | | |
| | - Built decoupled registry with lazy instantiation (src/detector_registry.py) | | |
| | - Implemented and contract-tested isolated ViT adapter (src/vit_adapter.py) | | |
| | - Researched Detector #3 candidates (DETECTOR_3_CANDIDATE_ANALYSIS.md): SigLIP, Swin, CNN, RAID | | |
| +---------------------------------------------------------------------------------------------------------+ | |
| | | |
| v | |
| +---------------------------------------------------------------------------------------------------------+ | |
| | PHASE 4: ConvNeXt/DRCT Integration & Joint Multi-Model Benchmark (October 2026 - Current State) | | |
| | - Workspace: 'multi model connection' | | |
| | - Selected Chen et al. 2024 / DRCT ConvNeXt-Base (16_acc0.9993.pth from RAID checkpoint collection) | | |
| | - Sourced temp_DRCT_reference codebase to confirm exact architecture, preprocessing, and label mapping | | |
| | - Implemented src/convnext_adapter.py and executed scripts/test_convnext_adapter.py (13/13 passed) | | |
| | - Verified all 3 adapters compliant with contract (scripts/contract_test_summary.json) | | |
| | - Ran scripts/run_detector_benchmark.py across all 3 detectors on benchmark manifest | | |
| | - Encountered Windows paging file memory exhaustion (os error 1455) for ViT during joint run | | |
| | - Observational agreement tracking recorded in detector_comparison.json (DISAGREE on test samples) | | |
| +---------------------------------------------------------------------------------------------------------+ | |
| ``` | |
| --- | |
| ## 6. Comparison of the Two Known Project Folders | |
| | Dimension | `C:\Users\user\Desktop\currently working` | `C:\Users\user\Desktop\multi model connection` | | |
| | :--- | :--- | :--- | | |
| | **Primary Focus** | Full-stack application deployment (Flask + React + Gradio) for dual-modality (facial images & videos). | Modular scientific evaluation framework for multi-model cross-evaluation of image deepfakes. | | |
| | **Model Architectures Present** | 1. `df_model.pt` (ResNeXt-50+LSTM video)<br>2. `dima806` (ViT-Base/16 image)<br>3. `CommunityForensics` (ViT-Small 384) | 1. `chen2024_convnext` (ConvNeXt-Base)<br>2. `dima806` (ViT-Base/16 image)<br>3. `univfd` (CLIP ViT-L/14 + Linear Probe) | | |
| | **Architectural Modularity** | Monolithic. Code is tightly coupled to Flask endpoints, MTCNN cascades, and specific model wrapper singletons. | Highly modular and decoupled. Centralized `detector_registry.py`, lazy loading, and standardized `DetectorResult` schema. | | |
| | **Adapter & Registry Layer** | None. Ad-hoc wrappers (`PilotDeepfakePipeline`, `DeepfakeModelWrapper`, `VideoModelWrapper`). | Standardized `src/detector_schema.py`, `src/detector_registry.py`, `src/vit_adapter.py`, `src/convnext_adapter.py`, `src/univfd_adapter.py`. | | |
| | **Evaluation Datasets** | Comprehensive: 10,905 labeled Kaggle face images (`Dataset/Test/Real` & `Fake`), 4 test videos. | Minimal: 5 diagnostic smoke-test samples (`benchmark/test_images/` & `benchmark/images/`). | | |
| | **Inference Pipeline** | Uses MTCNN face-detection cascading (primary, sensitive, 2x upscaled) and quality analysis heuristics. | Pure detector evaluation: passes full image through verified native preprocessing transforms without external face cropping. | | |
| | **Video Detection** | Fully implemented and validated (`video_inference.py`, `detectFakeVideo`). | Not present. Out of scope for current image-detection objective. | | |
| | **Benchmark Harness** | Ad-hoc scripts (`evaluate_model.py`, `evaluate_random_250.py`, `extended_validation_dima806.py`). | Standardized harness (`scripts/run_detector_benchmark.py`) outputting long-format and side-by-side JSON & CSV. | | |
| | **Version & Status** | Historical baseline and dataset repository (last major commits mid-September 2026). | **Active, current working directory** containing the latest multi-model code, ConvNeXt integration, and DRCT references. | | |
| --- | |
| ## 7. Duplicate, Missing, or Unverified Artifacts | |
| ### 1. Duplicate Artifacts | |
| - **`dima806 ViT` weights (`model.safetensors`):** Exact binary duplicate (343,223,968 bytes, SHA-256 `B38E7D3D...`) present in both `currently working` and `multi model connection`. | |
| - **`df_model.pt` weights:** Exact binary duplicate (226,597,567 bytes, SHA-256 `B893DE92...`) present in `currently working`, `Deepfake-Detection-main`, and `file 1`. | |
| - **`CommunityForensics` weights:** `model.safetensors` and `model_fixed.safetensors` in `currently working` are 100% identical copies (87,270,764 bytes, SHA-256 `275BA982...`). | |
| ### 2. Missing Artifacts | |
| - **`df_model.pt` in `multi model connection`:** Completely absent from the active project directory. While the task prompt noted it as a known location, physical inspection confirms it was never copied into `multi model connection`. | |
| - **Statistically Significant Labeled Benchmark Dataset in `multi model connection`:** Only 5 diagnostic images exist in `benchmark/images/` and `benchmark/test_images/`. A robust evaluation dataset spanning diverse generators (GAN, Diffusion, Splicing, Authentic) is missing from the main workspace. | |
| - **`UniversalFakeDetect` Alternative Checkpoints:** As documented in `UNIVFD_LIMITATIONS.md`, the repository contains only `fc_weights.pth` (ProGAN linear probe). No diffusion-adapted or modern generator probes are present. | |
| ### 3. Unverified Claims in Documentation | |
| - **ConvNeXt "99.93% Accuracy":** The checkpoint filename `16_acc0.9993.pth` reflects an internal training accuracy metric from DRCT on DRCT-2M (SDv2). It must **not** be cited as the model's accuracy on the project's target benchmark. | |
| - **Historical "95% Accuracy" in `currently working/README.md`:** TensorBoard event logs in `logs/` confirm the original Keras training runs never converged (loss remained $\sim 0.69$, accuracy hovered at $49.5\% - 54.8\%$). The claim in the old README is factually invalid. | |
| --- | |
| ## 8. Current Blockers and Technical Risks | |
| ### Blocker 1: Process Memory Exhaustion under Joint Model Execution (RESOLVED) | |
| - **Status:** **RESOLVED** on 2026-10-30 (See detailed verification in `scripts/MEMORY_SAFE_BENCHMARK_REPORT.md`). | |
| - **Resolution:** Refactored `scripts/run_detector_benchmark.py` to evaluate detector-by-detector sequentially rather than interleaved across samples. Added explicit `unload_model()` methods on all adapters, generational garbage collection (`gc.collect()`), CUDA cache clearing, and process isolation support (`--mode subprocess`). | |
| - **Verified Outcome:** All 3 detectors (`chen2024_convnext`, `univfd`, `vit`) now evaluate cleanly with 0 errors across all benchmark samples. ViT memory paging crash (`os error 1455`) is fully eliminated, with predictions matching historical baseline floats to 6 decimal places. | |
| ### Blocker 2: Absence of a Decision-Fusion Engine (Architectural) | |
| - **Current State:** The system only records an observational `agreement_status` string (`"AGREE"` if all predictions match, `"DISAGREE"` otherwise) in `detector_comparison.json`. | |
| - **Deficiency:** No logic exists to combine continuous probabilities, compute weighted confidence scores, handle conflicting verdicts, apply Bayesian evidence updating, or output a synthesized prediction with uncertainty calibration. | |
| ### Blocker 3: Outdated Single-Model Interface in `app.py` (Integration) | |
| - **Current State:** `app.py` in `multi model connection` remains hardcoded to UniversalFakeDetect alone. | |
| - **Deficiency:** It does not use `detector_registry.py`, does not expose the ViT or ConvNeXt adapters, and cannot display multi-model comparative results or fused outputs in the Gradio UI. | |
| ### Blocker 4: Dataset Scale and Diversity Mismatch (Methodological) | |
| - **Current State:** The benchmark in `multi model connection` comprises only 5 files (3 valid, 1 corrupted, 1 text). | |
| - **Deficiency:** Three valid images cannot establish statistical metrics (Accuracy, Precision, Recall, F1, ROC-AUC, ECE). Furthermore, the models have fundamentally different training domains: | |
| - `dima806 ViT`: Facial manipulations and crops. | |
| - `univfd`: ProGAN structural artifacts. | |
| - `chen2024_convnext`: Diffusion reconstruction contrastive artifacts. | |
| Evaluating them on arbitrary images without category-specific labeling produces misleading cross-model comparisons. | |
| ### Blocker 5: Video Modality Isolation (`df_model.pt`) | |
| - **Current State:** `df_model.pt` requires a 5D temporal video tensor `[B, 20, 3, 112, 112]`. | |
| - **Deficiency:** It cannot be directly evaluated on still images. Attempting to pass static images into `df_model.pt` produces meaningless representations. If video detection is part of the final system, it must remain isolated in a dedicated video pipeline. | |
| --- | |
| ## 9. Recommended Next Steps (Priority Order) | |
| 1. **Implement Memory-Safe Sequential Execution in Benchmark Harness:** *(COMPLETED)* | |
| Updated `scripts/run_detector_benchmark.py` with sequential model evaluation, explicit adapter deallocation (`unload_model()`), garbage collection, and subprocess isolation mode. Verified 100% success across all 3 detectors without paging errors. | |
| 2. **Assemble a Stratified Multi-Condition Benchmark Dataset:** | |
| Construct an expanded benchmark manifest (e.g. 100–300 images) containing balanced categories: | |
| - Authentic photographs (natural cameras, portraits, landscapes). | |
| - GAN-generated images (StyleGAN, ProGAN). | |
| - Diffusion-generated images (Stable Diffusion, Midjourney, Imagen). | |
| - Localized facial/composite manipulations. | |
| Port relevant samples from `currently working/Dataset/Test` for facial benchmarks. | |
| 3. **Design and Implement the Multi-Model Decision-Fusion Engine:** | |
| Create `src/fusion_engine.py` to accept multiple `DetectorResult` outputs and compute: | |
| - Dynamic domain routing (e.g. face vs general scene weighting). | |
| - Calibrated probability fusion (weighted soft voting, Bayesian logit pooling). | |
| - An explicit `UNCERTAIN` classification policy when model confidence is discordant. | |
| 4. **Update `app.py` to Serve the Multi-Model Framework:** | |
| Refactor `app.py` to use `src.detector_registry` and the fusion engine, presenting users with: | |
| - Per-detector independent predictions, probabilities, and latency. | |
| - The unified fusion verdict with confidence breakdown and agreement diagnostics. | |
| 5. **Decide Scope for `df_model.pt` (Video Pipeline):** | |
| Formally document whether video detection is within the scope of the current "Cross-Model Evaluation of Deepfake Image Detection" project. If image-only, archive `df_model.pt`. If dual-modality is required, port `video_inference.py` from `currently working` as a separate modality service. | |
| --- | |
| ## 10. Proposed Decision-Fusion Engine Architecture | |
| ### Architectural Overview | |
| ```text | |
| +----------------------------+ | |
| | Input Image (RGB) | | |
| +----------------------------+ | |
| | | |
| +--------------------+--------------------+ | |
| | | | | |
| v v v | |
| +------------------+ +------------------+ +------------------+ | |
| | UnivFDAdapter | | ViTAdapter | | ConvNeXtAdapter | | |
| | (CLIP ViT-L/14) | | (ViT-Base/16) | | (ConvNeXt-Base) | | |
| +------------------+ +------------------+ +------------------+ | |
| | | | | |
| v v v | |
| DetectorResult DetectorResult DetectorResult | |
| | | | | |
| +--------------------+--------------------+ | |
| | | |
| v | |
| +----------------------------+ | |
| | Decision-Fusion Engine | | |
| | (src/fusion_engine.py) | | |
| +----------------------------+ | |
| | | |
| +----------------------+----------------------+ | |
| | | | | |
| v v v | |
| [Domain Weighting] [Log-Odds / Bayesian] [Uncertainty Gate] | |
| Face: ViT high P(Fake) aggregation Threshold |P - 0.5| | |
| Diffusion: ConvNeXt Outlier dampening Disagreement flag | |
| | | | | |
| +----------------------+----------------------+ | |
| | | |
| v | |
| +----------------------------+ | |
| | FusedDetectorResult | | |
| | - Final Verdict | | |
| | - Fused Fake Probability | | |
| | - Confidence Score | | |
| | - Agreement Level | | |
| | - Contributing Weights | | |
| +----------------------------+ | |
| ``` | |
| ### Core Design Principles for the Proposed Fusion Engine | |
| 1. **Non-Invasive Modularity:** | |
| The fusion engine consumes standardized `DetectorResult` dataclass instances emitted by the existing adapters. Zero modifications to the underlying adapters or model checkpoints are required. | |
| 2. **Domain-Aware Dynamic Weighting:** | |
| Because detectors possess divergent domain specializations: | |
| - **ViT (`dima806`)** receives higher weight when facial features/landmarks are prominent ($w_\text{vit} \approx 0.50$). | |
| - **ConvNeXt (`chen2024`)** receives higher weight on full scenes and diffusion generation ($w_\text{conv} \approx 0.40$). | |
| - **UnivFD** provides high-precision corroboration for GAN structural artifacts ($w_\text{univ} \approx 0.10$). | |
| 3. **Calibrated Probability Fusion:** | |
| Instead of crude hard-voting (which discards confidence) or naive probability averaging (which suffers from overconfident outliers), combine raw log-odds: | |
| $$\text{Logit}_\text{fused} = \sum_{i=1}^N w_i \cdot \text{logit}_i$$ | |
| $$P_\text{fused}(\text{Fake}) = \frac{1}{1 + e^{-\text{Logit}_\text{fused}}}$$ | |
| 4. **Explicit Uncertainty Policy:** | |
| - If $|P_\text{fused} - 0.50| < 0.15$ or if strong binary conflict exists with high individual confidences ($P_1 > 0.85$ and $P_2 < 0.15$), classify as `UNCERTAIN` / `REVIEW_REQUIRED`. | |
| - Never force a false binary decision when detectors strongly disagree. | |
| 5. **Memory-Conscious Sequential Orchestration:** | |
| The fusion coordinator will evaluate detectors sequentially, clearing memory between passes if running on memory-limited hardware, ensuring robust execution in any environment. | |
| --- | |
| ## 11. Phase 6 Completion: Cross-Model Deepfake Detection & Ensemble Classifier | |
| ### 11.1 Phase 6 Overview & Execution Status | |
| - **Status:** **COMPLETED & VERIFIED (November 2026)** | |
| - **Test Corpus:** 200 pristine, leak-free test samples (100 REAL authentic FFHQ portraits, 100 FAKE StyleGAN portraits) frozen at [`benchmark/phase6_test_manifest.csv`](file:///C:/Users/user/Desktop/multi%20model%20connection/benchmark/phase6_test_manifest.csv). | |
| - **Execution Harness:** Fully sequential subprocess isolation, preventing Windows virtual memory exhaustion (`os error 1455`) while operating under an 8.0 GB RAM / CPU inference profile. | |
| - **Active Detector Models Evaluated:** | |
| 1. `vit` (`dima806/deepfake_vs_real_image_detection`) | |
| 2. `univfd` (`CLIP ViT-L/14 + linear probe`) | |
| 3. `chen2024_convnext` (`models/chen2024_convnext/16_acc0.9993.pth`) | |
| *(Incompatible 5D video model `df_model.pt` strictly excluded from 2D image pipelines).* | |
| ### 11.2 Verified Performance Comparison Matrix ($N = 200$) | |
| | Model / Fusion Strategy | Accuracy (%) | 95% Wilson CI (Acc) | Balanced Acc (%) | Precision | Recall (Fake Det.) | Specificity (Real Ret.) | F1-Score | Mean Latency (ms) | | |
| | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | | |
| | **ViT (`dima806`)** | **98.00%** | **[94.97%, 99.22%]** | **98.00%** | **0.9615** | **1.0000** | **0.9600** | **0.9804** | **192.4 ms** | | |
| | **UnivFD (CLIP ViT-L/14)** | 51.50% | [44.61%, 58.33%] | 51.50% | 0.5484 | 0.1700 | 0.8600 | 0.2595 | 942.1 ms | | |
| | **ConvNeXt (DRCT / RAID)** | 51.00% | [44.12%, 57.84%] | 51.00% | 0.5333 | 0.1600 | 0.8600 | 0.2462 | 252.7 ms | | |
| | **Ensemble (Majority Vote)** | 63.00% | [56.12%, 69.39%] | 63.00% | 0.8824 | 0.3000 | 0.9600 | 0.4478 | 1387.2 ms | | |
| | **Ensemble (Mean Fake Prob)** | 63.50% | [56.63%, 69.86%] | 63.50% | 0.8857 | 0.3100 | 0.9600 | 0.4593 | 1387.2 ms | | |
| | **Ensemble (ViT Priority)** | **98.00%** | **[94.97%, 99.22%]** | **98.00%** | **0.9615** | **1.0000** | **0.9600** | **0.9804** | **192.4 ms** | | |
| ### 11.3 Key Findings & Verified Insights | |
| 1. **ViT In-Domain Excellence:** ViT detected 100 out of 100 StyleGAN fake faces (100% Recall, 0 False Negatives) and achieved 96% Specificity (4 False Positives). | |
| 2. **UnivFD & ConvNeXt StyleGAN Blind Spot:** UnivFD and ConvNeXt suffered severe false-negative rates ($FN = 83$ and $FN = 84$, respectively), missing >80% of synthetic faces due to differing training distributions (ProGAN, general diffusion, and localized frequency artifacts). | |
| 3. **The Ensemble Degradation Paradox:** Unweighted democratic voting (**Majority Vote** at 63.00% and **Mean Fake Probability** at 63.50%) caused a catastrophic **-35.00% collapse** in accuracy. UnivFD and ConvNeXt formed an erroneous 2-to-1 majority on 70 out of 100 fake faces, outvoting ViT's correct predictions. | |
| 4. **Hierarchical Routing Success:** The **ViT-Priority Fallback** architecture successfully preserved the full 98.00% accuracy while maintaining secondary models as fallback guards. | |
| 5. **Phase 6 Artifacts:** | |
| - Manifest: [`benchmark/phase6_test_manifest.csv`](file:///C:/Users/user/Desktop/multi%20model%20connection/benchmark/phase6_test_manifest.csv) | |
| - Dataset Selection Report: [`docs/PHASE6_DATASET_SELECTION_REPORT.md`](file:///C:/Users/user/Desktop/multi%20model%20connection/docs/PHASE6_DATASET_SELECTION_REPORT.md) | |
| - Evaluation Report: [`docs/PHASE6_EVALUATION_REPORT.md`](file:///C:/Users/user/Desktop/multi%20model%20connection/docs/PHASE6_EVALUATION_REPORT.md) | |
| - Generalization Plan: [`docs/PHASE6_GENERALIZATION_PLAN.md`](file:///C:/Users/user/Desktop/multi%20model%20connection/docs/PHASE6_GENERALIZATION_PLAN.md) | |
| - Completion Report: [`docs/PHASE6_COMPLETION_REPORT.md`](file:///C:/Users/user/Desktop/multi%20model%20connection/docs/PHASE6_COMPLETION_REPORT.md) | |
| - Post-Execution Audit: [`docs/PHASE6_POST_EXECUTION_AUDIT.md`](file:///C:/Users/user/Desktop/multi%20model%20connection/docs/PHASE6_POST_EXECUTION_AUDIT.md) | |
| --- | |
| ## 12. Phase 7 Completion & Post-Execution Audit: External AI-Generated Image Generalization (GenImage) | |
| ### 12.1 Phase 7 Overview & Methodology | |
| - **Audit Status:** **VERIFIED WITH LIMITATIONS (November 2026)** | |
| - **Audit Deliverable:** [`docs/PHASE7_POST_EXECUTION_AUDIT.md`](file:///c:/Users/user/Desktop/multi%20model%20connection/docs/PHASE7_POST_EXECUTION_AUDIT.md) | |
| - **Test Corpus:** 200 external general-scene samples from official GenImage validation partition (`TheKernel01/Tiny-GenImage`): | |
| * **100 REAL:** Authentic ImageNet-1k photographs (`val/nature`). | |
| * **50 FAKE (Midjourney):** Commercial closed-source latent diffusion (unseen in training to all 3 detectors). | |
| * **50 FAKE (ADM):** Academic pixel-space guided diffusion (unseen in training to all 3 detectors). | |
| - **Frozen Manifest:** [`benchmark/external_generalization_manifest.csv`](file:///C:/Users/user/Desktop/multi%20model%20connection/benchmark/external_generalization_manifest.csv) | |
| - **Execution Harness:** Sequential subprocess isolation; zero memory exhaustion errors. | |
| ### 12.2 Verified Performance Matrix ($N = 200$) | |
| | Model / Strategy | Aggregate Acc (%) | 95% Wilson CI | Recall (Fake) | Specificity (Real) | F1-Score | ROC-AUC | Midjourney Acc (%) | ADM Acc (%) | Mean Latency (ms) | | |
| | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | | |
| | **ViT (`dima806`)** | 52.00% | [45.10%, 58.82%] | 18.00% | 86.00% | 0.2727 | 0.4673 | 64.00% | 62.67% | **390.1 ms** | | |
| | **UnivFD (CLIP ViT-L/14)** | **57.50%** | [50.57%, 64.15%] | 15.00% | **100.00%** | 0.2609 | 0.6662 | 66.67% | **76.67%** | 1494.1 ms | | |
| | **ConvNeXt (DRCT / RAID)** | 56.50% | [49.57%, 63.18%] | 13.00% | **100.00%** | 0.2301 | **0.7926** | **74.00%** | 68.00% | 516.8 ms | | |
| | **Ensemble (Majority Vote)** | 52.50% | [45.60%, 59.31%] | 5.00% | **100.00%** | 0.0952 | 0.7181* | 67.33% | 69.33% | 2401.0 ms | | |
| | **Ensemble (Mean Fake Prob)** | 52.50% | [45.60%, 59.31%] | 5.00% | **100.00%** | 0.0952 | 0.7181 | 67.33% | 69.33% | 2401.0 ms | | |
| | **Ensemble (ViT Priority)** | 52.00% | [45.10%, 58.82%] | 18.00% | 86.00% | 0.2727 | 0.4673 | 64.00% | 62.67% | **390.1 ms** | | |
| *\*Note: 0.7181 reflects the continuous mean probability vector attached to `FusionResult.fake_probability`. When evaluated on discrete vote counts (0, 1, 2, 3), Majority Voting ROC-AUC is **0.6385**; under hard binary decisions, it is **0.5250** (matching Balanced Accuracy).* | |
| ### 12.3 Verified Empirical Discoveries & Identified Limitations | |
| 1. **Compounded Domain & Generator Collapse for ViT:** ViT collapsed from 98.00% on facial StyleGAN down to **52.00% on general GenImage scenes** ($\text{ROC-AUC} = \mathbf{0.4673}$). This demonstrates that ViT's fine-tuned representations fail outside facial anatomy under a compounded semantic shift (faces $\to$ ImageNet objects) and generative shift (StyleGAN $\to$ diffusion). | |
| 2. **Zero False Positives vs. Severe Sensitivity Deficit:** Both UnivFD and ConvNeXt achieved **100.00% Specificity (0 False Positives)** on real ImageNet photographs. However, at default threshold $\tau = 0.50$, both suffer from severe sensitivity deficits (UnivFD recall: 15%; ConvNeXt recall: 13%). While ConvNeXt demonstrated strong latent discriminability ($\text{ROC-AUC} = \mathbf{0.7926}$, leading on Midjourney with 74.00% accuracy) and UnivFD on ADM ($\text{ROC-AUC} = \mathbf{0.9056}$), neither functions as an effective out-of-the-box detector without threshold recalibration. | |
| 3. **Shared Control Group Structure:** The Midjourney and ADM evaluations each evaluate 150 rows but share the **exact same 100 authentic ImageNet controls**. Total unique benchmark samples evaluated is 200 (not 300). | |
| 4. **Fusion Dynamics:** Prioritizing ViT in general-scene domains inherits ViT's 14 False Positives, whereas democratic fusion achieves 100% Specificity but collapses fake recall to 5.00% (missing 95/100 fakes). Democratic fusion accuracy (52.50%) is lower than UnivFD alone (57.50%) and ConvNeXt alone (56.50%). | |
| 5. **Documentation Discrepancy Noted:** Section 5 of [`docs/PHASE7_EXTERNAL_GENERALIZATION_REPORT.md`](file:///c:/Users/user/Desktop/multi%20model%20connection/docs/PHASE7_EXTERNAL_GENERALIZATION_REPORT.md) erroneously stated 91 three-way agreements (45.50%) and discrepant Cohen's Kappa figures. The verified three-way agreement count is **145 out of 200 (72.50%)** (86 agreed Real on authentic images + 59 agreed Real on fake images; 0 agreed Fake), with true $\kappa$ values of $0.0284$ (ViT vs UnivFD), $-0.0529$ (ViT vs ConvNeXt), and $0.0019$ (UnivFD vs ConvNeXt). | |
| 6. **Acceptance Verdict:** **Accepted as a Completed Benchmark Experiment with Documented Limitations**. | |
| ### 12.4 Phase 7 Master Artifacts | |
| - **Post-Execution Audit:** [`docs/PHASE7_POST_EXECUTION_AUDIT.md`](file:///c:/Users/user/Desktop/multi%20model%20connection/docs/PHASE7_POST_EXECUTION_AUDIT.md) | |
| - **Audit Runner Script:** [`scripts/audit_phase7_results.py`](file:///c:/Users/user/Desktop/multi%20model%20connection/scripts/audit_phase7_results.py) | |
| - **Dataset Audit:** [`docs/PHASE7_DATASET_AUDIT.md`](file:///c:/Users/user/Desktop/multi%20model%20connection/docs/PHASE7_DATASET_AUDIT.md) | |
| - **Evaluation Report:** [`docs/PHASE7_EXTERNAL_GENERALIZATION_REPORT.md`](file:///c:/Users/user/Desktop/multi%20model%20connection/docs/PHASE7_EXTERNAL_GENERALIZATION_REPORT.md) | |
| - **Manifest:** [`benchmark/external_generalization_manifest.csv`](file:///c:/Users/user/Desktop/multi%20model%20connection/benchmark/external_generalization_manifest.csv) | |
| - **Comparison CSV:** [`benchmark/results/phase7_model_comparison.csv`](file:///c:/Users/user/Desktop/multi%20model%20connection/benchmark/results/phase7_model_comparison.csv) | |
| - **Metrics JSON:** [`benchmark/results/phase7_combined_metrics.json`](file:///c:/Users/user/Desktop/multi%20model%20connection/benchmark/results/phase7_combined_metrics.json) | |
| --- | |
| ## 13. Root-Cause Investigation: Official Model Reference Parity Testing | |
| ### 13.1 Investigation Overview & Scope | |
| - **Investigation Deliverable:** [`docs/ROOT_CAUSE_MODEL_INFERENCE_INVESTIGATION.md`](file:///c:/Users/user/Desktop/multi%20model%20connection/docs/ROOT_CAUSE_MODEL_INFERENCE_INVESTIGATION.md) | |
| - **Audit Mode:** Scientific Parity Audit, Upstream Codebase Verification & Full Empirical 200-Sample Ablation (November 2026). | |
| - **Core Research Question:** Determine whether poor Phase 7 results (ViT 52.0%, UnivFD 57.5%, ConvNeXt 56.5%) stem from integration defects, preprocessing/extraction issues, output calculation bugs, or genuine domain/generator generalization boundaries. | |
| ### 13.2 Model-by-Model Status Assignments | |
| | Detector Model | Primary Status Assignment | Root Cause & Key Empirical Findings | | |
| | :--- | :---: | :--- | | |
| | **ViT (`dima806`)** | **OFFICIAL IMPLEMENTATION MATCHES** &<br>**GENUINE GENERALIZATION LIMITATION** | 100.0% exact match to upstream Hugging Face reference ($< 5 \times 10^{-7}$ diff across all 200 samples; 200/200 predictions match). Near random-chance performance (52.0% Acc, 0.4673 AUC) is caused exclusively by domain collapse from facial-only pretraining (Kaggle 140k). | | |
| | **ConvNeXt (DRCT `16_acc0.9993.pth`)** | **PREPROCESSING MISMATCH** (Midjourney)<br>& **GENUINE GENERALIZATION LIMITATION** (ADM) | Checkpoint and architecture match 100% (346 tensors). Adapter downsampling (`Resize 224`) destroyed high-frequency latent diffusion traces. Official native crop (`--is_crop`) restores Midjourney recall to **100.0% (50/50)**. However, ADM recall remains at **10.0% (5/50)**, matching the DRCT paper's published ICML 2024 GenImage benchmark (60.00% ADM accuracy). Overall accuracy under native crop rises to **77.00%** with **0.9270 ROC-AUC**. | | |
| | **UnivFD (CLIP ViT-L/14)** | **OFFICIAL IMPLEMENTATION MATCHES** (Arch/Weights)<br>& **GENUINE GENERALIZATION LIMITATION** (Primary Cause)<br>& **PREPROCESSING MISMATCH** (Local Divergence) | Weights match bit-for-bit. While adapter used standard CLIP preprocess instead of `validate.py` native crop, full-benchmark testing proves this had **net zero effect on fake recall** (15.00% under both). Failure is overwhelmingly driven by the 2018 ProGAN-only linear probe training. | | |
| | **Dataset Extraction** | **OFFICIAL IMPLEMENTATION MATCHES** | 200/200 files on disk match source Hugging Face Parquet shard byte-for-byte with zero alteration, re-encoding, or compression drift. | | |
| ### 13.3 Full 200-Sample Parity Comparison Matrix | |
| | Detector Model & Configuration | Total Accuracy | Real Specificity | Midjourney Recall | ADM Recall | Aggregate Fake Recall | ROC-AUC | Primary Failure Mechanism | | |
| | :--- | :---:| :---:| :---:| :---:| :---:| :--- | | |
| | **ViT (`dima806`)** — Phase 7 Adapter | 52.00% (104) | 86.00% (86) | 20.00% (10) | 16.00% (8) | 18.00% (18) | 0.4673 | Facial pretraining collapse | | |
| | **ViT (`dima806`)** — Official Hugging Face | 52.00% (104) | 86.00% (86) | 20.00% (10) | 16.00% (8) | 18.00% (18) | 0.4673 | Bit-for-bit parity ($< 5 \times 10^{-7}$) | | |
| | **UnivFD (`CLIP-L/14`)** — Phase 7 Adapter | 57.50% (115) | 100.00% (100) | 0.00% (0) | 30.00% (15) | 15.00% (15) | 0.6662 | ProGAN-only linear probe | | |
| | **UnivFD (`CLIP-L/14`)** — Official `validate.py` | 56.50% (113) | 98.00% (98) | 16.00% (8) | 14.00% (7) | 15.00% (15) | 0.8037 | Net-zero recall gain (15% vs 15%) | | |
| | **ConvNeXt (`DRCT`)** — Phase 7 Adapter | 56.50% (113) | 100.00% (100) | 22.00% (11) | 4.00% (2) | 13.00% (13) | 0.7926 | Preprocessing downsampling | | |
| | **ConvNeXt (`DRCT`)** — Official `--is_crop` | **77.00% (154)** | **99.00% (99)** | **100.00% (50)** | **10.00% (5)** | **55.00% (55)** | **0.9270** | MJ resolved (100%), ADM bounded (10%) | | |
| ### 13.4 Investigation Artifacts | |
| - **Primary Report:** [`docs/ROOT_CAUSE_MODEL_INFERENCE_INVESTIGATION.md`](file:///c:/Users/user/Desktop/multi%20model%20connection/docs/ROOT_CAUSE_MODEL_INFERENCE_INVESTIGATION.md) | |
| - **Audit Suite:** [`scripts/parity_audit/`](file:///c:/Users/user/Desktop/multi%20model%20connection/scripts/parity_audit/) | |
| - **Diagnostic JSON Records:** | |
| * `benchmark/results/parity_audit_convnext_full_200.json` | |
| * `benchmark/results/parity_audit_univfd_full_200.json` | |
| * `benchmark/results/parity_audit_vit_results.json` | |
| * `benchmark/results/parity_audit_dataset_extraction.json` | |
| * `benchmark/results/parity_audit_score_distributions.json` | |
| --- | |
| ## 14. Phase 7 Version 2: Corrected External Generalization Evaluation (Official Preprocessing Alignment) | |
| ### 14.1 Phase 7 V2 Overview & Execution Context | |
| - **Report Deliverable:** [`docs/PHASE7_V2_CORRECTED_EVALUATION_REPORT.md`](file:///c:/Users/user/Desktop/multi%20model%20connection/docs/PHASE7_V2_CORRECTED_EVALUATION_REPORT.md) | |
| - **Execution Mode:** Corrective Implementation & Empirical Re-evaluation (November 2026). | |
| - **Core Action:** Corrected preprocessing mismatches in `src/convnext_adapter.py` (restored native `DRCTNativeCropTransform` matching official `--is_crop`) and `src/univfd_adapter.py` (restored native `CenterCrop(224)` matching `validate.py`). Preserved ViT without modifications. | |
| - **Reference Parity:** Verified side-by-side on 30 deterministic samples against official author code with floating-point differences $< 10^{-6}$ (`parity_verification_audit.json`). | |
| - **Regression Tests:** 4/4 passed in [`tests/test_adapter_preprocessing_regression.py`](file:///c:/Users/user/Desktop/multi%20model%20connection/tests/test_adapter_preprocessing_regression.py). | |
| - **Evaluation Partition:** Exact same 200 manifest samples from GenImage (`benchmark/external_generalization_manifest.csv`: 100 Real ImageNet, 50 Midjourney, 50 ADM). Version 1 artifacts strictly preserved. | |
| ### 14.2 Empirical Performance Matrix (Phase 7 V2 vs. V1) | |
| | Detector / Strategy | Aggregate Acc (%) | 95% Wilson CI | Recall (Fake) | Specificity (Real) | F1-Score | ROC-AUC | Midjourney Recall | ADM Recall | V1 $\to$ V2 $\Delta$ Acc | | |
| | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | | |
| | **ViT (`dima806`)** | 52.00% | [45.10%, 58.82%] | 18.00% | 86.00% | 0.2727 | 0.4673 | 20.00% | 16.00% | $\pm 0.00\%$ | | |
| | **UnivFD (CLIP ViT-L/14)** | 56.50% | [49.57%, 63.18%] | 15.00% | 98.00% | 0.2564 | 0.8037 | 16.00% | 14.00% | $-1.00\%$ | | |
| | **ConvNeXt (DRCT / RAID)** | **77.00%** | **[70.69%, 82.29%]** | **55.00%** | **99.00%** | **0.7051** | **0.9271** | **100.00%** | 10.00% | $\mathbf{+20.50\%}$ | | |
| | **Ensemble (Majority Vote)** | 58.50% | [51.57%, 65.11%] | 18.00% | 99.00% | 0.3025 | 0.7662 | 30.00% | 6.00% | $+6.00\%$ | | |
| | **Ensemble (Mean Fake Prob)** | 61.50% | [54.60%, 67.97%] | 23.00% | **100.00%** | 0.3740 | 0.8309 | 40.00% | 6.00% | $+9.00\%$ | | |
| | **Ensemble (ViT Priority)** | 52.00% | [45.10%, 58.82%] | 18.00% | 86.00% | 0.2727 | 0.4673 | 20.00% | 16.00% | $\pm 0.00\%$ | | |
| ### 14.3 Key Empirical Discoveries & Architectural Implications | |
| 1. **Resolution of Midjourney Defect:** Restoring native cropping increased ConvNeXt Midjourney recall from 22.0% to **100.0% (50/50)** with cohort ROC-AUC of **0.9992**, raising overall accuracy to **77.00%** and ROC-AUC to **0.9271**. | |
| 2. **Persistence of ADM Generalization Boundary:** Even under official native crop, ConvNeXt detects only **10.0% (5/50)** of pixel-space guided diffusion (ADM) images, matching the ICML 2024 published limits (60.0% accuracy). | |
| 3. **Failure of Unweighted Democratic Fusion:** Unweighted fusion (Majority Vote: 58.50%, Mean Prob: 61.50%) significantly underperforms ConvNeXt alone (77.00%) because out-of-domain detectors (ViT and UnivFD) systematically outvote ConvNeXt on Midjourney samples. | |
| 4. **Architectural Recommendation:** Move from unweighted fusion to domain-conditioned routing: route faces to ViT; route general scenes exclusively to ConvNeXt. | |
| ### 14.4 Phase 7 Version 2 Artifacts | |
| - **Evaluation Report:** [`docs/PHASE7_V2_CORRECTED_EVALUATION_REPORT.md`](file:///c:/Users/user/Desktop/multi%20model%20connection/docs/PHASE7_V2_CORRECTED_EVALUATION_REPORT.md) | |
| - **Results Directory:** [`benchmark/results/phase7_v2_reference_preprocessing/`](file:///c:/Users/user/Desktop/multi%20model%20connection/benchmark/results/phase7_v2_reference_preprocessing/) | |
| - **Parity Audit:** `benchmark/results/phase7_v2_reference_preprocessing/parity_verification_audit.json` | |
| - **Comparison CSV:** `benchmark/results/phase7_v2_reference_preprocessing/phase7_v2_model_comparison.csv` | |
| - **Combined Metrics JSON:** `benchmark/results/phase7_v2_reference_preprocessing/phase7_v2_combined_metrics.json` | |