Multimodel_setup / docs /PHASE6_DATASET_SELECTION_REPORT.md
Thiru1122's picture
feat(deepfake-detection): comprehensive cross-model evaluation, parity audits, and Phase 7 v2 reference preprocessing
151f388
|
Raw History Blame Contribute Delete
5.9 kB

A newer version of the Gradio SDK is available: 6.32.0

Upgrade

Phase 6 Dataset Selection & Integrity Report

Project: Cross-Model Evaluation of Deepfake Image Detection Under Diverse Image Generation and Manipulation Conditions
Date: November 2026
Status: Frozen & Verified
Manifest Path: benchmark/phase6_test_manifest.csv


1. Executive Summary

As part of Phase 6 autonomous master goal execution, a rigorous, leak-free, balanced test set of 200 images (100 REAL, 100 FAKE) was selected from the primary test partition located at: C:\Users\user\Desktop\currently working\Dataset\Test

To ensure absolute scientific reproducibility and prevent optimistic evaluation bias from data leakage or repeated testing, the selection protocol strictly isolated these samples from all training data, validation splits, and prior benchmark runs (Phases 1 through 5, random_250, extended_predictions, and CommunityForensics).


2. Selection Methodology & Contamination Screening

2.1 Candidate Pool

  • Source Root: C:\Users\user\Desktop\currently working\Dataset\Test
    • Real subfolder: 5,413 candidate JPEG files (real_0.jpg to real_5412.jpg)
    • Fake subfolder: 5,492 candidate JPEG files (fake_0.jpg to fake_5491.jpg)
  • Dataset Lineage: Derived from the Kaggle 140k Real and Fake Faces dataset.
    • REAL images represent photographic captures of real human faces sourced from Flickr-Faces-HQ (FFHQ).
    • FAKE images represent synthetic facial images generated via StyleGAN / StyleGAN2.

2.2 Historical Leakage Screening & Exclusion Sets

Prior project audits, development phases, and historical scripts utilized several test subsets:

  1. Index Range 0 to 699: Screened and used in early project explorations and evaluation_manifest.csv (development val & held-out test).
  2. Prior Manifest Overlaps: benchmark/evaluation_manifest.csv utilized index ranges up to 1574 for held-out tests and smoke testing.
  3. Random 250 Evaluation: C:\Users\user\Desktop\currently working\random_250_evaluation_results.csv evaluated 250 samples spanning various index ranges.
  4. Extended Predictions: C:\Users\user\Desktop\currently working\extended_predictions.csv evaluated 300 samples across the dataset.
  5. CommunityForensics Evaluation: C:\Users\user\Desktop\currently working\CommunityForensics\community_predictions.csv.

To ensure complete statistical independence:

  • An exclusion database was compiled containing every filename referenced across all historical prediction logs, manifests, and scripts (touched_names).
  • A minimum index boundary of $\ge 2000$ was enforced for both Real and Fake candidate pools.
  • When traversing candidates starting at index 2000, each candidate was cross-referenced against touched_names.
  • Candidates identified in historical runs were disqualified:
    • real_2017.jpg, real_2041.jpg, real_2044.jpg (disqualified due to prior touch)
    • fake_2009.jpg, fake_2062.jpg (disqualified due to prior touch)
  • 100 consecutive pristine samples passing all filters were selected for each class:
    • REAL: real_2000.jpg through real_2102.jpg (100 selected, 3 excluded)
    • FAKE: fake_2000.jpg through fake_2101.jpg (100 selected, 2 excluded)

3. Data Integrity & Verification Audit

Every selected image underwent programmatic inspection verifying:

  1. File Readability & Decodability: Verified via PIL (Image.open(path).verify() and .convert('RGB')). Zero corruptions, zero truncated JPEGs, zero decode exceptions.
  2. Dimensions & Spatial Uniformity: Every image is exactly $256 \times 256$ pixels (width: 256, height: 256).
  3. Color Space: All images contain 3 8-bit RGB color channels (mode == 'RGB').
  4. Cryptographic Uniqueness: SHA-256 digests were computed for all 200 selected files. All 200 hashes are strictly unique (0 collision rate).
  5. Train / Validation Independence: Cross-verification against C:\Users\user\Desktop\currently working\Dataset\Train and Dataset\Validation confirmed zero filename collisions and zero hash collisions.

4. Manifest Schema

The manifest is stored at benchmark/phase6_test_manifest.csv. It contains 201 lines (1 header + 200 data rows) conforming to the following schema:

Column Type Description
sample_id String Unique identifier: phase6_test_real_001..100, phase6_test_fake_001..100
original_filename String Base filename, e.g., real_2000.jpg
relative_path String Project relative path: currently_working/Dataset/Test/...
absolute_path String Forward-slashed absolute file path on disk
ground_truth String Enum: REAL or FAKE
domain String facial
category String authentic_face or gan_face
source_dataset String Kaggle 140k Real and Fake Faces (FFHQ / StyleGAN)
generation_or_manipulation_method String Photographic Capture vs. StyleGAN / StyleGAN2 Generative Synthesis
format String JPEG
width Integer 256
height Integer 256
file_size_bytes Integer File size on disk in bytes
sha256 Hex String 64-character SHA-256 checksum
selection_reason String Pristine, decodable test sample with zero historical leakage

5. Verification Checklist

  • Exactly 100 REAL samples and 100 FAKE samples.
  • Balanced 50/50 test distribution ($N=200$).
  • Zero overlap with Phase 4 or Phase 5 evaluation sets.
  • Zero overlap with historical validation, random 250, or extended prediction logs.
  • 100% of images verified decodable, $256 \times 256$ RGB, non-corrupt.
  • Manifest frozen at benchmark/phase6_test_manifest.csv.