MIHMahmudEli/davidnet-experiments

DAVID-Net: 135-Run Experimental Registry and Benchmark Archive

Official experimental archive, multi-seed checkpoints, training event streams, and evaluation telemetry for DAVID-Net (Disentangled Audio-Visual Deepfake Network).

This repository contains the persistent, uncompressed experimental record for the entire 135-run / 5-seed empirical campaign across 27 experimental configurations (seeds $\mathcal{S} \in {7, 42, 123, 456, 2024}$).

• Source Code Repository: GitHub: MIHMahmudEli/david-net-av-v2
• Core Pretrained Model: Hugging Face: MIHMahmudEli/david-net-av-v2
• License: MIT License
• Format: SafeTensors (best_model/model.safetensors per run), JSON configurations, CSV telemetry

---

📌 Experimental Campaign Overview

The empirical validation of DAVID-Net reflects a comprehensive matrix of 135 fully converged training runs partitioned across four experimental stages:

1. Stage-0: Quadrant-Aware Contrastive Pretraining (QACP) (30 runs):
   • 6 self-supervised pretraining variants $\times$ 5 random seeds (seeds 7, 42, 123, 456, 2024).
   • Generates proxy synthetic media from pristine recordings using self-blended video and vocoder copy-synthesis speech.
2. Stage-1: In-Domain Baselines & Proposed Architecture (35 runs):
   • 7 model architectures $\times$ 5 random seeds: Video Probe, Audio Probe, AASIST, EfficientNet-B4, Late Fusion, DAVID-Net Phase A, and DAVID-Net End-to-End Phase B.
   • Evaluated under strict identity-disjoint FakeAVCeleb partitioning (0.0% train/test subject identity overlap).
3. Systematic Component Ablation Study (55 runs):
   • 11 ablation variants $\times$ 5 random seeds: isolating cross-modal synchronization (no_sync), subspace disentanglement (no_disentangle), localization heads (no_loc), modality dropout (no_moddrop), single-task loss (single_task), pretraining (no_qacp), and sub-component pretraining objectives.
4. Leave-One-Generator-Out (LOGO) Study (15 runs):
   • 3 withheld generator families (logo_wav2lip, logo_fsgan, logo_faceswap) $ imes$ 5 random seeds to measure zero-shot cross-generator generalization.

---

🔬 In-Domain Benchmark Summary ($N=5$ Seeds)

All 21 evaluated model configurations averaged over five independent seeds:

Model / Configuration	Seeds	Video AUC	Audio AUC	Clip AUC
aasist	5	—	1.000 ± 0.000	0.748 ± 0.004
audio-probe	5	—	0.999 ± 0.001	0.732 ± 0.001
davidnet	5	0.979 ± 0.003	0.999 ± 0.002	0.976 ± 0.006
davidnet-compose_quad	5	0.979 ± 0.004	1.000 ± 0.000	0.978 ± 0.003
davidnet-e2e	5	0.976 ± 0.005	1.000 ± 0.000	0.978 ± 0.005
davidnet-legacy	5	0.985 ± 0.004	1.000 ± 0.000	0.985 ± 0.002
davidnet-logo_faceswap	5	0.912 ± 0.006	1.000 ± 0.000	0.903 ± 0.003
davidnet-logo_fsgan	5	0.934 ± 0.004	1.000 ± 0.000	0.932 ± 0.007
davidnet-logo_wav2lip	5	0.425 ± 0.018	0.994 ± 0.010	0.919 ± 0.014
davidnet-no_disentangle	5	0.979 ± 0.002	1.000 ± 0.000	0.979 ± 0.001
davidnet-no_loc	5	0.980 ± 0.004	1.000 ± 0.000	0.981 ± 0.004
davidnet-no_moddrop	5	0.980 ± 0.003	1.000 ± 0.000	0.981 ± 0.004
davidnet-no_qacp	5	0.981 ± 0.002	1.000 ± 0.000	0.983 ± 0.003
davidnet-no_sync	5	0.973 ± 0.005	1.000 ± 0.000	0.975 ± 0.011
davidnet-qacp_no_copysynth	5	0.980 ± 0.003	0.999 ± 0.001	0.979 ± 0.006
davidnet-qacp_no_mismatch	5	0.976 ± 0.003	1.000 ± 0.000	0.977 ± 0.003
davidnet-qacp_no_selfblend	5	0.977 ± 0.006	1.000 ± 0.001	0.978 ± 0.006
davidnet-single_task	5	0.978 ± 0.003	1.000 ± 0.000	0.978 ± 0.005
effnet-b4	5	0.996 ± 0.001	—	0.926 ± 0.002
late-fusion	5	0.914 ± 0.020	1.000 ± 0.000	0.918 ± 0.015
video-probe	5	0.896 ± 0.008	—	0.830 ± 0.007

Key Findings

• Proposed Architecture (DAVID-Net Phase A): Achieves $0.976 \pm 0.006$ Clip AUC, with $0.977 \pm 0.003$ Video AUC and $0.999 \pm 0.002$ Audio AUC, outperforming monolithic multimodal Late Fusion ($0.918 \pm 0.015$).
• Cross-Generator Floor (LOGO): Maintains $\ge 90.3%$ Clip AUC across all withheld generator families under leave-one-generator-out evaluation.
• Empirical Calibration: Attains low Expected Calibration Error (ECE, 15 bins) of $0.032$ (visual) and $0.024$ (acoustic) on the test split.

---

📁 Repository Organization & File Hierarchy

Each of the 135 completed runs is stored in a dedicated directory:

experiments/EXP_{ID}_{NAME}_s{SEED}/
├── best_model/
│   ├── model.safetensors       # Pinned model weights for this seed (SafeTensors)
│   ├── config.json             # Architecture parameters
│   └── meta.json               # Checkpoint metadata & validation performance
├── configs/
│   ├── config.json             # Full experiment hyperparameter configuration
│   └── environment.json        # Execution environment & hardware telemetry
├── metrics/
│   ├── training_history.csv    # Per-epoch loss, learning rates, and validation metrics
│   └── eval_metrics.json       # Final test evaluation metrics
└── figures/                    # Generated ROC, PR, score distribution, and calibration plots

Global Reports & Aggregated Metrics (reports/)

• reports/raw_results.csv: Master tabular compilation of all 105 evaluated experimental runs.
• reports/summary.json: Aggregated statistical metrics, bootstrap confidence intervals, and pairwise DeLong tests ($z$-scores and Holm-corrected $p$-values).
• reports/tables/: Machine-readable CSV tables for main results, ablations, external benchmark transfers, LOGO splits, and protocol comparisons.

---

💻 Programmatic Usage: Accessing Run Artifacts

You can inspect and load the weights, training logs, or configs of any run programmatically using huggingface_hub:

import pandas as pd
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file

repo_id = "MIHMahmudEli/davidnet-experiments"

# 1. Download training history for EXP_011 (DAVID-Net Phase A, Seed 42)
history_path = hf_hub_download(
    repo_id=repo_id,
    filename="experiments/EXP_011_davidnet_s42/metrics/training_history.csv"
)
df_history = pd.read_csv(history_path)
print("Training Epochs:", len(df_history))

# 2. Download checkpoint weights (SafeTensors)
weights_path = hf_hub_download(
    repo_id=repo_id,
    filename="experiments/EXP_011_davidnet_s42/best_model/model.safetensors"
)
state_dict = load_file(weights_path)
print(f"Loaded {len(state_dict)} tensor keys.")

---

📋 Complete 135-Run Campaign Registry

ID	Configuration	Seed	Status	Test Clip AUC
EXP_001	video-probe	42	completed	0.8373
EXP_002	audio-probe	42	completed	0.7294
EXP_003	late-fusion	42	completed	0.9307
EXP_004	aasist	42	completed	0.7471
EXP_005	effnet-b4	42	completed	0.9232
EXP_006	qacp	42	completed
EXP_007	qacp-no_sync	42	completed
EXP_008	qacp-no_mismatch	42	completed
EXP_009	qacp-no_copysynth	42	completed
EXP_010	qacp-no_selfblend	42	completed
EXP_011	davidnet	42	completed	0.9725
EXP_012	davidnet-no_qacp	42	completed	0.9857
EXP_013	davidnet-no_sync	42	completed	0.9810
EXP_014	davidnet-no_disentangle	42	completed	0.9802
EXP_015	davidnet-no_loc	42	completed	0.9799
EXP_016	davidnet-single_task	42	completed	0.9747
EXP_017	davidnet-no_moddrop	42	completed	0.9762
EXP_018	davidnet-compose_quad	42	completed	0.9766
EXP_019	davidnet-qacp_no_mismatch	42	completed	0.9796
EXP_020	davidnet-qacp_no_copysynth	42	completed	0.9701
EXP_021	davidnet-qacp_no_selfblend	42	completed	0.9708
EXP_022	davidnet-logo_wav2lip	42	completed	0.9274
EXP_023	davidnet-logo_fsgan	42	completed	0.9350
EXP_024	davidnet-logo_faceswap	42	completed	0.9012
EXP_025	qacp-legacy	42	completed
EXP_026	davidnet-legacy	42	completed	0.9826
EXP_027	davidnet-e2e	42	completed	0.9793
EXP_028	video-probe	123	completed	0.8349
EXP_029	audio-probe	123	completed	0.7336
EXP_030	late-fusion	123	completed	0.9273
EXP_031	aasist	123	completed	0.7535
EXP_032	effnet-b4	123	completed	0.9259
EXP_033	qacp	123	completed
EXP_034	qacp-no_sync	123	completed
EXP_035	qacp-no_mismatch	123	completed
EXP_036	qacp-no_copysynth	123	completed
EXP_037	qacp-no_selfblend	123	completed
EXP_038	davidnet	123	completed	0.9829
EXP_039	davidnet-no_qacp	123	completed	0.9797
EXP_040	davidnet-no_sync	123	completed	0.9578
EXP_041	davidnet-no_disentangle	123	completed	0.9799
EXP_042	davidnet-no_loc	123	completed	0.9768
EXP_043	davidnet-single_task	123	completed	0.9779
EXP_044	davidnet-no_moddrop	123	completed	0.9803
EXP_045	davidnet-compose_quad	123	completed	0.9802
EXP_046	davidnet-qacp_no_mismatch	123	completed	0.9739
EXP_047	davidnet-qacp_no_copysynth	123	completed	0.9752
EXP_048	davidnet-qacp_no_selfblend	123	completed	0.9770
EXP_049	davidnet-logo_wav2lip	123	completed	0.8979
EXP_050	davidnet-logo_fsgan	123	completed	0.9330
EXP_051	davidnet-logo_faceswap	123	completed	0.9083
EXP_052	qacp-legacy	123	completed
EXP_053	davidnet-legacy	123	completed	0.9865
EXP_054	davidnet-e2e	123	completed	0.9723
EXP_055	video-probe	456	completed	0.8204
EXP_056	audio-probe	456	completed	0.7319
EXP_057	late-fusion	456	completed	0.9004
EXP_058	aasist	456	completed	0.7447
EXP_059	effnet-b4	456	completed	0.9274
EXP_060	qacp	456	completed
EXP_061	qacp-no_sync	456	completed
EXP_062	qacp-no_mismatch	456	completed
EXP_063	qacp-no_copysynth	456	completed
EXP_064	qacp-no_selfblend	456	completed
EXP_065	davidnet	456	completed	0.9726
EXP_066	davidnet-no_qacp	456	completed	0.9828
EXP_067	davidnet-no_sync	456	completed	0.9814
EXP_068	davidnet-no_disentangle	456	completed	0.9776
EXP_069	davidnet-no_loc	456	completed	0.9858
EXP_070	davidnet-single_task	456	completed	0.9819
EXP_071	davidnet-no_moddrop	456	completed	0.9855
EXP_072	davidnet-compose_quad	456	completed	0.9828
EXP_073	davidnet-qacp_no_mismatch	456	completed	0.9802
EXP_074	davidnet-qacp_no_copysynth	456	completed	0.9833
EXP_075	davidnet-qacp_no_selfblend	456	completed	0.9840
EXP_076	davidnet-logo_wav2lip	456	completed	0.9328
EXP_077	davidnet-logo_fsgan	456	completed	0.9316
EXP_078	davidnet-logo_faceswap	456	completed	0.9054
EXP_079	qacp-legacy	456	completed
EXP_080	davidnet-legacy	456	completed	0.9877
EXP_081	davidnet-e2e	456	completed	0.9748
EXP_082	audio-probe	7	completed	0.7315
EXP_083	qacp-no_selfblend	7	completed
EXP_084	late-fusion	2024	completed	0.9032
EXP_085	qacp-no_copysynth	2024	completed
EXP_086	davidnet-no_qacp	2024	completed	0.9806
EXP_087	qacp-no_mismatch	2024	completed
EXP_088	qacp-no_copysynth	7	completed
EXP_089	qacp	2024	completed
EXP_090	video-probe	2024	completed	0.8260
EXP_091	qacp-no_selfblend	2024	completed
EXP_092	video-probe	7	completed	0.8334
EXP_093	late-fusion	7	completed	0.9302
EXP_094	aasist	2024	completed	0.7455
EXP_095	qacp-no_sync	2024	completed
EXP_096	qacp-legacy	2024	completed
EXP_097	davidnet-qacp_no_mismatch	2024	completed	0.9761
EXP_098	davidnet-logo_wav2lip	2024	completed	0.9096
EXP_099	davidnet-qacp_no_selfblend	7	completed	0.9742
EXP_100	qacp	7	completed
EXP_101	qacp-no_sync	7	completed
EXP_102	qacp-no_mismatch	7	completed
EXP_103	qacp-legacy	7	completed
EXP_104	effnet-b4	2024	completed	0.9279
EXP_105	davidnet-legacy	2024	completed	0.9831
EXP_106	davidnet-compose_quad	7	completed	0.9740
EXP_107	davidnet-logo_fsgan	7	completed	0.9401
EXP_108	davidnet-no_moddrop	7	completed	0.9782
EXP_109	audio-probe	2024	completed	0.7313
EXP_110	davidnet-logo_wav2lip	7	completed	0.9256
EXP_111	davidnet-qacp_no_mismatch	7	completed	0.9750
EXP_112	davidnet-no_loc	7	completed	0.9804
EXP_113	davidnet-qacp_no_selfblend	2024	completed	0.9837
EXP_114	davidnet	2024	completed	0.9694
EXP_115	davidnet-no_disentangle	2024	completed	0.9799
EXP_116	davidnet-no_moddrop	2024	completed	0.9841
EXP_117	davidnet-no_sync	2024	completed	0.9826
EXP_118	aasist	7	completed	0.7507
EXP_119	davidnet-no_disentangle	7	completed	0.9778
EXP_120	davidnet-no_sync	7	completed	0.9700
EXP_121	davidnet-legacy	7	completed	0.9831
EXP_122	davidnet-qacp_no_copysynth	7	completed	0.9858
EXP_123	davidnet-compose_quad	2024	completed	0.9770
EXP_124	davidnet-no_qacp	7	completed	0.9852
EXP_125	davidnet-qacp_no_copysynth	2024	completed	0.9795
EXP_126	davidnet-logo_fsgan	2024	completed	0.9220
EXP_127	davidnet-no_loc	2024	completed	0.9838
EXP_128	davidnet-single_task	2024	completed	0.9841
EXP_129	davidnet-e2e	2024	completed	0.9845
EXP_130	davidnet-logo_faceswap	7	completed	0.9010
EXP_131	davidnet-single_task	7	completed	0.9730
EXP_132	effnet-b4	7	completed	0.9268
EXP_133	davidnet	7	completed	0.9807
EXP_134	davidnet-e2e	7	completed	0.9797
EXP_135	davidnet-logo_faceswap	2024	completed	0.9008

---

⚖️ Ethics & Responsible AI Statement

All experimental evaluations in this repository were conducted under strict academic research ethics. Seven benchmark corpora were acquired under their respective research end-user licensing agreements; no raw biometric media is redistributed. The system is strictly defensive by design and contains no generative synthesis capability.