| --- |
| license: mit |
| pipeline_tag: video-classification |
| library_name: pytorch |
| tags: |
| - deepfake-detection |
| - audio-visual |
| - multimodal |
| - digital-forensics |
| - safetensors |
| - experiments |
| - benchmarks |
| - fakeavceleb |
| - cross-dataset |
| - leave-one-generator-out |
| --- |
| |
| # DAVID-Net: 135-Run Experimental Registry and Benchmark Archive |
|
|
| Official experimental archive, multi-seed checkpoints, training event streams, and evaluation telemetry for **DAVID-Net** (*Disentangled Audio-Visual Deepfake Network*). |
|
|
| This repository contains the persistent, uncompressed experimental record for the entire **135-run / 5-seed empirical campaign** across 27 experimental configurations (seeds $\mathcal{S} \in \{7, 42, 123, 456, 2024\}$). |
|
|
| * **Source Code Repository:** [GitHub: MIHMahmudEli/david-net-av-v2](https://github.com/MIHMahmudEli/david-net-av-v2) |
| * **Core Pretrained Model:** [Hugging Face: MIHMahmudEli/david-net-av-v2](https://huggingface.co/MIHMahmudEli/david-net-av-v2) |
| * **License:** MIT License |
| * **Format:** SafeTensors (`best_model/model.safetensors` per run), JSON configurations, CSV telemetry |
|
|
| --- |
|
|
| ## 📌 Experimental Campaign Overview |
|
|
| The empirical validation of DAVID-Net reflects a comprehensive matrix of **135 fully converged training runs** partitioned across four experimental stages: |
|
|
| 1. **Stage-0: Quadrant-Aware Contrastive Pretraining (QACP)** (*30 runs*): |
| * 6 self-supervised pretraining variants $\times$ 5 random seeds (seeds 7, 42, 123, 456, 2024). |
| * Generates proxy synthetic media from pristine recordings using self-blended video and vocoder copy-synthesis speech. |
| 2. **Stage-1: In-Domain Baselines & Proposed Architecture** (*35 runs*): |
| * 7 model architectures $\times$ 5 random seeds: Video Probe, Audio Probe, AASIST, EfficientNet-B4, Late Fusion, DAVID-Net Phase A, and DAVID-Net End-to-End Phase B. |
| * Evaluated under strict identity-disjoint FakeAVCeleb partitioning (0.0% train/test subject identity overlap). |
| 3. **Systematic Component Ablation Study** (*55 runs*): |
| * 11 ablation variants $\times$ 5 random seeds: isolating cross-modal synchronization (`no_sync`), subspace disentanglement (`no_disentangle`), localization heads (`no_loc`), modality dropout (`no_moddrop`), single-task loss (`single_task`), pretraining (`no_qacp`), and sub-component pretraining objectives. |
| 4. **Leave-One-Generator-Out (LOGO) Study** (*15 runs*): |
| * 3 withheld generator families (`logo_wav2lip`, `logo_fsgan`, `logo_faceswap`) $ imes$ 5 random seeds to measure zero-shot cross-generator generalization. |
|
|
| --- |
|
|
| ## 🔬 In-Domain Benchmark Summary ($N=5$ Seeds) |
|
|
| All 21 evaluated model configurations averaged over five independent seeds: |
|
|
| | Model / Configuration | Seeds | Video AUC | Audio AUC | Clip AUC | |
| | :--- | :---: | :---: | :---: | :---: | |
| | `aasist` | 5 | — | 1.000 ± 0.000 | 0.748 ± 0.004 | |
| | `audio-probe` | 5 | — | 0.999 ± 0.001 | 0.732 ± 0.001 | |
| | `davidnet` | 5 | 0.979 ± 0.003 | 0.999 ± 0.002 | 0.976 ± 0.006 | |
| | `davidnet-compose_quad` | 5 | 0.979 ± 0.004 | 1.000 ± 0.000 | 0.978 ± 0.003 | |
| | `davidnet-e2e` | 5 | 0.976 ± 0.005 | 1.000 ± 0.000 | 0.978 ± 0.005 | |
| | `davidnet-legacy` | 5 | 0.985 ± 0.004 | 1.000 ± 0.000 | 0.985 ± 0.002 | |
| | `davidnet-logo_faceswap` | 5 | 0.912 ± 0.006 | 1.000 ± 0.000 | 0.903 ± 0.003 | |
| | `davidnet-logo_fsgan` | 5 | 0.934 ± 0.004 | 1.000 ± 0.000 | 0.932 ± 0.007 | |
| | `davidnet-logo_wav2lip` | 5 | 0.425 ± 0.018 | 0.994 ± 0.010 | 0.919 ± 0.014 | |
| | `davidnet-no_disentangle` | 5 | 0.979 ± 0.002 | 1.000 ± 0.000 | 0.979 ± 0.001 | |
| | `davidnet-no_loc` | 5 | 0.980 ± 0.004 | 1.000 ± 0.000 | 0.981 ± 0.004 | |
| | `davidnet-no_moddrop` | 5 | 0.980 ± 0.003 | 1.000 ± 0.000 | 0.981 ± 0.004 | |
| | `davidnet-no_qacp` | 5 | 0.981 ± 0.002 | 1.000 ± 0.000 | 0.983 ± 0.003 | |
| | `davidnet-no_sync` | 5 | 0.973 ± 0.005 | 1.000 ± 0.000 | 0.975 ± 0.011 | |
| | `davidnet-qacp_no_copysynth` | 5 | 0.980 ± 0.003 | 0.999 ± 0.001 | 0.979 ± 0.006 | |
| | `davidnet-qacp_no_mismatch` | 5 | 0.976 ± 0.003 | 1.000 ± 0.000 | 0.977 ± 0.003 | |
| | `davidnet-qacp_no_selfblend` | 5 | 0.977 ± 0.006 | 1.000 ± 0.001 | 0.978 ± 0.006 | |
| | `davidnet-single_task` | 5 | 0.978 ± 0.003 | 1.000 ± 0.000 | 0.978 ± 0.005 | |
| | `effnet-b4` | 5 | 0.996 ± 0.001 | — | 0.926 ± 0.002 | |
| | `late-fusion` | 5 | 0.914 ± 0.020 | 1.000 ± 0.000 | 0.918 ± 0.015 | |
| | `video-probe` | 5 | 0.896 ± 0.008 | — | 0.830 ± 0.007 | |
|
|
| ### Key Findings |
| * **Proposed Architecture (DAVID-Net Phase A)**: Achieves **$0.976 \pm 0.006$ Clip AUC**, with **$0.977 \pm 0.003$ Video AUC** and **$0.999 \pm 0.002$ Audio AUC**, outperforming monolithic multimodal Late Fusion ($0.918 \pm 0.015$). |
| * **Cross-Generator Floor (LOGO)**: Maintains $\ge 90.3\%$ Clip AUC across all withheld generator families under leave-one-generator-out evaluation. |
| * **Empirical Calibration**: Attains low Expected Calibration Error (ECE, 15 bins) of **$0.032$** (visual) and **$0.024$** (acoustic) on the test split. |
|
|
| --- |
|
|
| ## 📁 Repository Organization & File Hierarchy |
|
|
| Each of the 135 completed runs is stored in a dedicated directory: |
|
|
| ```text |
| experiments/EXP_{ID}_{NAME}_s{SEED}/ |
| ├── best_model/ |
| │ ├── model.safetensors # Pinned model weights for this seed (SafeTensors) |
| │ ├── config.json # Architecture parameters |
| │ └── meta.json # Checkpoint metadata & validation performance |
| ├── configs/ |
| │ ├── config.json # Full experiment hyperparameter configuration |
| │ └── environment.json # Execution environment & hardware telemetry |
| ├── metrics/ |
| │ ├── training_history.csv # Per-epoch loss, learning rates, and validation metrics |
| │ └── eval_metrics.json # Final test evaluation metrics |
| └── figures/ # Generated ROC, PR, score distribution, and calibration plots |
| ``` |
|
|
| ### Global Reports & Aggregated Metrics (`reports/`) |
|
|
| * [`reports/raw_results.csv`](reports/raw_results.csv): Master tabular compilation of all 105 evaluated experimental runs. |
| * [`reports/summary.json`](reports/summary.json): Aggregated statistical metrics, bootstrap confidence intervals, and pairwise DeLong tests ($z$-scores and Holm-corrected $p$-values). |
| * `reports/tables/`: Machine-readable CSV tables for main results, ablations, external benchmark transfers, LOGO splits, and protocol comparisons. |
|
|
| --- |
|
|
| ## 💻 Programmatic Usage: Accessing Run Artifacts |
|
|
| You can inspect and load the weights, training logs, or configs of any run programmatically using `huggingface_hub`: |
|
|
| ```python |
| import pandas as pd |
| from huggingface_hub import hf_hub_download |
| from safetensors.torch import load_file |
| |
| repo_id = "MIHMahmudEli/davidnet-experiments" |
| |
| # 1. Download training history for EXP_011 (DAVID-Net Phase A, Seed 42) |
| history_path = hf_hub_download( |
| repo_id=repo_id, |
| filename="experiments/EXP_011_davidnet_s42/metrics/training_history.csv" |
| ) |
| df_history = pd.read_csv(history_path) |
| print("Training Epochs:", len(df_history)) |
| |
| # 2. Download checkpoint weights (SafeTensors) |
| weights_path = hf_hub_download( |
| repo_id=repo_id, |
| filename="experiments/EXP_011_davidnet_s42/best_model/model.safetensors" |
| ) |
| state_dict = load_file(weights_path) |
| print(f"Loaded {len(state_dict)} tensor keys.") |
| ``` |
|
|
| --- |
|
|
| ## 📋 Complete 135-Run Campaign Registry |
|
|
| | ID | Configuration | Seed | Status | Test Clip AUC | |
| |:---|:---|:---:|:---:|:---:| |
| | EXP_001 | video-probe | 42 | completed | 0.8373 | |
| | EXP_002 | audio-probe | 42 | completed | 0.7294 | |
| | EXP_003 | late-fusion | 42 | completed | 0.9307 | |
| | EXP_004 | aasist | 42 | completed | 0.7471 | |
| | EXP_005 | effnet-b4 | 42 | completed | 0.9232 | |
| | EXP_006 | qacp | 42 | completed | | |
| | EXP_007 | qacp-no_sync | 42 | completed | | |
| | EXP_008 | qacp-no_mismatch | 42 | completed | | |
| | EXP_009 | qacp-no_copysynth | 42 | completed | | |
| | EXP_010 | qacp-no_selfblend | 42 | completed | | |
| | EXP_011 | davidnet | 42 | completed | 0.9725 | |
| | EXP_012 | davidnet-no_qacp | 42 | completed | 0.9857 | |
| | EXP_013 | davidnet-no_sync | 42 | completed | 0.9810 | |
| | EXP_014 | davidnet-no_disentangle | 42 | completed | 0.9802 | |
| | EXP_015 | davidnet-no_loc | 42 | completed | 0.9799 | |
| | EXP_016 | davidnet-single_task | 42 | completed | 0.9747 | |
| | EXP_017 | davidnet-no_moddrop | 42 | completed | 0.9762 | |
| | EXP_018 | davidnet-compose_quad | 42 | completed | 0.9766 | |
| | EXP_019 | davidnet-qacp_no_mismatch | 42 | completed | 0.9796 | |
| | EXP_020 | davidnet-qacp_no_copysynth | 42 | completed | 0.9701 | |
| | EXP_021 | davidnet-qacp_no_selfblend | 42 | completed | 0.9708 | |
| | EXP_022 | davidnet-logo_wav2lip | 42 | completed | 0.9274 | |
| | EXP_023 | davidnet-logo_fsgan | 42 | completed | 0.9350 | |
| | EXP_024 | davidnet-logo_faceswap | 42 | completed | 0.9012 | |
| | EXP_025 | qacp-legacy | 42 | completed | | |
| | EXP_026 | davidnet-legacy | 42 | completed | 0.9826 | |
| | EXP_027 | davidnet-e2e | 42 | completed | 0.9793 | |
| | EXP_028 | video-probe | 123 | completed | 0.8349 | |
| | EXP_029 | audio-probe | 123 | completed | 0.7336 | |
| | EXP_030 | late-fusion | 123 | completed | 0.9273 | |
| | EXP_031 | aasist | 123 | completed | 0.7535 | |
| | EXP_032 | effnet-b4 | 123 | completed | 0.9259 | |
| | EXP_033 | qacp | 123 | completed | | |
| | EXP_034 | qacp-no_sync | 123 | completed | | |
| | EXP_035 | qacp-no_mismatch | 123 | completed | | |
| | EXP_036 | qacp-no_copysynth | 123 | completed | | |
| | EXP_037 | qacp-no_selfblend | 123 | completed | | |
| | EXP_038 | davidnet | 123 | completed | 0.9829 | |
| | EXP_039 | davidnet-no_qacp | 123 | completed | 0.9797 | |
| | EXP_040 | davidnet-no_sync | 123 | completed | 0.9578 | |
| | EXP_041 | davidnet-no_disentangle | 123 | completed | 0.9799 | |
| | EXP_042 | davidnet-no_loc | 123 | completed | 0.9768 | |
| | EXP_043 | davidnet-single_task | 123 | completed | 0.9779 | |
| | EXP_044 | davidnet-no_moddrop | 123 | completed | 0.9803 | |
| | EXP_045 | davidnet-compose_quad | 123 | completed | 0.9802 | |
| | EXP_046 | davidnet-qacp_no_mismatch | 123 | completed | 0.9739 | |
| | EXP_047 | davidnet-qacp_no_copysynth | 123 | completed | 0.9752 | |
| | EXP_048 | davidnet-qacp_no_selfblend | 123 | completed | 0.9770 | |
| | EXP_049 | davidnet-logo_wav2lip | 123 | completed | 0.8979 | |
| | EXP_050 | davidnet-logo_fsgan | 123 | completed | 0.9330 | |
| | EXP_051 | davidnet-logo_faceswap | 123 | completed | 0.9083 | |
| | EXP_052 | qacp-legacy | 123 | completed | | |
| | EXP_053 | davidnet-legacy | 123 | completed | 0.9865 | |
| | EXP_054 | davidnet-e2e | 123 | completed | 0.9723 | |
| | EXP_055 | video-probe | 456 | completed | 0.8204 | |
| | EXP_056 | audio-probe | 456 | completed | 0.7319 | |
| | EXP_057 | late-fusion | 456 | completed | 0.9004 | |
| | EXP_058 | aasist | 456 | completed | 0.7447 | |
| | EXP_059 | effnet-b4 | 456 | completed | 0.9274 | |
| | EXP_060 | qacp | 456 | completed | | |
| | EXP_061 | qacp-no_sync | 456 | completed | | |
| | EXP_062 | qacp-no_mismatch | 456 | completed | | |
| | EXP_063 | qacp-no_copysynth | 456 | completed | | |
| | EXP_064 | qacp-no_selfblend | 456 | completed | | |
| | EXP_065 | davidnet | 456 | completed | 0.9726 | |
| | EXP_066 | davidnet-no_qacp | 456 | completed | 0.9828 | |
| | EXP_067 | davidnet-no_sync | 456 | completed | 0.9814 | |
| | EXP_068 | davidnet-no_disentangle | 456 | completed | 0.9776 | |
| | EXP_069 | davidnet-no_loc | 456 | completed | 0.9858 | |
| | EXP_070 | davidnet-single_task | 456 | completed | 0.9819 | |
| | EXP_071 | davidnet-no_moddrop | 456 | completed | 0.9855 | |
| | EXP_072 | davidnet-compose_quad | 456 | completed | 0.9828 | |
| | EXP_073 | davidnet-qacp_no_mismatch | 456 | completed | 0.9802 | |
| | EXP_074 | davidnet-qacp_no_copysynth | 456 | completed | 0.9833 | |
| | EXP_075 | davidnet-qacp_no_selfblend | 456 | completed | 0.9840 | |
| | EXP_076 | davidnet-logo_wav2lip | 456 | completed | 0.9328 | |
| | EXP_077 | davidnet-logo_fsgan | 456 | completed | 0.9316 | |
| | EXP_078 | davidnet-logo_faceswap | 456 | completed | 0.9054 | |
| | EXP_079 | qacp-legacy | 456 | completed | | |
| | EXP_080 | davidnet-legacy | 456 | completed | 0.9877 | |
| | EXP_081 | davidnet-e2e | 456 | completed | 0.9748 | |
| | EXP_082 | audio-probe | 7 | completed | 0.7315 | |
| | EXP_083 | qacp-no_selfblend | 7 | completed | | |
| | EXP_084 | late-fusion | 2024 | completed | 0.9032 | |
| | EXP_085 | qacp-no_copysynth | 2024 | completed | | |
| | EXP_086 | davidnet-no_qacp | 2024 | completed | 0.9806 | |
| | EXP_087 | qacp-no_mismatch | 2024 | completed | | |
| | EXP_088 | qacp-no_copysynth | 7 | completed | | |
| | EXP_089 | qacp | 2024 | completed | | |
| | EXP_090 | video-probe | 2024 | completed | 0.8260 | |
| | EXP_091 | qacp-no_selfblend | 2024 | completed | | |
| | EXP_092 | video-probe | 7 | completed | 0.8334 | |
| | EXP_093 | late-fusion | 7 | completed | 0.9302 | |
| | EXP_094 | aasist | 2024 | completed | 0.7455 | |
| | EXP_095 | qacp-no_sync | 2024 | completed | | |
| | EXP_096 | qacp-legacy | 2024 | completed | | |
| | EXP_097 | davidnet-qacp_no_mismatch | 2024 | completed | 0.9761 | |
| | EXP_098 | davidnet-logo_wav2lip | 2024 | completed | 0.9096 | |
| | EXP_099 | davidnet-qacp_no_selfblend | 7 | completed | 0.9742 | |
| | EXP_100 | qacp | 7 | completed | | |
| | EXP_101 | qacp-no_sync | 7 | completed | | |
| | EXP_102 | qacp-no_mismatch | 7 | completed | | |
| | EXP_103 | qacp-legacy | 7 | completed | | |
| | EXP_104 | effnet-b4 | 2024 | completed | 0.9279 | |
| | EXP_105 | davidnet-legacy | 2024 | completed | 0.9831 | |
| | EXP_106 | davidnet-compose_quad | 7 | completed | 0.9740 | |
| | EXP_107 | davidnet-logo_fsgan | 7 | completed | 0.9401 | |
| | EXP_108 | davidnet-no_moddrop | 7 | completed | 0.9782 | |
| | EXP_109 | audio-probe | 2024 | completed | 0.7313 | |
| | EXP_110 | davidnet-logo_wav2lip | 7 | completed | 0.9256 | |
| | EXP_111 | davidnet-qacp_no_mismatch | 7 | completed | 0.9750 | |
| | EXP_112 | davidnet-no_loc | 7 | completed | 0.9804 | |
| | EXP_113 | davidnet-qacp_no_selfblend | 2024 | completed | 0.9837 | |
| | EXP_114 | davidnet | 2024 | completed | 0.9694 | |
| | EXP_115 | davidnet-no_disentangle | 2024 | completed | 0.9799 | |
| | EXP_116 | davidnet-no_moddrop | 2024 | completed | 0.9841 | |
| | EXP_117 | davidnet-no_sync | 2024 | completed | 0.9826 | |
| | EXP_118 | aasist | 7 | completed | 0.7507 | |
| | EXP_119 | davidnet-no_disentangle | 7 | completed | 0.9778 | |
| | EXP_120 | davidnet-no_sync | 7 | completed | 0.9700 | |
| | EXP_121 | davidnet-legacy | 7 | completed | 0.9831 | |
| | EXP_122 | davidnet-qacp_no_copysynth | 7 | completed | 0.9858 | |
| | EXP_123 | davidnet-compose_quad | 2024 | completed | 0.9770 | |
| | EXP_124 | davidnet-no_qacp | 7 | completed | 0.9852 | |
| | EXP_125 | davidnet-qacp_no_copysynth | 2024 | completed | 0.9795 | |
| | EXP_126 | davidnet-logo_fsgan | 2024 | completed | 0.9220 | |
| | EXP_127 | davidnet-no_loc | 2024 | completed | 0.9838 | |
| | EXP_128 | davidnet-single_task | 2024 | completed | 0.9841 | |
| | EXP_129 | davidnet-e2e | 2024 | completed | 0.9845 | |
| | EXP_130 | davidnet-logo_faceswap | 7 | completed | 0.9010 | |
| | EXP_131 | davidnet-single_task | 7 | completed | 0.9730 | |
| | EXP_132 | effnet-b4 | 7 | completed | 0.9268 | |
| | EXP_133 | davidnet | 7 | completed | 0.9807 | |
| | EXP_134 | davidnet-e2e | 7 | completed | 0.9797 | |
| | EXP_135 | davidnet-logo_faceswap | 2024 | completed | 0.9008 | |
| |
| --- |
| |
| ## ⚖️ Ethics & Responsible AI Statement |
| |
| All experimental evaluations in this repository were conducted under strict academic research ethics. Seven benchmark corpora were acquired under their respective research end-user licensing agreements; no raw biometric media is redistributed. The system is strictly defensive by design and contains no generative synthesis capability. |
| |
