DAVID-Net: Disentangled Audio-Visual Forensic Attribution under Identity-Disjoint Evaluation

Official repository for DAVID-Net (Decoupled Audio-Visual Forensic Network), an asymmetric cross-modal transformer framework for fine-grained four-quadrant deepfake attribution, missing-modality fault tolerance, and millisecond-level temporal boundary localization.

Paper GitHub Datasets Experiments Live Demo API License: MIT


πŸ“Œ Framework Summary

Conventional deepfake detectors collapse multimodal evidence into an opaque binary verdict ("real vs. fake"). In digital forensics, judicial investigations, and platform moderation, an unadulterated binary metric yields little diagnostic value: it cannot identify which communication channel was manipulated, when the manipulation took place, or provide calibrated confidence bounds. Furthermore, models relying strictly on audio-visual temporal synchronization fail completely on double-fakes (FVFA) where synthetic video is lip-synced to cloned speech.

DAVID-Net reformulates deepfake detection as a Four-Quadrant Authenticity Attribution task:

Quadrant Visual Stream Acoustic Stream Forensic Diagnosis
RVRA Real ($y_v=0$) Real ($y_a=0$) Authentic Media: Genuine visual recording and authentic speech.
RVFA Real ($y_v=0$) Fake ($y_a=1$) Voice Cloning / Dubbing: Authentic video paired with cloned/synthetic speech.
FVRA Fake ($y_v=1$) Real ($y_a=0$) Facial Manipulation: Face swap or reenactment over genuine speech.
FVFA Fake ($y_v=1$) Fake ($y_a=1$) Dual Manipulation: End-to-end synthetic or spliced double-fake.

πŸ—οΈ Architecture & Methodological Highlights

  1. Decoupled Multimodal Representation:
    • Visual Stream: Aligned face crops and perioral mouth crops encoded via VideoMAE ViT-Base ($128 \times 768$ spatio-temporal tokens).
    • Acoustic Stream: 16,kHz raw speech waveforms encoded via WavLM Base+ ($50 \times 768$ acoustic tokens).
    • Hidden representation dimension $d = 768$.
  2. Geometric Subspace Disentanglement:
    • Enforces strict subspace orthogonality: $z_v \perp z_c$ and $z_a \perp z_c$ ($\mathcal{L}_{\text{ortho}} = |\cos(z_v, z_c^{(v)})| + |\cos(z_a, z_c^{(a)})|$).
    • Prevents cross-channel shortcut memorization and eliminates the double-fake trap.
  3. Fault-Tolerant Learnable Null Tokens:
    • Ingests silent video or isolated audio recordings without crashing or random guessing via learnable tokens $\mathbf{e}{\text{null}}^{(v)}, \mathbf{e}{\text{null}}^{(a)} \in \mathbb{R}^{1 \times 768}$.
  4. Windowed Temporal Synchronization:
    • Inner-product temporal agreement $s_t = \tilde{V}_t^\top \tilde{A}_t$ trained via bidirectional InfoNCE on authentic media.
  5. Decoupled Multi-Task Attribution Heads:
    • Independent projection heads for video veracity ($H_v$), audio veracity ($H_a$), four-quadrant attribution ($H_{quad}$), and frame-level boundary localization ($H_{loc}$).

πŸ“Š Empirical Evaluation Benchmarks (5-Seed Campaign)

All metrics reflect multi-seed evaluation across five independent seeds $\mathcal{S} \in {7, 42, 123, 456, 2024}$ from a 160-run evaluation campaign.

1. In-Domain Benchmark on FakeAVCeleb (Strict Identity-Disjoint Split)

Architecture Parameters Clip AUC Video AUC Audio AUC Macro-$F_1$ (%) Balanced Acc (%)
Audio Probe (WavLM) 94.7M $0.842 \pm 0.007$ β€” $0.999 \pm 0.002$ β€” β€”
Video Probe (VideoMAE) 86.6M $0.902 \pm 0.005$ $0.966 \pm 0.004$ β€” β€” β€”
AASIST 0.3M $0.748 \pm 0.004$ β€” $0.998 \pm 0.001$ β€” β€”
EfficientNet-B4 19.3M $0.926 \pm 0.002$ $0.991 \pm 0.001$ β€” β€” β€”
Late Fusion (V+A) 181.3M $0.918 \pm 0.015$ $0.963 \pm 0.007$ $0.998 \pm 0.002$ $79.7 \pm 3.2$ $80.1 \pm 2.8$
AVFF 186.2M $\mathbf{0.981 \pm 0.003}$ β€” β€” β€” β€”
DAVID-Net (Phase A, Frozen) 75.6M $0.976 \pm 0.006$ $0.979 \pm 0.003$ $\mathbf{0.999 \pm 0.002}$ $\mathbf{91.8 \pm 1.7}$ $\mathbf{91.3 \pm 1.0}$
DAVID-Net (Phase B, E2E) 105.5M $0.978 \pm 0.005$ $\mathbf{0.982 \pm 0.004}$ $\mathbf{0.999 \pm 0.001}$ $\mathbf{92.4 \pm 1.2}$ $\mathbf{92.0 \pm 0.8}$

Superiority over Late Fusion, EfficientNet-B4, and AASIST is statistically significant across all 5 seeds ($p < 0.05$, paired DeLong test with Holm--Bonferroni correction).

2. Zero-Shot Cross-Dataset Generalization (No Target Domain Adaptation)

Target Benchmark Modality Samples AASIST / Baseline DAVID-Net Gain ($\Delta$)
ASVspoof 2019 LA Audio 25,380 $0.648 \pm 0.013$ $\mathbf{0.839 \pm 0.019}$ +19.1%
In-the-Wild Audio Audio 31,779 $0.569 \pm 0.027$ $\mathbf{0.691 \pm 0.021}$ +12.2%
Celeb-DF v2 Video 518 $0.589 \pm 0.012$ $\mathbf{0.682 \pm 0.029}$ (Ph. B) +8.3%
DFDC-10 Multimodal 5,000 $0.619 \pm 0.024$ $\mathbf{0.647 \pm 0.020}$ +2.8%
DeepFakeTIMIT Video 640 $0.982 \pm 0.005$ $\mathbf{0.998 \pm 0.002}$ +1.6%
WaveFake Audio 11,000 $0.098 \pm 0.015$ $0.113 \pm 0.018$ (Vocoder Shift)

3. Leave-One-Generator-Out (LOGO) Resilience

Held-Out Generator Synthesis Type Video AUC Audio AUC Clip AUC Resilience Floor
logo_wav2lip Lip-Sync Reenactment $0.425 \pm 0.028$ $0.994 \pm 0.002$ $0.919 \pm 0.008$ $\ge 90.0%$
logo_fsgan Face Swapping (GAN) $0.884 \pm 0.016$ $0.999 \pm 0.001$ $0.902 \pm 0.012$ $\ge 90.0%$
logo_faceswap Autoencoder Face Swap $0.941 \pm 0.011$ $0.999 \pm 0.001$ $0.932 \pm 0.009$ $\ge 90.0%$
Average LOGO β€” β€” β€” $\mathbf{0.918 \pm 0.014}$ $\ge \mathbf{90.0%}$

⚑ Quickstart & Model Inference

1. Download Checkpoint via huggingface_hub

from huggingface_hub import hf_hub_download

# Download official release weights (~341 MB) and architecture config
model_path = hf_hub_download(
    repo_id="MIHMahmudEli/david-net-av",
    filename="david_net_best.safetensors"
)
config_path = hf_hub_download(
    repo_id="MIHMahmudEli/david-net-av",
    filename="config.json"
)
print(f"Model downloaded: {model_path}")

2. Standalone Inference CLI (from GitHub)

git clone https://github.com/MIHMahmudEli/david-net-av.git
cd david-net-av
pip install -r requirements.txt

# Multimodal analysis
python predict.py --input sample_video.mp4 --checkpoint checkpoints/david_net_best.safetensors

# Muted / surveillance video (routes through acoustic null token)
python predict.py --input silent_clip.mp4 --checkpoint checkpoints/david_net_best.safetensors

# Audio-only phone call (routes through visual null token)
python predict.py --input audio_call.wav --checkpoint checkpoints/david_net_best.safetensors

πŸ”’ Verification & Cryptographic Manifests

To ensure 0.0% identity leakage and complete auditability:

  • SPLITS_SHA256.json provides cryptographic checksums for all subject-disjoint partitions.
  • eval_summary.json provides structured DeLong test statistics, AUCs, and bootstrap confidence intervals across all 160 runs.
  • Preprocessed tensor cache shards are available on Hugging Face Datasets.

βš–οΈ License & Evidentiary Disclaimer

This model is licensed under the MIT License.

Forensic outputs provided by this system represent probabilistic calibration evidence ($T^* = 1.18$) and should not be construed as automated legal verdicts. Forensic practitioners must evaluate predictive intervals within broader chains of custody and corroborating investigative evidence.


πŸ“œ Citation

@article{hossain2026davidnet,
  title={DAVID-Net: Disentangled Audio-Visual Forensic Attribution under Identity-Disjoint Evaluation},
  author={Hossain, Moshin Ibna and Khan, Alif Hasan and Shakib, Shadman and Jam, MD. Yeahyea and Hasan, Mohammad Mahmudul},
  journal={Machine Learning with Applications},
  year={2026},
  note={Under review}
}

@thesis{hossain2026thesis,
  title={Decoupled Multimodal Forensic Attribution and Desynchronization Detection},
  author={Hossain, Moshin Ibna and Khan, Alif Hasan and Shakib, Shadman and Jam, MD. Yeahyea},
  school={American International University-Bangladesh},
  year={2026}
}
Downloads last month
19
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Evaluation results

  • Clip AUC on FakeAVCeleb (Strict Identity-Disjoint)
    self-reported
    0.976
  • Four-Quadrant Macro-F1 on FakeAVCeleb (Strict Identity-Disjoint)
    self-reported
    0.918