- DAVID-Net: Disentangled Audio-Visual Forensic Attribution under Identity-Disjoint Evaluation
DAVID-Net: Disentangled Audio-Visual Forensic Attribution under Identity-Disjoint Evaluation
Official repository for DAVID-Net (Decoupled Audio-Visual Forensic Network), an asymmetric cross-modal transformer framework for fine-grained four-quadrant deepfake attribution, missing-modality fault tolerance, and millisecond-level temporal boundary localization.
π Framework Summary
Conventional deepfake detectors collapse multimodal evidence into an opaque binary verdict ("real vs. fake"). In digital forensics, judicial investigations, and platform moderation, an unadulterated binary metric yields little diagnostic value: it cannot identify which communication channel was manipulated, when the manipulation took place, or provide calibrated confidence bounds. Furthermore, models relying strictly on audio-visual temporal synchronization fail completely on double-fakes (FVFA) where synthetic video is lip-synced to cloned speech.
DAVID-Net reformulates deepfake detection as a Four-Quadrant Authenticity Attribution task:
| Quadrant | Visual Stream | Acoustic Stream | Forensic Diagnosis |
|---|---|---|---|
| RVRA | Real ($y_v=0$) | Real ($y_a=0$) | Authentic Media: Genuine visual recording and authentic speech. |
| RVFA | Real ($y_v=0$) | Fake ($y_a=1$) | Voice Cloning / Dubbing: Authentic video paired with cloned/synthetic speech. |
| FVRA | Fake ($y_v=1$) | Real ($y_a=0$) | Facial Manipulation: Face swap or reenactment over genuine speech. |
| FVFA | Fake ($y_v=1$) | Fake ($y_a=1$) | Dual Manipulation: End-to-end synthetic or spliced double-fake. |
ποΈ Architecture & Methodological Highlights
- Decoupled Multimodal Representation:
- Visual Stream: Aligned face crops and perioral mouth crops encoded via VideoMAE ViT-Base ($128 \times 768$ spatio-temporal tokens).
- Acoustic Stream: 16,kHz raw speech waveforms encoded via WavLM Base+ ($50 \times 768$ acoustic tokens).
- Hidden representation dimension $d = 768$.
- Geometric Subspace Disentanglement:
- Enforces strict subspace orthogonality: $z_v \perp z_c$ and $z_a \perp z_c$ ($\mathcal{L}_{\text{ortho}} = |\cos(z_v, z_c^{(v)})| + |\cos(z_a, z_c^{(a)})|$).
- Prevents cross-channel shortcut memorization and eliminates the double-fake trap.
- Fault-Tolerant Learnable Null Tokens:
- Ingests silent video or isolated audio recordings without crashing or random guessing via learnable tokens $\mathbf{e}{\text{null}}^{(v)}, \mathbf{e}{\text{null}}^{(a)} \in \mathbb{R}^{1 \times 768}$.
- Windowed Temporal Synchronization:
- Inner-product temporal agreement $s_t = \tilde{V}_t^\top \tilde{A}_t$ trained via bidirectional InfoNCE on authentic media.
- Decoupled Multi-Task Attribution Heads:
- Independent projection heads for video veracity ($H_v$), audio veracity ($H_a$), four-quadrant attribution ($H_{quad}$), and frame-level boundary localization ($H_{loc}$).
π Empirical Evaluation Benchmarks (5-Seed Campaign)
All metrics reflect multi-seed evaluation across five independent seeds $\mathcal{S} \in {7, 42, 123, 456, 2024}$ from a 160-run evaluation campaign.
1. In-Domain Benchmark on FakeAVCeleb (Strict Identity-Disjoint Split)
| Architecture | Parameters | Clip AUC | Video AUC | Audio AUC | Macro-$F_1$ (%) | Balanced Acc (%) |
|---|---|---|---|---|---|---|
| Audio Probe (WavLM) | 94.7M | $0.842 \pm 0.007$ | β | $0.999 \pm 0.002$ | β | β |
| Video Probe (VideoMAE) | 86.6M | $0.902 \pm 0.005$ | $0.966 \pm 0.004$ | β | β | β |
| AASIST | 0.3M | $0.748 \pm 0.004$ | β | $0.998 \pm 0.001$ | β | β |
| EfficientNet-B4 | 19.3M | $0.926 \pm 0.002$ | $0.991 \pm 0.001$ | β | β | β |
| Late Fusion (V+A) | 181.3M | $0.918 \pm 0.015$ | $0.963 \pm 0.007$ | $0.998 \pm 0.002$ | $79.7 \pm 3.2$ | $80.1 \pm 2.8$ |
| AVFF | 186.2M | $\mathbf{0.981 \pm 0.003}$ | β | β | β | β |
| DAVID-Net (Phase A, Frozen) | 75.6M | $0.976 \pm 0.006$ | $0.979 \pm 0.003$ | $\mathbf{0.999 \pm 0.002}$ | $\mathbf{91.8 \pm 1.7}$ | $\mathbf{91.3 \pm 1.0}$ |
| DAVID-Net (Phase B, E2E) | 105.5M | $0.978 \pm 0.005$ | $\mathbf{0.982 \pm 0.004}$ | $\mathbf{0.999 \pm 0.001}$ | $\mathbf{92.4 \pm 1.2}$ | $\mathbf{92.0 \pm 0.8}$ |
Superiority over Late Fusion, EfficientNet-B4, and AASIST is statistically significant across all 5 seeds ($p < 0.05$, paired DeLong test with Holm--Bonferroni correction).
2. Zero-Shot Cross-Dataset Generalization (No Target Domain Adaptation)
| Target Benchmark | Modality | Samples | AASIST / Baseline | DAVID-Net | Gain ($\Delta$) |
|---|---|---|---|---|---|
| ASVspoof 2019 LA | Audio | 25,380 | $0.648 \pm 0.013$ | $\mathbf{0.839 \pm 0.019}$ | +19.1% |
| In-the-Wild Audio | Audio | 31,779 | $0.569 \pm 0.027$ | $\mathbf{0.691 \pm 0.021}$ | +12.2% |
| Celeb-DF v2 | Video | 518 | $0.589 \pm 0.012$ | $\mathbf{0.682 \pm 0.029}$ (Ph. B) | +8.3% |
| DFDC-10 | Multimodal | 5,000 | $0.619 \pm 0.024$ | $\mathbf{0.647 \pm 0.020}$ | +2.8% |
| DeepFakeTIMIT | Video | 640 | $0.982 \pm 0.005$ | $\mathbf{0.998 \pm 0.002}$ | +1.6% |
| WaveFake | Audio | 11,000 | $0.098 \pm 0.015$ | $0.113 \pm 0.018$ | (Vocoder Shift) |
3. Leave-One-Generator-Out (LOGO) Resilience
| Held-Out Generator | Synthesis Type | Video AUC | Audio AUC | Clip AUC | Resilience Floor |
|---|---|---|---|---|---|
logo_wav2lip |
Lip-Sync Reenactment | $0.425 \pm 0.028$ | $0.994 \pm 0.002$ | $0.919 \pm 0.008$ | $\ge 90.0%$ |
logo_fsgan |
Face Swapping (GAN) | $0.884 \pm 0.016$ | $0.999 \pm 0.001$ | $0.902 \pm 0.012$ | $\ge 90.0%$ |
logo_faceswap |
Autoencoder Face Swap | $0.941 \pm 0.011$ | $0.999 \pm 0.001$ | $0.932 \pm 0.009$ | $\ge 90.0%$ |
| Average LOGO | β | β | β | $\mathbf{0.918 \pm 0.014}$ | $\ge \mathbf{90.0%}$ |
β‘ Quickstart & Model Inference
1. Download Checkpoint via huggingface_hub
from huggingface_hub import hf_hub_download
# Download official release weights (~341 MB) and architecture config
model_path = hf_hub_download(
repo_id="MIHMahmudEli/david-net-av",
filename="david_net_best.safetensors"
)
config_path = hf_hub_download(
repo_id="MIHMahmudEli/david-net-av",
filename="config.json"
)
print(f"Model downloaded: {model_path}")
2. Standalone Inference CLI (from GitHub)
git clone https://github.com/MIHMahmudEli/david-net-av.git
cd david-net-av
pip install -r requirements.txt
# Multimodal analysis
python predict.py --input sample_video.mp4 --checkpoint checkpoints/david_net_best.safetensors
# Muted / surveillance video (routes through acoustic null token)
python predict.py --input silent_clip.mp4 --checkpoint checkpoints/david_net_best.safetensors
# Audio-only phone call (routes through visual null token)
python predict.py --input audio_call.wav --checkpoint checkpoints/david_net_best.safetensors
π Verification & Cryptographic Manifests
To ensure 0.0% identity leakage and complete auditability:
SPLITS_SHA256.jsonprovides cryptographic checksums for all subject-disjoint partitions.eval_summary.jsonprovides structured DeLong test statistics, AUCs, and bootstrap confidence intervals across all 160 runs.- Preprocessed tensor cache shards are available on Hugging Face Datasets.
βοΈ License & Evidentiary Disclaimer
This model is licensed under the MIT License.
Forensic outputs provided by this system represent probabilistic calibration evidence ($T^* = 1.18$) and should not be construed as automated legal verdicts. Forensic practitioners must evaluate predictive intervals within broader chains of custody and corroborating investigative evidence.
π Citation
@article{hossain2026davidnet,
title={DAVID-Net: Disentangled Audio-Visual Forensic Attribution under Identity-Disjoint Evaluation},
author={Hossain, Moshin Ibna and Khan, Alif Hasan and Shakib, Shadman and Jam, MD. Yeahyea and Hasan, Mohammad Mahmudul},
journal={Machine Learning with Applications},
year={2026},
note={Under review}
}
@thesis{hossain2026thesis,
title={Decoupled Multimodal Forensic Attribution and Desynchronization Detection},
author={Hossain, Moshin Ibna and Khan, Alif Hasan and Shakib, Shadman and Jam, MD. Yeahyea},
school={American International University-Bangladesh},
year={2026}
}
- Downloads last month
- 19
Evaluation results
- Clip AUC on FakeAVCeleb (Strict Identity-Disjoint)self-reported0.976
- Four-Quadrant Macro-F1 on FakeAVCeleb (Strict Identity-Disjoint)self-reported0.918