DAVE: A Decoupled Audio-Visual Enhancement Framework for Real-World Speech Separation

Paper (arXiv:2608.09288) · Code (GitHub)

DAVE performs audio-visual speech enhancement and target speaker extraction on real-world meeting recordings. An audio-only separation backbone (TIGER-M, 2.56M parameters) reconstructs two anonymous speech streams; a four-cue weighted-fusion module assigns speaker identities from face video; a selective enhancement chain finalizes output. Because the backbone never consumes video, degraded visual input cannot corrupt the reconstructed audio.

🏆 Challenge Results (official leaderboard, seven-column rank-mean)

Track 1 — 4th of 17 teams (top 5 shown):

Rank	Team	SI-SDR↑	PESQ↑	STOI↑	UTMOS↑	DNSMOS-OVRL↑	CER↓	Spk-Sim↑
1	audioman	10.699	2.933	0.841	2.074	1.832	0.123	0.749
2	AITD	12.716	3.000	0.852	2.100	1.697	0.145	0.769
3	twilight	9.162	2.775	0.825	2.145	1.944	0.135	0.742
4	DAVE (ours)	10.232	2.717	0.817	2.765	2.022	0.171	0.726
5	SUSTechAILab	10.416	2.651	0.823	1.998	1.770	0.164	0.737

Track 2 — 3rd of 13 teams (top 5 shown):

Rank	Team	SI-SDR↑	PESQ↑	STOI↑	UTMOS↑	DNSMOS-OVRL↑	CER↓	Spk-Sim↑
1	AITD	12.255	2.939	0.844	2.129	1.670	0.153	0.764
2	audioman	10.223	2.888	0.830	2.094	1.760	0.148	0.744
3	DAVE (ours)	8.933	2.615	0.788	2.766	2.012	0.220	0.707
4	twilight	7.047	2.561	0.776	2.170	1.896	0.220	0.712
5	SUSTechAILab	9.541	2.554	0.804	2.019	1.733	0.211	0.726

Model Architecture

Component	Details
Separation backbone	TIGER (audio-only), in_channels=512, out_channels=256, num_blocks=12, upsampling_depth=5, win=640, stride=160, num_sources=2
Parameters	2,557,228 (2.56M)
Sample rate	16 kHz, mono
Initialization	TIGER-speech pretrained (JusperLee/TIGER-speech)
Released checkpoint	real_avse_tiger_m.ckpt (challenge submission, md5 15800af8)

Training

A six-term loss, introduced as a curriculum (one term at a time, verified for no-regression before its weight is raised):

L = L_SI-SDR + λ_cer·L_CER + λ_spk·L_spk + λ_utmos·L_UTMOS + λ_stoi·L_STOI + λ_pesq·L_PESQ

Loss term	Weight (final)	Description
L_SI-SDR	1.0	PIT negative SI-SDR over both stream orders
L_CER	3.0	teacher-forced cross-entropy, frozen Fun-ASR-Nano teacher
L_spk	5.0	1 − cosine, WeSpeaker cnceleb-resnet34-LM (frozen)
L_UTMOS	1.0	differentiable naturalness critic, clamped
L_STOI	5.0	differentiable STOI surrogate (clean-reference samples only)
L_PESQ	1.0	differentiable P.862 surrogate (clean-reference samples only)

Hyperparameter	Value
Optimizer	Adam, peak LR 1e-4, 200-step linear warmup, ReduceLROnPlateau (×0.5, patience 2)
Batch size	1 per GPU × 7 A800 (DDP)
Audio length	2–6 s per training pair
Gradient clipping	5.0
Precision	FP32 (TF32 matmul enabled)

Training data: the open 12,000-pair subset is released as DAVE-Corpus (sources: AliMeeting / AISHELL-4 / MUSAN, CC BY-SA 4.0). The full training pool additionally used subsets built from MISP-Meeting and official challenge dev data, which cannot be redistributed — see DATA_LICENSE.

Training code and the data-synthesis pipeline: GitHub — TaurenMountain/DAVE.

Quick Start

Separation is self-contained — the TIGER-M model code is bundled under src/models/, so only torch, numpy, soundfile are needed:

pip install torch numpy soundfile

python inference.py --mixture samples/echoset_01/mix.wav --out output
python inference.py --mixture path/to/dir --out output --device cuda

Output: output/<name>/s1.wav, s2.wav (16 kHz mono). Input must be 16 kHz. samples/echoset_01/ contains a playable example pair (mix / s1 / s2, from the Apache-2.0 EchoSet subset).

Python API

import sys, torch, soundfile as sf
sys.path.insert(0, "src")
from models import TIGER

model = TIGER(sample_rate=16000, in_channels=512, out_channels=256,
              num_blocks=12, upsampling_depth=5, win=640, stride=160, num_sources=2)
state = torch.load("model/real_avse_tiger_m.ckpt", map_location="cpu")["state_dict"]
model.load_state_dict(state, strict=False)
model.eval()

mix, sr = sf.read("samples/echoset_01/mix.wav", dtype="float32")  # 16 kHz mono
with torch.no_grad():
    est = model(torch.from_numpy(mix[None]))
sf.write("s1.wav", est[0, 0].numpy(), 16000)
sf.write("s2.wav", est[0, 1].numpy(), 16000)

Files

• model/real_avse_tiger_m.ckpt — TIGER-M weights (2.56M parameters), the challenge submission checkpoint;
• inference.py + src/models/ — self-contained separation CLI;
• samples/ — one example pair for format inspection.

Training code, the data-synthesis + transcription pipeline, and reference implementations of speaker attribution / enhancement / evaluation live in the GitHub repository.

Citation

@misc{zhou2026dave,
  title         = {{DAVE}: A Decoupled Audio-Visual Enhancement Framework for Real-World Speech Separation},
  author        = {Zhou, Wei and Ning, Wanyi and Guo, Yinshang and Fang, Qianxiao and Qian, Haitao and Li, Yingpeng},
  year          = {2026},
  eprint        = {2608.09288},
  archivePrefix = {arXiv},
  primaryClass  = {cs.SD},
  url           = {https://arxiv.org/abs/2608.09288}
}