espnet/frontia_vad_v1_offline: voice activity detection, offline

One of two ESPnet voice activity detectors trained together:

	model	reads ahead	for
this one (offline)	dual-mode TCN, run at its largest look-ahead	up to 124 frames (0.99 s) of future audio	whole recordings, cleaning a corpus

Each 8 ms frame (16 kHz, 512-point STFT without centring, 128-sample hop) gets the probability that someone is speaking. The model was trained as one-speaker diarization (diar1, num_spk 1): every speaker of a recording merged into one "speech" label.

Use

from espnet2.bin.diar_inference import VoiceActivityDetector

vad = VoiceActivityDetector.for_mode("offline", "espnet/frontia_vad_v1_offline")
for seg in vad.detect_file("recording.wav"):
    print(f"{seg.start:.2f}\t{seg.end:.2f}")

From a terminal, espnet detect recording.wav (offline model) or espnet detect --live (online model, from the microphone); from an agent, the MCP tool detect of espnet-mcp. DiarizeSpeech.from_pretrained loads it too, as a num_spk=1 diarization model.

Decision rule

A frame is speech when its probability is above 0.8, and the 0/1 sequence is then median-filtered over 11 frames: the setting with the lowest detection error on AMI MH dev. Look-ahead used: 124 (the model's max_look_ahead).

Latency: a frame's probability reads 124 frames (0.99 s) of future audio, and the median filter 5 frames (40 ms) more, so this model is for whole recordings rather than live streams.

Results

Detection error (%) = (missed + false-alarm speech) / reference speech, md-eval with a UEM over each whole recording, collar 0, at the rule above; the last two columns are each set's own best setting, for reference.

set	speech h	DetER (tuned)	FA	Miss	DetER (set's own best)	its threshold, median
aishell4_eval	11.5	7.63	2.15	5.48	5.87	0.5, 11
alimeeting_dev	3.9	6.34	3.54	2.80	6.09	0.7, 11
alimeeting_eval	9.9	5.72	2.81	2.91	5.53	0.7, 11
ami_dev	75.1	8.62	3.35	5.27	7.74	0.6, 11
ami_eval	90.1	9.09	1.64	7.45	6.65	0.6, 11
ami_mh_dev	7.6	9.07	5.07	4.01	9.07	0.8, 11
ami_mh_eval	7.3	10.21	3.85	6.37	9.50	0.7, 11
ava_speech_eval	19.9	31.59	21.31	10.28	28.39	0.9, 11
chime6_dev	117.9	10.00	1.62	8.38	4.31	0.3, 11
chime6_eval	122.7	14.16	3.66	10.50	12.80	0.6, 11
dev	329.2	7.32	1.73	5.59	5.25	0.5, 11
dipco_dev	98.1	12.56	2.39	10.17	7.74	0.5, 11
dipco_eval	92.0	10.26	3.25	7.01	8.48	0.6, 11
earnings21_eval	32.8	10.13	6.37	3.76	10.13	0.8, 11
icsi_dev	12.5	11.25	2.74	8.52	10.45	0.7, 11
icsi_eval	11.9	15.38	2.29	13.10	12.78	0.5, 11
notsofar_dev	123.7	3.57	0.74	2.82	2.83	0.6, 11
notsofar_eval	398.5	3.34	1.34	2.00	3.08	0.7, 11
ten_vad_eval	0.1	19.53	16.72	2.81	17.52	0.9, 11
voxconverse_dev	18.9	4.78	2.50	2.28	4.55	0.7, 11
voxconverse_eval	39.0	7.52	5.55	1.97	7.52	0.8, 11

Training

Recipe: egs2/frontia_vad_v1/diar1 in ESPnet. The config is in this repository.

training data	licence
AMI (every microphone)	CC BY 4.0
CHiME-6, OpenSLR SLR150 (every array and binaural channel)	CC BY-SA 4.0 (the ShareAlike condition is to be removed shortly, leaving CC BY 4.0)
NOTSOFAR-1 240825.1	CC BY 4.0
ICSI	CC BY 4.0
noise: MUSAN noise/ and music/; CHiME-3 background	CC / public domain; CC BY 4.0

The other sets in the table above are used for evaluation only.