espnet/frontia_vad_v1_online: voice activity detection, online

One of two ESPnet voice activity detectors trained together:

	model	reads ahead	for
this one (online)	causal TCN	nothing: every frame is decided from the audio up to it	live streams and the microphone

Each 8 ms frame (16 kHz, 512-point STFT without centring, 128-sample hop) gets the probability that someone is speaking. The model was trained as one-speaker diarization (diar1, num_spk 1): every speaker of a recording merged into one "speech" label.

Use

from espnet2.bin.diar_inference import VoiceActivityDetector

vad = VoiceActivityDetector.for_mode("online", "espnet/frontia_vad_v1_online")
for seg in vad.detect_file("recording.wav"):
    print(f"{seg.start:.2f}\t{seg.end:.2f}")

From a terminal, espnet detect recording.wav (offline model) or espnet detect --live (online model, from the microphone); from an agent, the MCP tool detect of espnet-mcp. DiarizeSpeech.from_pretrained loads it too, as a num_spk=1 diarization model.

Decision rule

A frame is speech when its probability is above 0.7, and the 0/1 sequence is then median-filtered over 11 frames: the setting with the lowest detection error on AMI MH dev. Look-ahead used: 0.

Latency: frame i covers samples [128 i, 128 i + 512), so its probability is ready 32 ms after the frame starts. The median filter then needs the frames after it: a median of 11 frames settles a decision 5 frames (40 ms) later. With median 1 the decision is final as soon as the probability is.

Results

Detection error (%) = (missed + false-alarm speech) / reference speech, md-eval with a UEM over each whole recording, collar 0, at the rule above; the last two columns are each set's own best setting, for reference.

set	speech h	DetER (tuned)	FA	Miss	DetER (set's own best)	its threshold, median
aishell4_eval	11.5	8.16	3.79	4.37	7.88	0.6, 11
alimeeting_dev	3.9	7.40	5.16	2.23	7.40	0.7, 11
alimeeting_eval	9.9	7.09	4.79	2.29	7.09	0.7, 11
ami_dev	75.1	10.03	4.71	5.33	9.66	0.6, 11
ami_eval	90.1	10.01	2.89	7.12	8.66	0.5, 11
ami_mh_dev	7.6	10.93	7.64	3.28	10.93	0.7, 11
ami_mh_eval	7.3	11.31	6.31	5.00	11.31	0.7, 11
ava_speech_eval	19.9	50.01	44.13	5.88	32.67	0.9, 11
chime6_dev	117.9	9.60	1.92	7.69	4.62	0.3, 11
chime6_eval	122.7	14.65	5.35	9.30	14.59	0.6, 11
dev	329.2	7.81	2.66	5.15	6.52	0.4, 11
dipco_dev	98.1	10.46	4.13	6.33	8.50	0.5, 11
dipco_eval	92.0	10.01	5.35	4.66	9.37	0.6, 11
earnings21_eval	32.8	14.60	10.37	4.23	14.60	0.7, 11
icsi_dev	12.5	11.74	5.31	6.44	11.74	0.7, 11
icsi_eval	11.9	14.53	3.83	10.70	13.47	0.5, 11
notsofar_dev	123.7	4.35	1.85	2.50	4.23	0.6, 11
notsofar_eval	398.5	4.34	2.80	1.55	4.34	0.7, 11
ten_vad_eval	0.1	22.64	21.30	1.34	16.20	0.9, 11
voxconverse_dev	18.9	4.93	3.88	1.05	4.93	0.7, 11
voxconverse_eval	39.0	8.24	7.31	0.93	8.06	0.8, 11

Training

Recipe: egs2/frontia_vad_v1/diar1 in ESPnet. The config is in this repository.

training data	licence
AMI (every microphone)	CC BY 4.0
CHiME-6, OpenSLR SLR150 (every array and binaural channel)	CC BY-SA 4.0 (the ShareAlike condition is to be removed shortly, leaving CC BY 4.0)
NOTSOFAR-1 240825.1	CC BY 4.0
ICSI	CC BY 4.0
noise: MUSAN noise/ and music/; CHiME-3 background	CC / public domain; CC BY 4.0

The other sets in the table above are used for evaluation only.