Download README.md from KoljaB/TurnEndDetection-multilingual: direct link, hf CLI and curl.
- Browser
- Download file 14.5 kB
-
https://huggingface.co/KoljaB/TurnEndDetection-multilingual/resolve/main/README.md
- Command line
-
hf download hf://KoljaB/TurnEndDetection-multilingual/README.md
-
curl -L -H "Authorization: Bearer $HF_TOKEN" -o README.md https://huggingface.co/KoljaB/TurnEndDetection-multilingual/resolve/main/README.md
license: apache-2.0
language:
- en
- de
- es
- fr
- it
- pt
- zh
library_name: onnxruntime
pipeline_tag: audio-classification
base_model:
- openai/whisper-tiny
- jhu-clsp/mmBERT-base
tags:
- turn-detection
- end-of-turn
- voice-assistant
- conversational-ai
- onnx
TurnEndDetection multilingual (v3)
GitHub · X · YouTube · Discord
People pause in the middle of a turn all the time, and almost half of those thinking pauses come right after a grammatically complete sentence. A model that only reads the words cannot tell "finished sentence, still thinking" from "done, your turn". Two things can: how the last words were spoken, and what the conversation was about. "Friday" is a whole turn after "Which day?", and only a beginning after "Tell me about your week".
This model listens to the last three seconds before a pause, reads the words, and reads the last few messages of the conversation. It returns one calibrated probability that the speaker is done and a suggested wait, so a voice assistant can answer at once when the user is done and hold back when they are not.
- Input: the text of the current utterance, the last 3 seconds of audio (16 kHz mono), and optionally the messages before the turn.
- Output:
done, the probability that the speaker is done, andwait_s, how long to stay silent before answering. Also the three raw numbers:sentence_end,turn_endandcontext_end. - Languages: English, German, Spanish, French, Italian, Portuguese, Chinese.
- Speed (4 CPU threads): voice + text about 32 ms; the conversation model 20 to 125 ms more, in parallel.
- Size: 369 MB text model, 369 MB conversation model, two audio models of 10 MB each (ONNX).
What is new in v3: a second audio model for clips that already hold some silence, the conversation model, and a calibrated rule that turns everything into one probability and a wait. Each part is optional; the files of v2 are unchanged.
Quick start
pip install onnxruntime tokenizers numpy huggingface_hub
import sys
from huggingface_hub import snapshot_download
sys.path.insert(0, snapshot_download("KoljaB/TurnEndDetection-multilingual"))
from inference import TurnModel
model = TurnModel()
out = model(
"Friday morning please",
audio, # float32 numpy array, 16 kHz mono, up to the moment you ask
tail_s=0.2, # how much silence the clip ends with
history=[{"role": "assistant", "content": "Which day would you like to travel instead?"}],
)
out["done"] # e.g. 0.87: probably done
out["wait_s"] # e.g. 0.25: answer after a quarter second of silence
With the same question before it and the same test clip, "Friday in the" gave done 0.12 and
wait_s 1.38. The exact numbers depend on the audio.
Everything but the text or the audio is optional. Without history the conversation model is
skipped; an empty list means "first turn of the conversation". Without tail_s the clip is
treated as ending right after the last word. Audio shorter than 3 seconds is fine. Without audio
you get sentence_end (and context_end) only.
Ask twice per pause
The voice says more once a little silence has passed, and the conversation model needs up to 125 ms. So ask twice:
- As soon as the pause begins (30 to 60 ms of silence in the clip,
tail_sabout 0.04). You get a firstdoneandwait_safter about 35 ms. Use it to prepare your reply. - Again 0.18 s into the pause (
tail_sabout 0.19), with the same text and history. Now both audio models and the conversation count. Replace the wait with this answer.
You were not going to answer before a quarter second of silence anyway, so the second ask costs
no time. inference.py runs the conversation model in a thread beside the rest; in a server,
start it on the first ask and reuse its result for the second (it depends only on the text and
the history, not on the audio).
The wait
done is the probability that the speaker is done at the moment the pause begins. A thinking
pause usually ends soon, so every further moment of silence makes "done" more likely.
wait_s is the moment at which the probability reaches the level you ask for (default 0.75),
between a minimum of 0.25 s and a maximum of 1.4 s. wait_seconds(done, level, minimum, maximum)
in inference.py lets you choose all three.
Measured on eot-bench (seven languages, the half not used for fitting the calibration): how long real turn ends wait, and how many thinking pauses get cut into.
| Setup | Level | Turn ends wait p50 / p75 / p90 | Thinking pauses cut into |
|---|---|---|---|
| one ask at the start of the pause (v2) | 0.75 | 0.37 / 0.58 / 0.86 s | 26.0 % |
| two asks, two audio models | 0.75 | 0.28 / 0.52 / 0.76 s | 21.7 % |
| two asks, two audio models, conversation | 0.75 | 0.28 / 0.45 / 0.73 s | 19.8 % |
| two asks, two audio models, conversation | 0.70 | 0.28 / 0.38 / 0.65 s | 22.6 % |
| two asks, two audio models, conversation | 0.80 | 0.30 / 0.54 / 0.82 s | 17.0 % |
English and German are easier than the average: at level 0.75 the full setup cuts into 17 % and 13 % of their thinking pauses. Pause lengths in the benchmark are given in steps of 0.1 s; a pause of 0.5 s is counted as anywhere between 0.45 and 0.55 s.
The three raw numbers
| Output | From | Meaning |
|---|---|---|
sentence_end |
the words | the sentence is grammatically complete. Exactly the output of SentenceFinishedClassification-multilingual v2. |
turn_end |
the voice and the words | the speaker is done. With tail_s of 0.12 or more it is the mean of both audio models. |
context_end |
the words and the conversation | the speaker is done, given what was said before. |
None of them is centred on 0.5; use done, which is calibrated, or fit your own bars.
Results
AUC for "this pause is the end of the turn" (0.5 is guessing, 1.0 is perfect) on real user turns from eot-bench data. None of it was used in training. The table is the half of the pauses that was not used for fitting the calibration, 360 to 460 per language.
| Language | Words only (sentence_end) |
Words + conversation (context_end) |
Voice + words, first ask | Voice + words, second ask | Everything | Turn ends answered at once* |
|---|---|---|---|---|---|---|
| English | 0.71 | 0.86 | 0.86 | 0.89 | 0.92 | 47 → 61 % |
| German | 0.73 | 0.85 | 0.81 | 0.87 | 0.91 | 32 → 42 % |
| Spanish | 0.71 | 0.81 | 0.88 | 0.90 | 0.92 | 37 → 49 % |
| French | 0.75 | 0.86 | 0.87 | 0.90 | 0.93 | 42 → 63 % |
| Italian | 0.71 | 0.83 | 0.79 | 0.88 | 0.90 | 38 → 53 % |
| Portuguese | 0.68 | 0.82 | 0.80 | 0.84 | 0.88 | 28 → 37 % |
| Chinese | 0.70 | 0.77 | 0.76 | 0.84 | 0.86 | 25 → 34 % |
| all | 0.71 | 0.83 | 0.82 | 0.87 | 0.90 | 34 → 44 % |
* while cutting into 5 % of the thinking pauses: second ask without → with the conversation. "First ask" is a clip ending 0.03 s into the silence, "second ask" one ending 0.24 s into it with both audio models. These numbers come from our own protocol on the public data and are not comparable to results from LiveKit's benchmark harness.
English conversation between people (voice + words, one audio model)
| Test | Words only | Voice only | Voice + words |
|---|---|---|---|
| Open Yap, 36,964 pauses from 34 speakers the model never heard | 0.67 | - | 0.77 (0.76 with 0.03 s of silence) |
| otoSpeech task-oriented, 14,900 pauses from 58 sessions (no transcripts) | - | 0.71 | - |
Conversation between people is harder than talking to an assistant: whether a pause becomes a change of speaker also depends on the listener.
Speed
Measured with the same script and the same files on two machines: CPU only (onnxruntime), 4 threads, median of 50 runs after 5 warm-up runs.
| Part | Windows, AMD Ryzen 9 3900X | Linux, Intel i9-13900KF, efficiency cores | Linux, Intel i9-13900KF, performance cores |
|---|---|---|---|
| Audio features (spectrogram of 3 s) | 1.2 ms | 0.9 ms | 0.6 ms |
| Text model, short utterance | 10.8 ms | 12.7 ms | 5.3 ms |
| Text model, long utterance (24 tokens) | 21.7 ms | 23.0 ms | 10.1 ms |
| Audio model (one) | 9.5 ms | 8.7 ms | 3.5 ms |
| Audio models (both, second ask) | 19.4 ms | 17.7 ms | 7.2 ms |
| Conversation model, short (11 tokens) | 14.2 ms | 15.8 ms | 6.6 ms |
| Conversation model, typical (55 tokens) | 44.8 ms | 44.5 ms | 18.4 ms |
| Conversation model, at its limit (128 tokens) | 110 ms | 99 ms | 42 ms |
| First ask (voice + text) | 22.5 ms | 47 ms | 21 ms |
| Memory with all models loaded | about 1.0 GB | about 1.3 GB | about 1.3 GB |
On the efficiency cores the first ask took 47 ms although its parts add up to about 22 ms: with several models in one process, idle worker threads have to wake up each time the process switches to another model. Windows and the performance cores did not show this.
The conversation model does not delay the answer. It is the slowest part, but it runs beside the rest: start it at the first ask, when the pause begins (about 0.04 s of silence), and use its result at the second ask, about 0.19 s into the pause. Nobody answers before a quarter second of silence anyway, so even 100 to 125 ms fit into the pause without anyone hearing it. If the machine is busy and the result is not ready in time, decide without it (voice + text only): the answer gets slightly less accurate, never later.
The conversation model was trained with up to 224 tokens of input. inference.py limits it to
128: the benchmark result is the same and it is ready in time for the second ask.
Training
Audio models:
openai/whisper-tinyencoder (multilingual, 4 layers, Apache-2.0), fine-tuned on 3-second clips.audio.onnxsaw clips ending at a random point between 0.03 and 0.24 s into the silence,audio_late.onnxonly clips ending 0.24 s into it.Audio data: 193,842 real pauses from 843 two-speaker conversations (189 speakers, about 500 hours) of Open Yap 1K: 68,997 ends of turn, where the other person took over, and 124,845 thinking pauses, after which the same person went on. Pauses were found with a voice-activity detector (Silero VAD) on each speaker's own channel, the way a live system finds them.
Text model: SentenceFinishedClassification-multilingual v2 (
jhu-clsp/mmBERT-base, MIT), frozen. Its training is described on its own page.Conversation model: the same text model, trained further to answer a different question: given the last four messages and the utterance so far, is the speaker done? Its targets are judgements by Claude Haiku 5.5 (a number from 0 to 100 per pause), not human labels. It learned from about 24,000 pauses (1,000 more were held out for validation):
- 4,932 from 567 short spoken dialogues between a person and a voice assistant in the seven languages, written by Claude Sonnet 5.5 for this purpose, with the person's thinking pauses marked;
- about 12,000 real pauses from Open Yap conversations;
- about 8,100 user turns from WildChat in the six languages other than English, whole or cut at a random word.
The written dialogues did most of the work: with the other two sources alone the model reached AUC 0.77 on eot-bench, with them 0.82.
Calibration and the pause-length curve were fitted on half of the eot-bench pauses; the tables above use the other half.
This repository holds model weights only, no training data and no audio.
Limits
- The audio models learned from English conversations between people. They carry over to the other six languages in the test above, but were not trained on them.
- The conversation model learned from a language model's judgement, mostly on written dialogues. Those dialogues were made to resemble spoken requests to an assistant, the kind of conversation eot-bench also contains, so its benchmark numbers may be kinder than what you see on your own users. Measure on your own recordings.
- The conversation model reads the last four messages, each cut to 400 characters, and 128 tokens in all. Anything earlier is invisible to it.
- The calibration and the pause-length curve come from one benchmark. Other speakers pause differently; the level is the knob to turn.
- It shortens waits and avoids cut-ins; it does not remove either. One thinking pause in five is still cut into at the default level.
- Overlapping speech, background voices and music were not part of the tests.
- Japanese, Korean and Russian are not covered.
Files
| File | What |
|---|---|
onnx/audio.onnx |
audio encoder and turn-end head, 10 MB |
onnx/audio_late.onnx |
the same for clips that hold a quarter second of silence, 10 MB (optional) |
onnx/text.onnx |
text model, 369 MB |
onnx/context.onnx |
conversation model, 369 MB (optional) |
inference.py |
spectrogram, text clean-up, calibration, the wait rule and a minimal runner |
mel_filters.npy, tokenizer.json |
needed by inference.py |
eval.json |
the numbers above |
Versions
- v3 (this one): adds
audio_late.onnx,context.onnx, the calibrateddoneandwait_s.audio.onnxandtext.onnxare the files of v2; code written for v2 keeps working and gets the samesentence_endandturn_endwhen it passes notail_s. - v2 (git tag
v2): the audio model was retrained with a random amount of trailing silence (0.03 to 0.24 s), so it can be asked right when a pause begins. - v1 (git tag
v1): every training clip ended 0.24 s into the silence. Its audio model isaudio_late.onnxin v3.
Credits
Open Yap 1K by The Agentic Data Company. Whisper by OpenAI. mmBERT by JHU CLSP. eot-bench data by LiveKit. otoSpeech by oto. WildChat by the Allen Institute for AI. Silero VAD by the Silero team. Training targets and dialogues for the conversation model by Claude (Anthropic).