You need to agree to share your Hugging Face information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

Log in or Sign Up to review the conditions and access this model content.

TurnEndDetection multilingual (v3)

GitHub Β· X Β· YouTube Β· Discord

People pause in the middle of a turn all the time, and almost half of those thinking pauses come right after a grammatically complete sentence. A model that only reads the words cannot tell "finished sentence, still thinking" from "done, your turn". Two things can: how the last words were spoken, and what the conversation was about. "Friday" is a whole turn after "Which day?", and only a beginning after "Tell me about your week".

This model listens to the last three seconds before a pause, reads the words, and reads the last few messages of the conversation. It returns one calibrated probability that the speaker is done and a suggested wait, so a voice assistant can answer at once when the user is done and hold back when they are not.

  • Input: the text of the current utterance, the last 3 seconds of audio (16 kHz mono), and optionally the messages before the turn.
  • Output: done, the probability that the speaker is done, and wait_s, how long to stay silent before answering. Also the three raw numbers: sentence_end, turn_end and context_end.
  • Languages: English, German, Spanish, French, Italian, Portuguese, Chinese.
  • Speed (4 CPU threads): voice + text about 32 ms; the conversation model 20 to 125 ms more, in parallel.
  • Size: 369 MB text model, 369 MB conversation model, two audio models of 10 MB each (ONNX).

What is new in v3: a second audio model for clips that already hold some silence, the conversation model, and a calibrated rule that turns everything into one probability and a wait. Each part is optional; the files of v2 are unchanged.

Quick start

pip install onnxruntime tokenizers numpy huggingface_hub
import sys
from huggingface_hub import snapshot_download

sys.path.insert(0, snapshot_download("KoljaB/TurnEndDetection-multilingual"))
from inference import TurnModel

model = TurnModel()
out = model(
    "Friday morning please",
    audio,                      # float32 numpy array, 16 kHz mono, up to the moment you ask
    tail_s=0.2,                 # how much silence the clip ends with
    history=[{"role": "assistant", "content": "Which day would you like to travel instead?"}],
)
out["done"]      # e.g. 0.87: probably done
out["wait_s"]    # e.g. 0.25: answer after a quarter second of silence

With the same question before it and the same test clip, "Friday in the" gave done 0.12 and wait_s 1.38. The exact numbers depend on the audio.

Everything but the text or the audio is optional. Without history the conversation model is skipped; an empty list means "first turn of the conversation". Without tail_s the clip is treated as ending right after the last word. Audio shorter than 3 seconds is fine. Without audio you get sentence_end (and context_end) only.

Ask twice per pause

The voice says more once a little silence has passed, and the conversation model needs up to 125 ms. So ask twice:

  1. As soon as the pause begins (30 to 60 ms of silence in the clip, tail_s about 0.04). You get a first done and wait_s after about 35 ms. Use it to prepare your reply.
  2. Again 0.18 s into the pause (tail_s about 0.19), with the same text and history. Now both audio models and the conversation count. Replace the wait with this answer.

You were not going to answer before a quarter second of silence anyway, so the second ask costs no time. inference.py runs the conversation model in a thread beside the rest; in a server, start it on the first ask and reuse its result for the second (it depends only on the text and the history, not on the audio).

The wait

done is the probability that the speaker is done at the moment the pause begins. A thinking pause usually ends soon, so every further moment of silence makes "done" more likely. wait_s is the moment at which the probability reaches the level you ask for (default 0.75), between a minimum of 0.25 s and a maximum of 1.4 s. wait_seconds(done, level, minimum, maximum) in inference.py lets you choose all three.

Measured on eot-bench (seven languages, the half not used for fitting the calibration): how long real turn ends wait, and how many thinking pauses get cut into.

Setup Level Turn ends wait p50 / p75 / p90 Thinking pauses cut into
one ask at the start of the pause (v2) 0.75 0.37 / 0.58 / 0.86 s 26.0 %
two asks, two audio models 0.75 0.28 / 0.52 / 0.76 s 21.7 %
two asks, two audio models, conversation 0.75 0.28 / 0.45 / 0.73 s 19.8 %
two asks, two audio models, conversation 0.70 0.28 / 0.38 / 0.65 s 22.6 %
two asks, two audio models, conversation 0.80 0.30 / 0.54 / 0.82 s 17.0 %

English and German are easier than the average: at level 0.75 the full setup cuts into 17 % and 13 % of their thinking pauses. Pause lengths in the benchmark are given in steps of 0.1 s; a pause of 0.5 s is counted as anywhere between 0.45 and 0.55 s.

The three raw numbers

Output From Meaning
sentence_end the words the sentence is grammatically complete. Exactly the output of SentenceFinishedClassification-multilingual v2.
turn_end the voice and the words the speaker is done. With tail_s of 0.12 or more it is the mean of both audio models.
context_end the words and the conversation the speaker is done, given what was said before.

None of them is centred on 0.5; use done, which is calibrated, or fit your own bars.

Results

AUC for "this pause is the end of the turn" (0.5 is guessing, 1.0 is perfect) on real user turns from eot-bench data. None of it was used in training. The table is the half of the pauses that was not used for fitting the calibration, 360 to 460 per language.

Language Words only (sentence_end) Words + conversation (context_end) Voice + words, first ask Voice + words, second ask Everything Turn ends answered at once*
English 0.71 0.86 0.86 0.89 0.92 47 β†’ 61 %
German 0.73 0.85 0.81 0.87 0.91 32 β†’ 42 %
Spanish 0.71 0.81 0.88 0.90 0.92 37 β†’ 49 %
French 0.75 0.86 0.87 0.90 0.93 42 β†’ 63 %
Italian 0.71 0.83 0.79 0.88 0.90 38 β†’ 53 %
Portuguese 0.68 0.82 0.80 0.84 0.88 28 β†’ 37 %
Chinese 0.70 0.77 0.76 0.84 0.86 25 β†’ 34 %
all 0.71 0.83 0.82 0.87 0.90 34 β†’ 44 %

* while cutting into 5 % of the thinking pauses: second ask without β†’ with the conversation. "First ask" is a clip ending 0.03 s into the silence, "second ask" one ending 0.24 s into it with both audio models. These numbers come from our own protocol on the public data and are not comparable to results from LiveKit's benchmark harness.

English conversation between people (voice + words, one audio model)

Test Words only Voice only Voice + words
Open Yap, 36,964 pauses from 34 speakers the model never heard 0.67 - 0.77 (0.76 with 0.03 s of silence)
otoSpeech task-oriented, 14,900 pauses from 58 sessions (no transcripts) - 0.71 -

Conversation between people is harder than talking to an assistant: whether a pause becomes a change of speaker also depends on the listener.

Speed

Measured with the same script and the same files on two machines: CPU only (onnxruntime), 4 threads, median of 50 runs after 5 warm-up runs.

Part Windows, AMD Ryzen 9 3900X Linux, Intel i9-13900KF, efficiency cores Linux, Intel i9-13900KF, performance cores
Audio features (spectrogram of 3 s) 1.2 ms 0.9 ms 0.6 ms
Text model, short utterance 10.8 ms 12.7 ms 5.3 ms
Text model, long utterance (24 tokens) 21.7 ms 23.0 ms 10.1 ms
Audio model (one) 9.5 ms 8.7 ms 3.5 ms
Audio models (both, second ask) 19.4 ms 17.7 ms 7.2 ms
Conversation model, short (11 tokens) 14.2 ms 15.8 ms 6.6 ms
Conversation model, typical (55 tokens) 44.8 ms 44.5 ms 18.4 ms
Conversation model, at its limit (128 tokens) 110 ms 99 ms 42 ms
First ask (voice + text) 22.5 ms 47 ms 21 ms
Memory with all models loaded about 1.0 GB about 1.3 GB about 1.3 GB

On the efficiency cores the first ask took 47 ms although its parts add up to about 22 ms: with several models in one process, idle worker threads have to wake up each time the process switches to another model. Windows and the performance cores did not show this.

The conversation model does not delay the answer. It is the slowest part, but it runs beside the rest: start it at the first ask, when the pause begins (about 0.04 s of silence), and use its result at the second ask, about 0.19 s into the pause. Nobody answers before a quarter second of silence anyway, so even 100 to 125 ms fit into the pause without anyone hearing it. If the machine is busy and the result is not ready in time, decide without it (voice + text only): the answer gets slightly less accurate, never later.

The conversation model was trained with up to 224 tokens of input. inference.py limits it to 128: the benchmark result is the same and it is ready in time for the second ask.

Training

  • Audio models: openai/whisper-tiny encoder (multilingual, 4 layers, Apache-2.0), fine-tuned on 3-second clips. audio.onnx saw clips ending at a random point between 0.03 and 0.24 s into the silence, audio_late.onnx only clips ending 0.24 s into it.

  • Audio data: 193,842 real pauses from 843 two-speaker conversations (189 speakers, about 500 hours) of Open Yap 1K: 68,997 ends of turn, where the other person took over, and 124,845 thinking pauses, after which the same person went on. Pauses were found with a voice-activity detector (Silero VAD) on each speaker's own channel, the way a live system finds them.

  • Text model: SentenceFinishedClassification-multilingual v2 (jhu-clsp/mmBERT-base, MIT), frozen. Its training is described on its own page.

  • Conversation model: the same text model, trained further to answer a different question: given the last four messages and the utterance so far, is the speaker done? Its targets are judgements by Claude Haiku 5.5 (a number from 0 to 100 per pause), not human labels. It learned from about 24,000 pauses (1,000 more were held out for validation):

    • 4,932 from 567 short spoken dialogues between a person and a voice assistant in the seven languages, written by Claude Sonnet 5.5 for this purpose, with the person's thinking pauses marked;
    • about 12,000 real pauses from Open Yap conversations;
    • about 8,100 user turns from WildChat in the six languages other than English, whole or cut at a random word.

    The written dialogues did most of the work: with the other two sources alone the model reached AUC 0.77 on eot-bench, with them 0.82.

  • Calibration and the pause-length curve were fitted on half of the eot-bench pauses; the tables above use the other half.

This repository holds model weights only, no training data and no audio.

Limits

  • The audio models learned from English conversations between people. They carry over to the other six languages in the test above, but were not trained on them.
  • The conversation model learned from a language model's judgement, mostly on written dialogues. Those dialogues were made to resemble spoken requests to an assistant, the kind of conversation eot-bench also contains, so its benchmark numbers may be kinder than what you see on your own users. Measure on your own recordings.
  • The conversation model reads the last four messages, each cut to 400 characters, and 128 tokens in all. Anything earlier is invisible to it.
  • The calibration and the pause-length curve come from one benchmark. Other speakers pause differently; the level is the knob to turn.
  • It shortens waits and avoids cut-ins; it does not remove either. One thinking pause in five is still cut into at the default level.
  • Overlapping speech, background voices and music were not part of the tests.
  • Japanese, Korean and Russian are not covered.

Files

File What
onnx/audio.onnx audio encoder and turn-end head, 10 MB
onnx/audio_late.onnx the same for clips that hold a quarter second of silence, 10 MB (optional)
onnx/text.onnx text model, 369 MB
onnx/context.onnx conversation model, 369 MB (optional)
inference.py spectrogram, text clean-up, calibration, the wait rule and a minimal runner
mel_filters.npy, tokenizer.json needed by inference.py
eval.json the numbers above

Versions

  • v3 (this one): adds audio_late.onnx, context.onnx, the calibrated done and wait_s. audio.onnx and text.onnx are the files of v2; code written for v2 keeps working and gets the same sentence_end and turn_end when it passes no tail_s.
  • v2 (git tag v2): the audio model was retrained with a random amount of trailing silence (0.03 to 0.24 s), so it can be asked right when a pause begins.
  • v1 (git tag v1): every training clip ended 0.24 s into the silence. Its audio model is audio_late.onnx in v3.

Credits

Open Yap 1K by The Agentic Data Company. Whisper by OpenAI. mmBERT by JHU CLSP. eot-bench data by LiveKit. otoSpeech by oto. WildChat by the Allen Institute for AI. Silero VAD by the Silero team. Training targets and dialogues for the conversation model by Claude (Anthropic).

Downloads last month
10
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for KoljaB/TurnEndDetection-multilingual

Quantized
(298)
this model