KoljaB's picture
README: measured speed on Windows and Linux; why the conversation model does not delay the answer
da6fc7f verified
|
Raw History Blame Contribute Delete
14.5 kB
---
license: apache-2.0
language: [en, de, es, fr, it, pt, zh]
library_name: onnxruntime
pipeline_tag: audio-classification
base_model:
- openai/whisper-tiny
- jhu-clsp/mmBERT-base
tags: [turn-detection, end-of-turn, voice-assistant, conversational-ai, onnx]
---
# TurnEndDetection multilingual (v3)
[GitHub](https://github.com/KoljaB/) · [X](https://x.com/LonLigrin) · [YouTube](https://www.youtube.com/@Linguflex) · [Discord](https://discord.com/invite/f556hqRjpv)
People pause in the middle of a turn all the time, and almost half of those thinking pauses come
right after a grammatically complete sentence. A model that only reads the words cannot tell
"finished sentence, still thinking" from "done, your turn". Two things can: how the last words
were spoken, and what the conversation was about. "Friday" is a whole turn after "Which day?",
and only a beginning after "Tell me about your week".
This model listens to the last three seconds before a pause, reads the words, and reads the last
few messages of the conversation. It returns one calibrated probability that the speaker is done
and a suggested wait, so a voice assistant can answer at once when the user is done and hold back
when they are not.
- **Input:** the text of the current utterance, the last 3 seconds of audio (16 kHz mono), and optionally the messages before the turn.
- **Output:** `done`, the probability that the speaker is done, and `wait_s`, how long to stay silent before answering. Also the three raw numbers: `sentence_end`, `turn_end` and `context_end`.
- **Languages:** English, German, Spanish, French, Italian, Portuguese, Chinese.
- **Speed (4 CPU threads):** voice + text about 32 ms; the conversation model 20 to 125 ms more, in parallel.
- **Size:** 369 MB text model, 369 MB conversation model, two audio models of 10 MB each (ONNX).
What is new in v3: a second audio model for clips that already hold some silence, the conversation
model, and a calibrated rule that turns everything into one probability and a wait. Each part is
optional; the files of v2 are unchanged.
## Quick start
```bash
pip install onnxruntime tokenizers numpy huggingface_hub
```
```python
import sys
from huggingface_hub import snapshot_download
sys.path.insert(0, snapshot_download("KoljaB/TurnEndDetection-multilingual"))
from inference import TurnModel
model = TurnModel()
out = model(
"Friday morning please",
audio, # float32 numpy array, 16 kHz mono, up to the moment you ask
tail_s=0.2, # how much silence the clip ends with
history=[{"role": "assistant", "content": "Which day would you like to travel instead?"}],
)
out["done"] # e.g. 0.87: probably done
out["wait_s"] # e.g. 0.25: answer after a quarter second of silence
```
With the same question before it and the same test clip, "Friday in the" gave `done` 0.12 and
`wait_s` 1.38. The exact numbers depend on the audio.
Everything but the text or the audio is optional. Without `history` the conversation model is
skipped; an empty list means "first turn of the conversation". Without `tail_s` the clip is
treated as ending right after the last word. Audio shorter than 3 seconds is fine. Without audio
you get `sentence_end` (and `context_end`) only.
## Ask twice per pause
The voice says more once a little silence has passed, and the conversation model needs up to
125 ms. So ask twice:
1. **As soon as the pause begins** (30 to 60 ms of silence in the clip, `tail_s` about 0.04).
You get a first `done` and `wait_s` after about 35 ms. Use it to prepare your reply.
2. **Again 0.18 s into the pause** (`tail_s` about 0.19), with the same text and history. Now
both audio models and the conversation count. Replace the wait with this answer.
You were not going to answer before a quarter second of silence anyway, so the second ask costs
no time. `inference.py` runs the conversation model in a thread beside the rest; in a server,
start it on the first ask and reuse its result for the second (it depends only on the text and
the history, not on the audio).
## The wait
`done` is the probability that the speaker is done at the moment the pause begins. A thinking
pause usually ends soon, so every further moment of silence makes "done" more likely.
`wait_s` is the moment at which the probability reaches the level you ask for (default 0.75),
between a minimum of 0.25 s and a maximum of 1.4 s. `wait_seconds(done, level, minimum, maximum)`
in `inference.py` lets you choose all three.
Measured on eot-bench (seven languages, the half not used for fitting the calibration): how
long real turn ends wait, and how many thinking pauses get cut into.
| Setup | Level | Turn ends wait p50 / p75 / p90 | Thinking pauses cut into |
|---|---|---|---|
| one ask at the start of the pause (v2) | 0.75 | 0.37 / 0.58 / 0.86 s | 26.0 % |
| two asks, two audio models | 0.75 | 0.28 / 0.52 / 0.76 s | 21.7 % |
| **two asks, two audio models, conversation** | **0.75** | **0.28 / 0.45 / 0.73 s** | **19.8 %** |
| two asks, two audio models, conversation | 0.70 | 0.28 / 0.38 / 0.65 s | 22.6 % |
| two asks, two audio models, conversation | 0.80 | 0.30 / 0.54 / 0.82 s | 17.0 % |
English and German are easier than the average: at level 0.75 the full setup cuts into 17 % and
13 % of their thinking pauses. Pause lengths in the benchmark are given in steps of 0.1 s; a
pause of 0.5 s is counted as anywhere between 0.45 and 0.55 s.
## The three raw numbers
| Output | From | Meaning |
|---|---|---|
| `sentence_end` | the words | the sentence is grammatically complete. Exactly the output of [SentenceFinishedClassification-multilingual v2](https://huggingface.co/KoljaB/SentenceFinishedClassification-multilingual). |
| `turn_end` | the voice and the words | the speaker is done. With `tail_s` of 0.12 or more it is the mean of both audio models. |
| `context_end` | the words and the conversation | the speaker is done, given what was said before. |
None of them is centred on 0.5; use `done`, which is calibrated, or fit your own bars.
## Results
AUC for "this pause is the end of the turn" (0.5 is guessing, 1.0 is perfect) on real user turns
from [eot-bench data](https://huggingface.co/datasets/livekit/eot-bench-data). None of it was
used in training. The table is the half of the pauses that was not used for fitting the
calibration, 360 to 460 per language.
| Language | Words only (`sentence_end`) | Words + conversation (`context_end`) | Voice + words, first ask | Voice + words, second ask | **Everything** | Turn ends answered at once* |
|---|---|---|---|---|---|---|
| English | 0.71 | 0.86 | 0.86 | 0.89 | **0.92** | 47 → 61 % |
| German | 0.73 | 0.85 | 0.81 | 0.87 | **0.91** | 32 → 42 % |
| Spanish | 0.71 | 0.81 | 0.88 | 0.90 | **0.92** | 37 → 49 % |
| French | 0.75 | 0.86 | 0.87 | 0.90 | **0.93** | 42 → 63 % |
| Italian | 0.71 | 0.83 | 0.79 | 0.88 | **0.90** | 38 → 53 % |
| Portuguese | 0.68 | 0.82 | 0.80 | 0.84 | **0.88** | 28 → 37 % |
| Chinese | 0.70 | 0.77 | 0.76 | 0.84 | **0.86** | 25 → 34 % |
| all | 0.71 | 0.83 | 0.82 | 0.87 | **0.90** | 34 → 44 % |
\* while cutting into 5 % of the thinking pauses: second ask without → with the conversation.
"First ask" is a clip ending 0.03 s into the silence, "second ask" one ending 0.24 s into it with
both audio models. These numbers come from our own protocol on the public data and are not
comparable to results from LiveKit's benchmark harness.
### English conversation between people (voice + words, one audio model)
| Test | Words only | Voice only | Voice + words |
|---|---|---|---|
| Open Yap, 36,964 pauses from 34 speakers the model never heard | 0.67 | - | **0.77** (0.76 with 0.03 s of silence) |
| [otoSpeech task-oriented](https://huggingface.co/datasets/otoearth/otoSpeech-full-duplex-task-oriented-20h), 14,900 pauses from 58 sessions (no transcripts) | - | **0.71** | - |
Conversation between people is harder than talking to an assistant: whether a pause becomes a
change of speaker also depends on the listener.
## Speed
Measured with the same script and the same files on two machines: CPU only (onnxruntime),
4 threads, median of 50 runs after 5 warm-up runs.
| Part | Windows, AMD Ryzen 9 3900X | Linux, Intel i9-13900KF, efficiency cores | Linux, Intel i9-13900KF, performance cores |
|---|---|---|---|
| Audio features (spectrogram of 3 s) | 1.2 ms | 0.9 ms | 0.6 ms |
| Text model, short utterance | 10.8 ms | 12.7 ms | 5.3 ms |
| Text model, long utterance (24 tokens) | 21.7 ms | 23.0 ms | 10.1 ms |
| Audio model (one) | 9.5 ms | 8.7 ms | 3.5 ms |
| Audio models (both, second ask) | 19.4 ms | 17.7 ms | 7.2 ms |
| Conversation model, short (11 tokens) | 14.2 ms | 15.8 ms | 6.6 ms |
| Conversation model, typical (55 tokens) | 44.8 ms | 44.5 ms | 18.4 ms |
| Conversation model, at its limit (128 tokens) | 110 ms | 99 ms | 42 ms |
| **First ask** (voice + text) | **22.5 ms** | **47 ms** | **21 ms** |
| Memory with all models loaded | about 1.0 GB | about 1.3 GB | about 1.3 GB |
On the efficiency cores the first ask took 47 ms although its parts add up to about 22 ms:
with several models in one process, idle worker threads have to wake up each time the
process switches to another model. Windows and the performance cores did not show this.
**The conversation model does not delay the answer.** It is the slowest part, but it runs
beside the rest: start it at the first ask, when the pause begins (about 0.04 s of silence),
and use its result at the second ask, about 0.19 s into the pause. Nobody answers before a
quarter second of silence anyway, so even 100 to 125 ms fit into the pause without anyone
hearing it. If the machine is busy and the result is not ready in time, decide without it
(voice + text only): the answer gets slightly less accurate, never later.
The conversation model was trained with up to 224 tokens of input. `inference.py` limits it to
128: the benchmark result is the same and it is ready in time for the second ask.
## Training
- **Audio models:** [`openai/whisper-tiny`](https://huggingface.co/openai/whisper-tiny) encoder (multilingual, 4 layers, Apache-2.0), fine-tuned on 3-second clips. `audio.onnx` saw clips ending at a random point between 0.03 and 0.24 s into the silence, `audio_late.onnx` only clips ending 0.24 s into it.
- **Audio data:** 193,842 real pauses from 843 two-speaker conversations (189 speakers, about 500 hours) of [Open Yap 1K](https://huggingface.co/datasets/TheAgenticDataCompany/open-yap-1k): 68,997 ends of turn, where the other person took over, and 124,845 thinking pauses, after which the same person went on. Pauses were found with a voice-activity detector (Silero VAD) on each speaker's own channel, the way a live system finds them.
- **Text model:** SentenceFinishedClassification-multilingual v2 ([`jhu-clsp/mmBERT-base`](https://huggingface.co/jhu-clsp/mmBERT-base), MIT), frozen. Its training is described on its own page.
- **Conversation model:** the same text model, trained further to answer a different question: given the last four messages and the utterance so far, is the speaker done? Its targets are judgements by Claude Haiku 5.5 (a number from 0 to 100 per pause), not human labels. It learned from about 24,000 pauses (1,000 more were held out for validation):
- 4,932 from 567 short spoken dialogues between a person and a voice assistant in the seven languages, written by Claude Sonnet 5.5 for this purpose, with the person's thinking pauses marked;
- about 12,000 real pauses from Open Yap conversations;
- about 8,100 user turns from [WildChat](https://huggingface.co/datasets/allenai/WildChat-1M) in the six languages other than English, whole or cut at a random word.
The written dialogues did most of the work: with the other two sources alone the model reached AUC 0.77 on eot-bench, with them 0.82.
- **Calibration and the pause-length curve** were fitted on half of the eot-bench pauses; the tables above use the other half.
This repository holds model weights only, no training data and no audio.
## Limits
- The audio models learned from English conversations between people. They carry over to the other six languages in the test above, but were not trained on them.
- The conversation model learned from a language model's judgement, mostly on written dialogues. Those dialogues were made to resemble spoken requests to an assistant, the kind of conversation eot-bench also contains, so its benchmark numbers may be kinder than what you see on your own users. Measure on your own recordings.
- The conversation model reads the last four messages, each cut to 400 characters, and 128 tokens in all. Anything earlier is invisible to it.
- The calibration and the pause-length curve come from one benchmark. Other speakers pause differently; the level is the knob to turn.
- It shortens waits and avoids cut-ins; it does not remove either. One thinking pause in five is still cut into at the default level.
- Overlapping speech, background voices and music were not part of the tests.
- Japanese, Korean and Russian are not covered.
## Files
| File | What |
|---|---|
| `onnx/audio.onnx` | audio encoder and turn-end head, 10 MB |
| `onnx/audio_late.onnx` | the same for clips that hold a quarter second of silence, 10 MB (optional) |
| `onnx/text.onnx` | text model, 369 MB |
| `onnx/context.onnx` | conversation model, 369 MB (optional) |
| `inference.py` | spectrogram, text clean-up, calibration, the wait rule and a minimal runner |
| `mel_filters.npy`, `tokenizer.json` | needed by `inference.py` |
| `eval.json` | the numbers above |
## Versions
- **v3** (this one): adds `audio_late.onnx`, `context.onnx`, the calibrated `done` and `wait_s`.
`audio.onnx` and `text.onnx` are the files of v2; code written for v2 keeps working and gets
the same `sentence_end` and `turn_end` when it passes no `tail_s`.
- **v2** (git tag `v2`): the audio model was retrained with a random amount of trailing silence
(0.03 to 0.24 s), so it can be asked right when a pause begins.
- **v1** (git tag `v1`): every training clip ended 0.24 s into the silence. Its audio model is
`audio_late.onnx` in v3.
## Credits
Open Yap 1K by The Agentic Data Company. Whisper by OpenAI. mmBERT by JHU CLSP. eot-bench data by
LiveKit. otoSpeech by oto. WildChat by the Allen Institute for AI. Silero VAD by the Silero team.
Training targets and dialogues for the conversation model by Claude (Anthropic).