|
Download README.md from KoljaB/TurnEndDetection-multilingual: direct link, hf CLI and curl.
- Browser
- Download file 14.5 kB
-
https://huggingface.co/KoljaB/TurnEndDetection-multilingual/resolve/main/README.md
- Command line
-
hf download hf://KoljaB/TurnEndDetection-multilingual/README.md
-
curl -L -H "Authorization: Bearer $HF_TOKEN" -o README.md https://huggingface.co/KoljaB/TurnEndDetection-multilingual/resolve/main/README.md
14.5 kB
| license: apache-2.0 | |
| language: [en, de, es, fr, it, pt, zh] | |
| library_name: onnxruntime | |
| pipeline_tag: audio-classification | |
| base_model: | |
| - openai/whisper-tiny | |
| - jhu-clsp/mmBERT-base | |
| tags: [turn-detection, end-of-turn, voice-assistant, conversational-ai, onnx] | |
| # TurnEndDetection multilingual (v3) | |
| [GitHub](https://github.com/KoljaB/) · [X](https://x.com/LonLigrin) · [YouTube](https://www.youtube.com/@Linguflex) · [Discord](https://discord.com/invite/f556hqRjpv) | |
| People pause in the middle of a turn all the time, and almost half of those thinking pauses come | |
| right after a grammatically complete sentence. A model that only reads the words cannot tell | |
| "finished sentence, still thinking" from "done, your turn". Two things can: how the last words | |
| were spoken, and what the conversation was about. "Friday" is a whole turn after "Which day?", | |
| and only a beginning after "Tell me about your week". | |
| This model listens to the last three seconds before a pause, reads the words, and reads the last | |
| few messages of the conversation. It returns one calibrated probability that the speaker is done | |
| and a suggested wait, so a voice assistant can answer at once when the user is done and hold back | |
| when they are not. | |
| - **Input:** the text of the current utterance, the last 3 seconds of audio (16 kHz mono), and optionally the messages before the turn. | |
| - **Output:** `done`, the probability that the speaker is done, and `wait_s`, how long to stay silent before answering. Also the three raw numbers: `sentence_end`, `turn_end` and `context_end`. | |
| - **Languages:** English, German, Spanish, French, Italian, Portuguese, Chinese. | |
| - **Speed (4 CPU threads):** voice + text about 32 ms; the conversation model 20 to 125 ms more, in parallel. | |
| - **Size:** 369 MB text model, 369 MB conversation model, two audio models of 10 MB each (ONNX). | |
| What is new in v3: a second audio model for clips that already hold some silence, the conversation | |
| model, and a calibrated rule that turns everything into one probability and a wait. Each part is | |
| optional; the files of v2 are unchanged. | |
| ## Quick start | |
| ```bash | |
| pip install onnxruntime tokenizers numpy huggingface_hub | |
| ``` | |
| ```python | |
| import sys | |
| from huggingface_hub import snapshot_download | |
| sys.path.insert(0, snapshot_download("KoljaB/TurnEndDetection-multilingual")) | |
| from inference import TurnModel | |
| model = TurnModel() | |
| out = model( | |
| "Friday morning please", | |
| audio, # float32 numpy array, 16 kHz mono, up to the moment you ask | |
| tail_s=0.2, # how much silence the clip ends with | |
| history=[{"role": "assistant", "content": "Which day would you like to travel instead?"}], | |
| ) | |
| out["done"] # e.g. 0.87: probably done | |
| out["wait_s"] # e.g. 0.25: answer after a quarter second of silence | |
| ``` | |
| With the same question before it and the same test clip, "Friday in the" gave `done` 0.12 and | |
| `wait_s` 1.38. The exact numbers depend on the audio. | |
| Everything but the text or the audio is optional. Without `history` the conversation model is | |
| skipped; an empty list means "first turn of the conversation". Without `tail_s` the clip is | |
| treated as ending right after the last word. Audio shorter than 3 seconds is fine. Without audio | |
| you get `sentence_end` (and `context_end`) only. | |
| ## Ask twice per pause | |
| The voice says more once a little silence has passed, and the conversation model needs up to | |
| 125 ms. So ask twice: | |
| 1. **As soon as the pause begins** (30 to 60 ms of silence in the clip, `tail_s` about 0.04). | |
| You get a first `done` and `wait_s` after about 35 ms. Use it to prepare your reply. | |
| 2. **Again 0.18 s into the pause** (`tail_s` about 0.19), with the same text and history. Now | |
| both audio models and the conversation count. Replace the wait with this answer. | |
| You were not going to answer before a quarter second of silence anyway, so the second ask costs | |
| no time. `inference.py` runs the conversation model in a thread beside the rest; in a server, | |
| start it on the first ask and reuse its result for the second (it depends only on the text and | |
| the history, not on the audio). | |
| ## The wait | |
| `done` is the probability that the speaker is done at the moment the pause begins. A thinking | |
| pause usually ends soon, so every further moment of silence makes "done" more likely. | |
| `wait_s` is the moment at which the probability reaches the level you ask for (default 0.75), | |
| between a minimum of 0.25 s and a maximum of 1.4 s. `wait_seconds(done, level, minimum, maximum)` | |
| in `inference.py` lets you choose all three. | |
| Measured on eot-bench (seven languages, the half not used for fitting the calibration): how | |
| long real turn ends wait, and how many thinking pauses get cut into. | |
| | Setup | Level | Turn ends wait p50 / p75 / p90 | Thinking pauses cut into | | |
| |---|---|---|---| | |
| | one ask at the start of the pause (v2) | 0.75 | 0.37 / 0.58 / 0.86 s | 26.0 % | | |
| | two asks, two audio models | 0.75 | 0.28 / 0.52 / 0.76 s | 21.7 % | | |
| | **two asks, two audio models, conversation** | **0.75** | **0.28 / 0.45 / 0.73 s** | **19.8 %** | | |
| | two asks, two audio models, conversation | 0.70 | 0.28 / 0.38 / 0.65 s | 22.6 % | | |
| | two asks, two audio models, conversation | 0.80 | 0.30 / 0.54 / 0.82 s | 17.0 % | | |
| English and German are easier than the average: at level 0.75 the full setup cuts into 17 % and | |
| 13 % of their thinking pauses. Pause lengths in the benchmark are given in steps of 0.1 s; a | |
| pause of 0.5 s is counted as anywhere between 0.45 and 0.55 s. | |
| ## The three raw numbers | |
| | Output | From | Meaning | | |
| |---|---|---| | |
| | `sentence_end` | the words | the sentence is grammatically complete. Exactly the output of [SentenceFinishedClassification-multilingual v2](https://huggingface.co/KoljaB/SentenceFinishedClassification-multilingual). | | |
| | `turn_end` | the voice and the words | the speaker is done. With `tail_s` of 0.12 or more it is the mean of both audio models. | | |
| | `context_end` | the words and the conversation | the speaker is done, given what was said before. | | |
| None of them is centred on 0.5; use `done`, which is calibrated, or fit your own bars. | |
| ## Results | |
| AUC for "this pause is the end of the turn" (0.5 is guessing, 1.0 is perfect) on real user turns | |
| from [eot-bench data](https://huggingface.co/datasets/livekit/eot-bench-data). None of it was | |
| used in training. The table is the half of the pauses that was not used for fitting the | |
| calibration, 360 to 460 per language. | |
| | Language | Words only (`sentence_end`) | Words + conversation (`context_end`) | Voice + words, first ask | Voice + words, second ask | **Everything** | Turn ends answered at once* | | |
| |---|---|---|---|---|---|---| | |
| | English | 0.71 | 0.86 | 0.86 | 0.89 | **0.92** | 47 → 61 % | | |
| | German | 0.73 | 0.85 | 0.81 | 0.87 | **0.91** | 32 → 42 % | | |
| | Spanish | 0.71 | 0.81 | 0.88 | 0.90 | **0.92** | 37 → 49 % | | |
| | French | 0.75 | 0.86 | 0.87 | 0.90 | **0.93** | 42 → 63 % | | |
| | Italian | 0.71 | 0.83 | 0.79 | 0.88 | **0.90** | 38 → 53 % | | |
| | Portuguese | 0.68 | 0.82 | 0.80 | 0.84 | **0.88** | 28 → 37 % | | |
| | Chinese | 0.70 | 0.77 | 0.76 | 0.84 | **0.86** | 25 → 34 % | | |
| | all | 0.71 | 0.83 | 0.82 | 0.87 | **0.90** | 34 → 44 % | | |
| \* while cutting into 5 % of the thinking pauses: second ask without → with the conversation. | |
| "First ask" is a clip ending 0.03 s into the silence, "second ask" one ending 0.24 s into it with | |
| both audio models. These numbers come from our own protocol on the public data and are not | |
| comparable to results from LiveKit's benchmark harness. | |
| ### English conversation between people (voice + words, one audio model) | |
| | Test | Words only | Voice only | Voice + words | | |
| |---|---|---|---| | |
| | Open Yap, 36,964 pauses from 34 speakers the model never heard | 0.67 | - | **0.77** (0.76 with 0.03 s of silence) | | |
| | [otoSpeech task-oriented](https://huggingface.co/datasets/otoearth/otoSpeech-full-duplex-task-oriented-20h), 14,900 pauses from 58 sessions (no transcripts) | - | **0.71** | - | | |
| Conversation between people is harder than talking to an assistant: whether a pause becomes a | |
| change of speaker also depends on the listener. | |
| ## Speed | |
| Measured with the same script and the same files on two machines: CPU only (onnxruntime), | |
| 4 threads, median of 50 runs after 5 warm-up runs. | |
| | Part | Windows, AMD Ryzen 9 3900X | Linux, Intel i9-13900KF, efficiency cores | Linux, Intel i9-13900KF, performance cores | | |
| |---|---|---|---| | |
| | Audio features (spectrogram of 3 s) | 1.2 ms | 0.9 ms | 0.6 ms | | |
| | Text model, short utterance | 10.8 ms | 12.7 ms | 5.3 ms | | |
| | Text model, long utterance (24 tokens) | 21.7 ms | 23.0 ms | 10.1 ms | | |
| | Audio model (one) | 9.5 ms | 8.7 ms | 3.5 ms | | |
| | Audio models (both, second ask) | 19.4 ms | 17.7 ms | 7.2 ms | | |
| | Conversation model, short (11 tokens) | 14.2 ms | 15.8 ms | 6.6 ms | | |
| | Conversation model, typical (55 tokens) | 44.8 ms | 44.5 ms | 18.4 ms | | |
| | Conversation model, at its limit (128 tokens) | 110 ms | 99 ms | 42 ms | | |
| | **First ask** (voice + text) | **22.5 ms** | **47 ms** | **21 ms** | | |
| | Memory with all models loaded | about 1.0 GB | about 1.3 GB | about 1.3 GB | | |
| On the efficiency cores the first ask took 47 ms although its parts add up to about 22 ms: | |
| with several models in one process, idle worker threads have to wake up each time the | |
| process switches to another model. Windows and the performance cores did not show this. | |
| **The conversation model does not delay the answer.** It is the slowest part, but it runs | |
| beside the rest: start it at the first ask, when the pause begins (about 0.04 s of silence), | |
| and use its result at the second ask, about 0.19 s into the pause. Nobody answers before a | |
| quarter second of silence anyway, so even 100 to 125 ms fit into the pause without anyone | |
| hearing it. If the machine is busy and the result is not ready in time, decide without it | |
| (voice + text only): the answer gets slightly less accurate, never later. | |
| The conversation model was trained with up to 224 tokens of input. `inference.py` limits it to | |
| 128: the benchmark result is the same and it is ready in time for the second ask. | |
| ## Training | |
| - **Audio models:** [`openai/whisper-tiny`](https://huggingface.co/openai/whisper-tiny) encoder (multilingual, 4 layers, Apache-2.0), fine-tuned on 3-second clips. `audio.onnx` saw clips ending at a random point between 0.03 and 0.24 s into the silence, `audio_late.onnx` only clips ending 0.24 s into it. | |
| - **Audio data:** 193,842 real pauses from 843 two-speaker conversations (189 speakers, about 500 hours) of [Open Yap 1K](https://huggingface.co/datasets/TheAgenticDataCompany/open-yap-1k): 68,997 ends of turn, where the other person took over, and 124,845 thinking pauses, after which the same person went on. Pauses were found with a voice-activity detector (Silero VAD) on each speaker's own channel, the way a live system finds them. | |
| - **Text model:** SentenceFinishedClassification-multilingual v2 ([`jhu-clsp/mmBERT-base`](https://huggingface.co/jhu-clsp/mmBERT-base), MIT), frozen. Its training is described on its own page. | |
| - **Conversation model:** the same text model, trained further to answer a different question: given the last four messages and the utterance so far, is the speaker done? Its targets are judgements by Claude Haiku 5.5 (a number from 0 to 100 per pause), not human labels. It learned from about 24,000 pauses (1,000 more were held out for validation): | |
| - 4,932 from 567 short spoken dialogues between a person and a voice assistant in the seven languages, written by Claude Sonnet 5.5 for this purpose, with the person's thinking pauses marked; | |
| - about 12,000 real pauses from Open Yap conversations; | |
| - about 8,100 user turns from [WildChat](https://huggingface.co/datasets/allenai/WildChat-1M) in the six languages other than English, whole or cut at a random word. | |
| The written dialogues did most of the work: with the other two sources alone the model reached AUC 0.77 on eot-bench, with them 0.82. | |
| - **Calibration and the pause-length curve** were fitted on half of the eot-bench pauses; the tables above use the other half. | |
| This repository holds model weights only, no training data and no audio. | |
| ## Limits | |
| - The audio models learned from English conversations between people. They carry over to the other six languages in the test above, but were not trained on them. | |
| - The conversation model learned from a language model's judgement, mostly on written dialogues. Those dialogues were made to resemble spoken requests to an assistant, the kind of conversation eot-bench also contains, so its benchmark numbers may be kinder than what you see on your own users. Measure on your own recordings. | |
| - The conversation model reads the last four messages, each cut to 400 characters, and 128 tokens in all. Anything earlier is invisible to it. | |
| - The calibration and the pause-length curve come from one benchmark. Other speakers pause differently; the level is the knob to turn. | |
| - It shortens waits and avoids cut-ins; it does not remove either. One thinking pause in five is still cut into at the default level. | |
| - Overlapping speech, background voices and music were not part of the tests. | |
| - Japanese, Korean and Russian are not covered. | |
| ## Files | |
| | File | What | | |
| |---|---| | |
| | `onnx/audio.onnx` | audio encoder and turn-end head, 10 MB | | |
| | `onnx/audio_late.onnx` | the same for clips that hold a quarter second of silence, 10 MB (optional) | | |
| | `onnx/text.onnx` | text model, 369 MB | | |
| | `onnx/context.onnx` | conversation model, 369 MB (optional) | | |
| | `inference.py` | spectrogram, text clean-up, calibration, the wait rule and a minimal runner | | |
| | `mel_filters.npy`, `tokenizer.json` | needed by `inference.py` | | |
| | `eval.json` | the numbers above | | |
| ## Versions | |
| - **v3** (this one): adds `audio_late.onnx`, `context.onnx`, the calibrated `done` and `wait_s`. | |
| `audio.onnx` and `text.onnx` are the files of v2; code written for v2 keeps working and gets | |
| the same `sentence_end` and `turn_end` when it passes no `tail_s`. | |
| - **v2** (git tag `v2`): the audio model was retrained with a random amount of trailing silence | |
| (0.03 to 0.24 s), so it can be asked right when a pause begins. | |
| - **v1** (git tag `v1`): every training clip ended 0.24 s into the silence. Its audio model is | |
| `audio_late.onnx` in v3. | |
| ## Credits | |
| Open Yap 1K by The Agentic Data Company. Whisper by OpenAI. mmBERT by JHU CLSP. eot-bench data by | |
| LiveKit. otoSpeech by oto. WildChat by the Allen Institute for AI. Silero VAD by the Silero team. | |
| Training targets and dialogues for the conversation model by Claude (Anthropic). | |