language:
- en
pipeline_tag: audio-classification
tags:
- turn-detection
- end-of-turn
- eou
- endpointing
- turn-taking
- interruption-detection
- barge-in
- backchannel-detection
- full-duplex
- voice-activity-detection
- conversational-ai
- voice-agents
- real-time
- low-latency
- streaming
- audio
- speech
metrics:
- recall
- false-positive-rate
- latency
extra_gated_prompt: >-
Access to Echo Omni Duplex is provided through Zero Runtime. Visit
https://zeroruntime.ai/?utm_source=huggingface&utm_medium=referral&utm_campaign=huggingface
to get started.
Echo Omni Duplex listens to both sides of the conversation at once.
It knows who holds the floor, continuously, not once per turn.
🎧 Built for agents that listen while they speak, the full-duplex member of the Echo family.
A voice agent has to make one decision over and over: speak, or stay quiet. Most turn detectors answer it by waiting for silence, and a 500 ms gap looks identical whether someone finished their sentence, paused to think, or took a breath. Agents built on timers talk over people or leave them hanging.
Echo Omni Duplex answers it differently. It listens to both channels continuously, the user and the agent, and keeps a running estimate of who holds the conversational floor. Because it never stops listening to either side, it can tell a real barge-in from an "mm-hm", and a finished sentence from a breath, while the audio is still arriving.
🎯 What it does
Four things can happen around the moment a speaker stops or starts. Only two of them are events.
| Situation | What it means |
|---|---|
| End of turn | The speaker has given up the floor |
| Interruption | The listener is taking the floor mid-turn |
| Mid-turn pause | The speaker stopped, but has not finished |
| Backchannel | "uh-huh", "right", "okay". Acknowledgement, not a bid for the floor |
The last two are where most systems fail. A mid-turn pause looks like an ending, and a backchannel looks like a barge-in, so detectors fire on both. Echo Omni Duplex is built around telling them apart.
🔌 Input and output
Input: two continuous audio streams:
| Channel | Carries |
|---|---|
| Channel 1 | The user's microphone |
| Channel 2 | The agent's own output audio |
Output: emitted continuously as the conversation runs:
| Field | Description |
|---|---|
| Floor | A running estimate of which side currently holds the conversational floor |
| Event | A committed decision at the moment it is made: end-of-turn, interruption, or backchannel, an acknowledgement the agent should not stop for |
| Confidence | The floor probability the decision was made on, reported with each event |
Both channels are consumed as a live stream. There is no per-turn segmentation step and no transcript: the model sees the conversation as it happens. It runs in a live call, and how that is verified is set out under Causality below.
Why channel 2 matters. The strongest available cue for "has this person finished?" lives on the other side of the conversation, not the speaker's own. In a voice agent that other side is the agent, and its audio is known exactly. Feeding it in is the difference between guessing and knowing.
🧱 How it is built
Echo Omni Duplex is a single neural network rather than a pipeline of separate models, and it works in two stages. First, each side of the conversation is encoded on its own by a pretrained front end that stays frozen during training, so the user's audio and the agent's audio are never summed into one signal. Second, a causal model combines the two encoded streams, reading only the recent past and never anything ahead of the current moment. Both channels pass through the same weights, because turn-taking is symmetric: whichever side is speaking, the question being asked is the same one.
Because it only ever reads a bounded window of recent audio, the cost per frame stays constant however long the call runs. It works at 50 Hz and, rather than classifying the present, predicts who is about to be speaking, which is what lets it commit before a silence has gone on long enough to be conclusive on its own.
A deterministic commit policy sits on top and turns that continuous estimate into end-of-turn, interruption and backchannel decisions. It is separate from the network and tunable without retraining.
It is audio only: no transcript, no speech recognition, no language model. Small enough to run far faster than real time on a single GPU, alongside speech-to-text, an LLM and text-to-speech.
🎓 Training and validation
Echo Omni Duplex was trained on an internally curated corpus of two-channel, full-duplex English conversations. The corpus composition and the hours of data used are internal and not disclosed.
The model was validated on a held-out split of that same corpus, and its operating point was then selected on the public TurnBench dev set using the official scorer.
🌍 Supported languages
| Code | Language |
|---|---|
en |
English |
📊 Performance
Measured on the public development split of TurnBench, the benchmark described below, using its own scorer.
| Task | Recall ↑ | False-positive rate ↓ | Latency p50 ↓ |
|---|---|---|---|
| End-of-turn | 0.965 | 0.103 | 775 ms |
| Interruption | 0.988 | 0.098 | 590 ms |
Latency here is not compute time. It is the benchmark's own measure: how long after the real event the model committed to its decision. Most of that window is evidence the model is still waiting for, because deciding sooner means deciding on less. Every system on the benchmark pays it.
The false-positive rates are the numbers users feel. On end-of-turn, a false positive is the agent claiming the floor during a mid-turn pause, cutting somebody off mid-sentence. On interruption, it is the agent stopping because it mistook a backchannel, a cough or its own echo for a barge-in.
Echo Omni Duplex has also been evaluated on our own internal test set. Those results are not published here yet.
Results are measured on the benchmark described below. Performance may vary depending on language, deployment configuration, user behaviour and application requirements.
🏆 TurnBench
TurnBench is Sesame AI Labs' public benchmark for conversational turn-taking: dual-channel conversation scored on recall, false-positive rate and latency.
It scores committed decisions, not probability curves, which is what a deployed agent actually has to produce. At every moment a system either takes the floor or yields, and that is what gets measured.
Its authors summarise the state of the field this way:
"No system is fast, selective, and high-recall at the same time."
That is the trade-off Echo Omni Duplex was built against. On the interruption track it is higher recall, lower false-positive rate, and roughly 300 milliseconds sooner than Voice Activity Projection, the strongest of the benchmark's own baselines, all three at once and scored the same way on the same data.
It is not uniformly ahead. At least one published system commits interruptions faster still, at materially lower recall. The trade-off has moved; it has not disappeared.
🎚️ How the operating point was chosen
The model outputs a continuous estimate of who holds the floor. A policy turns that into discrete events, using thresholds and delays that have to be fixed before submitting. How they were picked:
- On the dev split only. Chosen against the official scorer on the public TurnBench dev set. The test split was never scored.
- Highest recall inside a 0.10 false-positive rate on dev. That budget is the one the benchmark's own published baselines were selected under, and it is well inside the 0.15 test ceiling a submission has to clear, so the choice is deliberately conservative.
- One setting for both files. The same policy produced the dev and test predictions. Nothing was tuned per split.
- We measured how much of the dev result was luck. Split the 38 dev conversations in half at random, pick the setting on one half, score it on the other, and repeat. The median drop in recall between the half it was chosen on and the half it was not was 0.0052, so the setting is not specific to the conversations that produced it.
⏱️ Causality
The output at any moment uses audio up to that moment and nothing after it.
Timestamps. Each one is a commit time: the moment the system decided, not the moment the event happened. We then report it 0.5 s later still, so a reported time is never earlier than the point at which the decision was made. The latency figures above are measured from those reported times, so they include that 0.5 s.
By construction. The model reads only a bounded window of the recent past, the voice-activity gate is causal, and the input gain uses only audio already heard.
By measurement. Construction arguments are easy to get wrong, so we tested it directly with a truncation check.
- What was tested. The public TurnBench dev split, 38 two-channel recordings of real human conversation. Each was cut at random points, and the whole system was re-run from scratch on the shortened audio, then compared against the run on the full recording.
- What the result means. If a system were secretly using audio from later in the file, removing that audio would change its earlier decisions. It does not: 330 of 336 event lists came back exactly as before, so the decisions stand on what had already been heard.
| what it means | ||
|---|---|---|
| Random cuts | 84, across 18 conversations | how many times the pipeline was re-run on shortened audio |
| Events compared | 19,828 | every event early enough to be checked, summed over all cuts |
| Event lists identical | 330 of 336 | each cut produces 4 lists, one per speaker per task; 330 came back exactly the same |
| Voice activity before each cut | identical | zero frames differed, so nothing ahead of the model was reading the future |
🔬 Beyond TurnBench
An independent test set. Krisp turn-taking-test-v1, 2,730 single-channel English clips: has the speaker finished, or are they only pausing? We used this dataset as testing set for our model.
| AUC ↑ | Balanced accuracy ↑ | |
|---|---|---|
| Echo Omni Duplex | 0.884 | 0.816 |
🎛️ The Echo family
Echo Omni Duplex is the full-duplex member. The other three decide one turn at a time, which suits a half-duplex pipeline; Duplex decides continuously, which is what an agent needs when it is listening and speaking at once.
| Model | Modality | Decides | Best for |
|---|---|---|---|
echo-small |
Semantic | Per turn | The lowest latency. The default when responsiveness matters most |
echo-large |
Semantic | Per turn | Higher accuracy, when it matters more than raw speed |
echo-omni |
Audio + semantic | Per turn | The widest coverage, with acoustic understanding on top of the semantics |
echo-omni-duplex |
Two-channel audio | Continuously | Full-duplex agents that listen while they speak |
🚀 Get access
Echo Omni Duplex is served for you. There is nothing to download, host, or keep running. Plug it straight into your voice pipeline alongside your existing STT, LLM and TTS.
# Set ZERORUNTIME_AUTH_TOKEN in your environment.
from zeroruntime.inference import TurnDetector
# Full duplex: both channels streamed continuously, events pushed back as they commit
turn_detector = TurnDetector(model="echo-omni-duplex")
Echo Omni Duplex by Zero Runtime