---
license: bsd-3-clause
language:
- en
library_name: onnx
tags:
- pipecat
- turn-detection
- end-of-turn
- voice-agents
- streaming
- onnx
---

# Streaming Turn

Streaming, audio-only turn-taking model for [Pipecat](https://github.com/pipecat-ai/pipecat) voice agents. On every 40 ms of user audio it gives the probability that the user's turn is over, and whether speech over the bot is a backchannel or a floor-take. It never sees a transcript.

- 4.0M parameters: log-mel, causal conv encoder, GRU. 16 kHz mono in, 25 frames a second out.
- 0.37 ms per 40 ms chunk on one Apple M-series core (ONNX Runtime).
- English only. Experimental: one training run per version, and little live testing.

## Versions

| Folder | End the turn when |
|---|---|
| `streaming-turn-v0.4` (latest) | `eot` ≥ 0.5, or `eot_late` ≥ 0.4 |
| `streaming-turn-v0.2` | `eot` ≥ 0.5, or 0.8 s after the last sound if `eot` reached 0.3 |

v0.4 adds `eot_late`, a second end-of-turn output for a user who has gone quiet.

Each folder holds `streaming-turn-fp32.onnx` (the streaming model), `model.json` (tensor names, chunk size, class lists) and a `.pt` file (PyTorch config and weights).

## Use with Pipecat

`agent.turn_analyzers` is in the code repository, [jptaylor/streaming-turn](https://github.com/jptaylor/streaming-turn).

```python
from pathlib import Path

from huggingface_hub import snapshot_download
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.frames.frames import BotStartedSpeakingFrame, BotStoppedSpeakingFrame
from pipecat.processors.aggregators.llm_response_universal import (
    LLMContext,
    LLMContextAggregatorPair,
    LLMUserAggregatorParams,
)
from pipecat.turns.user_stop import TurnAnalyzerUserTurnStopStrategy
from pipecat.turns.user_turn_strategies import UserTurnStrategies

from agent.turn_analyzers import StreamingTurnAnalyzer, StreamingTurnParams

VERSION = "streaming-turn-v0.4"
repo = snapshot_download("jptaylor/streaming-turn", allow_patterns=f"{VERSION}/*")
analyzer = StreamingTurnAnalyzer(
    Path(repo) / VERSION,
    params=StreamingTurnParams(threshold=0.5, late_threshold=0.4),
    # v0.2: StreamingTurnParams(threshold=0.5, likely_threshold=0.3)
)


class StreamingTurnStop(TurnAnalyzerUserTurnStopStrategy):
    """Tells the model when the bot is speaking: it was trained with that input."""

    async def process_frame(self, frame):
        if isinstance(frame, (BotStartedSpeakingFrame, BotStoppedSpeakingFrame)):
            analyzer.set_bot_speaking(isinstance(frame, BotStartedSpeakingFrame))
        return await super().process_frame(frame)


user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    LLMContext(),
    user_params=LLMUserAggregatorParams(
        vad_analyzer=SileroVADAnalyzer(),
        user_turn_strategies=UserTurnStrategies(stop=[StreamingTurnStop(turn_analyzer=analyzer)]),
    ),
)
```

## Outputs

One frame per 40 ms chunk.

| Output | |
|---|---|
| `eot` | probability that the user's turn is over |
| `eot_late` | the same, for a user who has gone quiet (v0.4 only) |
| `intent` | continuer, assessment, negative, floor_take, non_speech. Backchannel = continuer + assessment; floor-take = negative + floor_take |
| `leadin` | none, acknowledgement, hesitation, marker, reaction (barely beats guessing) |
| `vad`, `vap` | auxiliary: voice activity now, and both speakers' activity over the next 2 s |

Inputs: `chunk` (640 samples, float32) and `bot_state` (is the bot speaking, and for how long), plus the streaming state tensors named in `model.json`.

## Results

Held-out Open Yap 1K conversations (1,259 turn ends, 8,191 pauses), replayed through Pipecat's Silero VAD with nobody answering, at the settings above:

| | Cuts in on a pause | Answers within 500 ms | Within 1 s | Waits for the 3 s timeout |
|---|---|---|---|---|
| v0.4 | 4.9% | 56.4% | 79.7% | 2.9% |
| v0.2 | 4.6% | 51.4% | 82.2% | 15.5% |

On Smart Turn v3.2's English test clips (7,820, read 200 ms after the speech ends): v0.4 93.3%, v0.2 93.8%, Smart Turn v3.2 93.5%.

## Limitations

- The held-out conversations are from a corpus the model trained on (different conversations).
- Keep `bot_state` up to date. v0.4 holds back until the bot has spoken in a session.
- A finished sentence it takes for an unfinished one waits for the 3 s timeout.
- Half of the end-of-turn targets on Smart Turn clips are Smart Turn v3.2's own predictions, so it is not independent of v3.2.
- Trained on human-to-human conversation and single-speaker clips, with simulated echo. Phone codecs are untested.

## Training data

[Open Yap 1K](https://theagenticdatacompany.com/open-yap-1k) v1.0, [otoSpeech-full-duplex-280h](https://huggingface.co/datasets/otoearth/otoSpeech-full-duplex-280h) (CC BY 4.0), the English part of the [Smart Turn v3.2 training data](https://huggingface.co/datasets/pipecat-ai/smart-turn-data-v3.2-train) (CC BY 4.0), and 600 synthetic hard negatives. A frozen [Whisper base.en](https://huggingface.co/openai/whisper-base.en) encoder was the distillation target, and Smart Turn v3.2 gave soft end-of-turn labels. No training data is included here.