File size: 5,137 Bytes
bfa34ed b6cb557 bfa34ed b6cb557 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 | ---
license: bsd-3-clause
language:
- en
library_name: onnx
tags:
- pipecat
- turn-detection
- end-of-turn
- voice-agents
- streaming
- onnx
---
# Streaming Turn
Streaming, audio-only turn-taking model for [Pipecat](https://github.com/pipecat-ai/pipecat) voice agents. On every 40 ms of user audio it gives the probability that the user's turn is over, and whether speech over the bot is a backchannel or a floor-take. It never sees a transcript.
- 4.0M parameters: log-mel, causal conv encoder, GRU. 16 kHz mono in, 25 frames a second out.
- 0.37 ms per 40 ms chunk on one Apple M-series core (ONNX Runtime).
- English only. Experimental: one training run per version, and little live testing.
## Versions
| Folder | End the turn when |
|---|---|
| `streaming-turn-v0.4` (latest) | `eot` ≥ 0.5, or `eot_late` ≥ 0.4 |
| `streaming-turn-v0.2` | `eot` ≥ 0.5, or 0.8 s after the last sound if `eot` reached 0.3 |
v0.4 adds `eot_late`, a second end-of-turn output for a user who has gone quiet.
Each folder holds `streaming-turn-fp32.onnx` (the streaming model), `model.json` (tensor names, chunk size, class lists) and a `.pt` file (PyTorch config and weights).
## Use with Pipecat
`agent.turn_analyzers` is in the code repository, [jptaylor/streaming-turn](https://github.com/jptaylor/streaming-turn).
```python
from pathlib import Path
from huggingface_hub import snapshot_download
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.frames.frames import BotStartedSpeakingFrame, BotStoppedSpeakingFrame
from pipecat.processors.aggregators.llm_response_universal import (
LLMContext,
LLMContextAggregatorPair,
LLMUserAggregatorParams,
)
from pipecat.turns.user_stop import TurnAnalyzerUserTurnStopStrategy
from pipecat.turns.user_turn_strategies import UserTurnStrategies
from agent.turn_analyzers import StreamingTurnAnalyzer, StreamingTurnParams
VERSION = "streaming-turn-v0.4"
repo = snapshot_download("jptaylor/streaming-turn", allow_patterns=f"{VERSION}/*")
analyzer = StreamingTurnAnalyzer(
Path(repo) / VERSION,
params=StreamingTurnParams(threshold=0.5, late_threshold=0.4),
# v0.2: StreamingTurnParams(threshold=0.5, likely_threshold=0.3)
)
class StreamingTurnStop(TurnAnalyzerUserTurnStopStrategy):
"""Tells the model when the bot is speaking: it was trained with that input."""
async def process_frame(self, frame):
if isinstance(frame, (BotStartedSpeakingFrame, BotStoppedSpeakingFrame)):
analyzer.set_bot_speaking(isinstance(frame, BotStartedSpeakingFrame))
return await super().process_frame(frame)
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
LLMContext(),
user_params=LLMUserAggregatorParams(
vad_analyzer=SileroVADAnalyzer(),
user_turn_strategies=UserTurnStrategies(stop=[StreamingTurnStop(turn_analyzer=analyzer)]),
),
)
```
## Outputs
One frame per 40 ms chunk.
| Output | |
|---|---|
| `eot` | probability that the user's turn is over |
| `eot_late` | the same, for a user who has gone quiet (v0.4 only) |
| `intent` | continuer, assessment, negative, floor_take, non_speech. Backchannel = continuer + assessment; floor-take = negative + floor_take |
| `leadin` | none, acknowledgement, hesitation, marker, reaction (barely beats guessing) |
| `vad`, `vap` | auxiliary: voice activity now, and both speakers' activity over the next 2 s |
Inputs: `chunk` (640 samples, float32) and `bot_state` (is the bot speaking, and for how long), plus the streaming state tensors named in `model.json`.
## Results
Held-out Open Yap 1K conversations (1,259 turn ends, 8,191 pauses), replayed through Pipecat's Silero VAD with nobody answering, at the settings above:
| | Cuts in on a pause | Answers within 500 ms | Within 1 s | Waits for the 3 s timeout |
|---|---|---|---|---|
| v0.4 | 4.9% | 56.4% | 79.7% | 2.9% |
| v0.2 | 4.6% | 51.4% | 82.2% | 15.5% |
On Smart Turn v3.2's English test clips (7,820, read 200 ms after the speech ends): v0.4 93.3%, v0.2 93.8%, Smart Turn v3.2 93.5%.
## Limitations
- The held-out conversations are from a corpus the model trained on (different conversations).
- Keep `bot_state` up to date. v0.4 holds back until the bot has spoken in a session.
- A finished sentence it takes for an unfinished one waits for the 3 s timeout.
- Half of the end-of-turn targets on Smart Turn clips are Smart Turn v3.2's own predictions, so it is not independent of v3.2.
- Trained on human-to-human conversation and single-speaker clips, with simulated echo. Phone codecs are untested.
## Training data
[Open Yap 1K](https://theagenticdatacompany.com/open-yap-1k) v1.0, [otoSpeech-full-duplex-280h](https://huggingface.co/datasets/otoearth/otoSpeech-full-duplex-280h) (CC BY 4.0), the English part of the [Smart Turn v3.2 training data](https://huggingface.co/datasets/pipecat-ai/smart-turn-data-v3.2-train) (CC BY 4.0), and 600 synthetic hard negatives. A frozen [Whisper base.en](https://huggingface.co/openai/whisper-base.en) encoder was the distillation target, and Smart Turn v3.2 gave soft end-of-turn labels. No training data is included here.
|