Streaming Turn v0.4 and v0.2: ONNX models, PyTorch weights, model card

b6cb557 verified 1 day ago

5.14 kB

---
license: bsd-3-clause
language:
- en
library_name: onnx
tags:
- pipecat
- turn-detection
- end-of-turn
- voice-agents
- streaming
- onnx
---
# Streaming Turn

Streaming, audio-only turn-taking model for [Pipecat](https://github.com/pipecat-ai/pipecat) voice agents. On every 40 ms of user audio it gives the probability that the user's turn is over, and whether speech over the bot is a backchannel or a floor-take. It never sees a transcript.

- 4.0M parameters: log-mel, causal conv encoder, GRU. 16 kHz mono in, 25 frames a second out.
- 0.37 ms per 40 ms chunk on one Apple M-series core (ONNX Runtime).
- English only. Experimental: one training run per version, and little live testing.

## Versions

| Folder | End the turn when |
|---|---|
| `streaming-turn-v0.4` (latest) | `eot` ≥ 0.5, or `eot_late` ≥ 0.4 |
| `streaming-turn-v0.2` | `eot` ≥ 0.5, or 0.8 s after the last sound if `eot` reached 0.3 |

v0.4 adds `eot_late`, a second end-of-turn output for a user who has gone quiet.

Each folder holds `streaming-turn-fp32.onnx` (the streaming model), `model.json` (tensor names, chunk size, class lists) and a `.pt` file (PyTorch config and weights).

## Use with Pipecat

`agent.turn_analyzers` is in the code repository, [jptaylor/streaming-turn](https://github.com/jptaylor/streaming-turn).

```python
from pathlib import Path
from huggingface_hub import snapshot_download
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.frames.frames import BotStartedSpeakingFrame, BotStoppedSpeakingFrame
from pipecat.processors.aggregators.llm_response_universal import (
    LLMContext,
    LLMContextAggregatorPair,
    LLMUserAggregatorParams,
)
from pipecat.turns.user_stop import TurnAnalyzerUserTurnStopStrategy
from pipecat.turns.user_turn_strategies import UserTurnStrategies
from agent.turn_analyzers import StreamingTurnAnalyzer, StreamingTurnParams
VERSION = "streaming-turn-v0.4"
repo = snapshot_download("jptaylor/streaming-turn", allow_patterns=f"{VERSION}/*")
analyzer = StreamingTurnAnalyzer(
    Path(repo) / VERSION,
    params=StreamingTurnParams(threshold=0.5, late_threshold=0.4),
    # v0.2: StreamingTurnParams(threshold=0.5, likely_threshold=0.3)
)
class StreamingTurnStop(TurnAnalyzerUserTurnStopStrategy):
    """Tells the model when the bot is speaking: it was trained with that input."""
    async def process_frame(self, frame):
        if isinstance(frame, (BotStartedSpeakingFrame, BotStoppedSpeakingFrame)):
            analyzer.set_bot_speaking(isinstance(frame, BotStartedSpeakingFrame))
        return await super().process_frame(frame)
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    LLMContext(),
    user_params=LLMUserAggregatorParams(
        vad_analyzer=SileroVADAnalyzer(),
        user_turn_strategies=UserTurnStrategies(stop=[StreamingTurnStop(turn_analyzer=analyzer)]),
    ),
)
```

## Outputs

One frame per 40 ms chunk.

| Output | |
|---|---|
| `eot` | probability that the user's turn is over |
| `eot_late` | the same, for a user who has gone quiet (v0.4 only) |
| `intent` | continuer, assessment, negative, floor_take, non_speech. Backchannel = continuer + assessment; floor-take = negative + floor_take |
| `leadin` | none, acknowledgement, hesitation, marker, reaction (barely beats guessing) |
| `vad`, `vap` | auxiliary: voice activity now, and both speakers' activity over the next 2 s |
Inputs: `chunk` (640 samples, float32) and `bot_state` (is the bot speaking, and for how long), plus the streaming state tensors named in `model.json`.

## Results

Held-out Open Yap 1K conversations (1,259 turn ends, 8,191 pauses), replayed through Pipecat's Silero VAD with nobody answering, at the settings above:

| | Cuts in on a pause | Answers within 500 ms | Within 1 s | Waits for the 3 s timeout |
|---|---|---|---|---|
| v0.4 | 4.9% | 56.4% | 79.7% | 2.9% |
| v0.2 | 4.6% | 51.4% | 82.2% | 15.5% |

On Smart Turn v3.2's English test clips (7,820, read 200 ms after the speech ends): v0.4 93.3%, v0.2 93.8%, Smart Turn v3.2 93.5%.

## Limitations

- The held-out conversations are from a corpus the model trained on (different conversations).
- Keep `bot_state` up to date. v0.4 holds back until the bot has spoken in a session.
- A finished sentence it takes for an unfinished one waits for the 3 s timeout.
- Half of the end-of-turn targets on Smart Turn clips are Smart Turn v3.2's own predictions, so it is not independent of v3.2.
- Trained on human-to-human conversation and single-speaker clips, with simulated echo. Phone codecs are untested.

## Training data

[Open Yap 1K](https://theagenticdatacompany.com/open-yap-1k) v1.0, [otoSpeech-full-duplex-280h](https://huggingface.co/datasets/otoearth/otoSpeech-full-duplex-280h) (CC BY 4.0), the English part of the [Smart Turn v3.2 training data](https://huggingface.co/datasets/pipecat-ai/smart-turn-data-v3.2-train) (CC BY 4.0), and 600 synthetic hard negatives. A frozen [Whisper base.en](https://huggingface.co/openai/whisper-base.en) encoder was the distillation target, and Smart Turn v3.2 gave soft end-of-turn labels. No training data is included here.

	---
	license: bsd-3-clause
	language:
	- en
	library_name: onnx
	tags:
	- pipecat
	- turn-detection
	- end-of-turn
	- voice-agents
	- streaming
	- onnx
	---

	# Streaming Turn

	Streaming, audio-only turn-taking model for [Pipecat](https://github.com/pipecat-ai/pipecat) voice agents. On every 40 ms of user audio it gives the probability that the user's turn is over, and whether speech over the bot is a backchannel or a floor-take. It never sees a transcript.

	- 4.0M parameters: log-mel, causal conv encoder, GRU. 16 kHz mono in, 25 frames a second out.
	- 0.37 ms per 40 ms chunk on one Apple M-series core (ONNX Runtime).
	- English only. Experimental: one training run per version, and little live testing.

	## Versions

	\| Folder \| End the turn when \|
	\|---\|---\|
	\| `streaming-turn-v0.4` (latest) \| `eot` ≥ 0.5, or `eot_late` ≥ 0.4 \|
	\| `streaming-turn-v0.2` \| `eot` ≥ 0.5, or 0.8 s after the last sound if `eot` reached 0.3 \|

	v0.4 adds `eot_late`, a second end-of-turn output for a user who has gone quiet.

	Each folder holds `streaming-turn-fp32.onnx` (the streaming model), `model.json` (tensor names, chunk size, class lists) and a `.pt` file (PyTorch config and weights).

	## Use with Pipecat

	`agent.turn_analyzers` is in the code repository, [jptaylor/streaming-turn](https://github.com/jptaylor/streaming-turn).

	```python
	from pathlib import Path

	from huggingface_hub import snapshot_download
	from pipecat.audio.vad.silero import SileroVADAnalyzer
	from pipecat.frames.frames import BotStartedSpeakingFrame, BotStoppedSpeakingFrame
	from pipecat.processors.aggregators.llm_response_universal import (
	LLMContext,
	LLMContextAggregatorPair,
	LLMUserAggregatorParams,
	)
	from pipecat.turns.user_stop import TurnAnalyzerUserTurnStopStrategy
	from pipecat.turns.user_turn_strategies import UserTurnStrategies

	from agent.turn_analyzers import StreamingTurnAnalyzer, StreamingTurnParams

	VERSION = "streaming-turn-v0.4"
	repo = snapshot_download("jptaylor/streaming-turn", allow_patterns=f"{VERSION}/*")
	analyzer = StreamingTurnAnalyzer(
	Path(repo) / VERSION,
	params=StreamingTurnParams(threshold=0.5, late_threshold=0.4),
	# v0.2: StreamingTurnParams(threshold=0.5, likely_threshold=0.3)
	)


	class StreamingTurnStop(TurnAnalyzerUserTurnStopStrategy):
	"""Tells the model when the bot is speaking: it was trained with that input."""

	async def process_frame(self, frame):
	if isinstance(frame, (BotStartedSpeakingFrame, BotStoppedSpeakingFrame)):
	analyzer.set_bot_speaking(isinstance(frame, BotStartedSpeakingFrame))
	return await super().process_frame(frame)


	user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
	LLMContext(),
	user_params=LLMUserAggregatorParams(
	vad_analyzer=SileroVADAnalyzer(),
	user_turn_strategies=UserTurnStrategies(stop=[StreamingTurnStop(turn_analyzer=analyzer)]),
	),
	)
	```

	## Outputs

	One frame per 40 ms chunk.

	\| Output \| \|
	\|---\|---\|
	\| `eot` \| probability that the user's turn is over \|
	\| `eot_late` \| the same, for a user who has gone quiet (v0.4 only) \|
	\| `intent` \| continuer, assessment, negative, floor_take, non_speech. Backchannel = continuer + assessment; floor-take = negative + floor_take \|
	\| `leadin` \| none, acknowledgement, hesitation, marker, reaction (barely beats guessing) \|
	\| `vad`, `vap` \| auxiliary: voice activity now, and both speakers' activity over the next 2 s \|

	Inputs: `chunk` (640 samples, float32) and `bot_state` (is the bot speaking, and for how long), plus the streaming state tensors named in `model.json`.

	## Results

	Held-out Open Yap 1K conversations (1,259 turn ends, 8,191 pauses), replayed through Pipecat's Silero VAD with nobody answering, at the settings above:

	\| \| Cuts in on a pause \| Answers within 500 ms \| Within 1 s \| Waits for the 3 s timeout \|
	\|---\|---\|---\|---\|---\|
	\| v0.4 \| 4.9% \| 56.4% \| 79.7% \| 2.9% \|
	\| v0.2 \| 4.6% \| 51.4% \| 82.2% \| 15.5% \|

	On Smart Turn v3.2's English test clips (7,820, read 200 ms after the speech ends): v0.4 93.3%, v0.2 93.8%, Smart Turn v3.2 93.5%.

	## Limitations

	- The held-out conversations are from a corpus the model trained on (different conversations).
	- Keep `bot_state` up to date. v0.4 holds back until the bot has spoken in a session.
	- A finished sentence it takes for an unfinished one waits for the 3 s timeout.
	- Half of the end-of-turn targets on Smart Turn clips are Smart Turn v3.2's own predictions, so it is not independent of v3.2.
	- Trained on human-to-human conversation and single-speaker clips, with simulated echo. Phone codecs are untested.

	## Training data

	[Open Yap 1K](https://theagenticdatacompany.com/open-yap-1k) v1.0, [otoSpeech-full-duplex-280h](https://huggingface.co/datasets/otoearth/otoSpeech-full-duplex-280h) (CC BY 4.0), the English part of the [Smart Turn v3.2 training data](https://huggingface.co/datasets/pipecat-ai/smart-turn-data-v3.2-train) (CC BY 4.0), and 600 synthetic hard negatives. A frozen [Whisper base.en](https://huggingface.co/openai/whisper-base.en) encoder was the distillation target, and Smart Turn v3.2 gave soft end-of-turn labels. No training data is included here.