jptaylor/streaming-turn

Streaming Turn

Streaming, audio-only turn-taking model for Pipecat voice agents. On every 40 ms of user audio it gives the probability that the user's turn is over, and whether speech over the bot is a backchannel or a floor-take. It never sees a transcript.

• 4.0M parameters: log-mel, causal conv encoder, GRU. 16 kHz mono in, 25 frames a second out.
• 0.37 ms per 40 ms chunk on one Apple M-series core (ONNX Runtime).
• English only. Experimental: one training run per version, and little live testing.

Versions

Folder	End the turn when
streaming-turn-v0.4 (latest)	eot ≥ 0.5, or eot_late ≥ 0.4
streaming-turn-v0.2	eot ≥ 0.5, or 0.8 s after the last sound if eot reached 0.3

v0.4 adds eot_late, a second end-of-turn output for a user who has gone quiet.

Each folder holds streaming-turn-fp32.onnx (the streaming model), model.json (tensor names, chunk size, class lists) and a .pt file (PyTorch config and weights).

Use with Pipecat

agent.turn_analyzers is in the code repository, jptaylor/streaming-turn.

from pathlib import Path

from huggingface_hub import snapshot_download
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.frames.frames import BotStartedSpeakingFrame, BotStoppedSpeakingFrame
from pipecat.processors.aggregators.llm_response_universal import (
    LLMContext,
    LLMContextAggregatorPair,
    LLMUserAggregatorParams,
)
from pipecat.turns.user_stop import TurnAnalyzerUserTurnStopStrategy
from pipecat.turns.user_turn_strategies import UserTurnStrategies

from agent.turn_analyzers import StreamingTurnAnalyzer, StreamingTurnParams

VERSION = "streaming-turn-v0.4"
repo = snapshot_download("jptaylor/streaming-turn", allow_patterns=f"{VERSION}/*")
analyzer = StreamingTurnAnalyzer(
    Path(repo) / VERSION,
    params=StreamingTurnParams(threshold=0.5, late_threshold=0.4),
    # v0.2: StreamingTurnParams(threshold=0.5, likely_threshold=0.3)
)


class StreamingTurnStop(TurnAnalyzerUserTurnStopStrategy):
    """Tells the model when the bot is speaking: it was trained with that input."""

    async def process_frame(self, frame):
        if isinstance(frame, (BotStartedSpeakingFrame, BotStoppedSpeakingFrame)):
            analyzer.set_bot_speaking(isinstance(frame, BotStartedSpeakingFrame))
        return await super().process_frame(frame)


user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    LLMContext(),
    user_params=LLMUserAggregatorParams(
        vad_analyzer=SileroVADAnalyzer(),
        user_turn_strategies=UserTurnStrategies(stop=[StreamingTurnStop(turn_analyzer=analyzer)]),
    ),
)

Outputs

One frame per 40 ms chunk.

Output
eot	probability that the user's turn is over
eot_late	the same, for a user who has gone quiet (v0.4 only)
intent	continuer, assessment, negative, floor_take, non_speech. Backchannel = continuer + assessment; floor-take = negative + floor_take
leadin	none, acknowledgement, hesitation, marker, reaction (barely beats guessing)
vad, vap	auxiliary: voice activity now, and both speakers' activity over the next 2 s

Inputs: chunk (640 samples, float32) and bot_state (is the bot speaking, and for how long), plus the streaming state tensors named in model.json.

Results

Held-out Open Yap 1K conversations (1,259 turn ends, 8,191 pauses), replayed through Pipecat's Silero VAD with nobody answering, at the settings above:

	Cuts in on a pause	Answers within 500 ms	Within 1 s	Waits for the 3 s timeout
v0.4	4.9%	56.4%	79.7%	2.9%
v0.2	4.6%	51.4%	82.2%	15.5%

On Smart Turn v3.2's English test clips (7,820, read 200 ms after the speech ends): v0.4 93.3%, v0.2 93.8%, Smart Turn v3.2 93.5%.

Limitations

• The held-out conversations are from a corpus the model trained on (different conversations).
• Keep bot_state up to date. v0.4 holds back until the bot has spoken in a session.
• A finished sentence it takes for an unfinished one waits for the 3 s timeout.
• Half of the end-of-turn targets on Smart Turn clips are Smart Turn v3.2's own predictions, so it is not independent of v3.2.
• Trained on human-to-human conversation and single-speaker clips, with simulated echo. Phone codecs are untested.

Training data

Open Yap 1K v1.0, otoSpeech-full-duplex-280h (CC BY 4.0), the English part of the Smart Turn v3.2 training data (CC BY 4.0), and 600 synthetic hard negatives. A frozen Whisper base.en encoder was the distillation target, and Smart Turn v3.2 gave soft end-of-turn labels. No training data is included here.