Pipecat Sidechat (experimental, rc0.3.1)
โ ๏ธ Experimental. Outputs, names and calibration may change between releases.
A small text classifier for a voice agent in a conversation with several people. For each line it says whether the agent should speak now, wait to see if anyone else does, or stay quiet. It runs on the CPU, and speakers can be anonymous ("Speaker 1").
| Default | agent-1x |
|
|---|---|---|
| Requests that name the agent, answered | 99.6% | 99.7% |
| People's lines answered by mistake | 0.2% | 0.3% |
| People's lines held for a pause | 0.8% | 0.7% |
| Real spoken requests without the name, held for a pause | 91% | 87% |
| Time per line, int8, on 1 vCPU | 303 ms | 7 ms |
| Download, int8 | 325 MB | 195 MB |
The first three rows are on Duplex-MPE, 52,336 lines. The fourth is on recorded requests placed in real meetings, as Deepgram heard them.
Code: github.com/pipecat-ai/sidechat. Training data: pipecat-ai/sidechat-gen.
Builds
| Build | Folder | For |
|---|---|---|
| Default, 150M | repo root | 1 vCPU or more; the more accurate one |
agent-1x, 32M two-tower |
agent-1x/ |
0.5 vCPU, 1 GiB; encodes the earlier lines while the person talks |
Each has fp32 and int8 files. int8 decides the same as fp32 on 98.5% to 99.9% of test lines.
Use
In Pipecat
PipecatSidechat goes between the user aggregator and the LLM, and lets the LLM answer only the turns meant
for the bot:
from pipecat.turns.sidechat import PipecatSidechat
sidechat = PipecatSidechat(agent_name="Atlas")
pipeline = Pipeline([
transport.input(), stt, user_aggregator,
sidechat,
llm, tts, transport.output(), assistant_aggregator,
])
It is in pipecat-ai/pipecat#6934 until that is released,
with a full bot in examples/turn-management/turn-management-sidechat.py.
On its own
pip install git+https://github.com/pipecat-ai/sidechat
from huggingface_hub import snapshot_download
from sidechat.runtime import load
from sidechat.schema import Agent, SidechatRequest, Turn
path = snapshot_download("pipecat-ai/sidechat", allow_patterns=["agent-1x/*.json", "agent-1x/*-int8.onnx"])
model = load(f"{path}/agent-1x", threads=1, int8=True)
request = SidechatRequest(
agent=Agent(name="Atlas"),
speaker="Speaker 1",
text="Can you check that for us?",
history=[Turn(speaker="Atlas", text="Want me to look up flights?", agent=True)],
)
model.prepare(request) # agent-1x only: encode the earlier lines while the person talks
answer = model.answer(request)
print(answer.respond, model.decide(answer)) # 0.97 respond
| Output | Meaning |
|---|---|
respond |
probability the agent should speak now; about 0.5 for a question left open to the room |
intent |
for the agent directly or indirectly, open to the room, about the agent, or other |
unclear |
the line fits the agent and someone else equally |
decide gives respond, wait (speak only if nobody else does) or ignore, at the thresholds in
config.json. For the default build, load the repo root with model.onnx or model-int8.onnx.
Give your speech-to-text service the agent's name as a keyterm (Deepgram's keyterm). An unusual name is often
misheard otherwise, and a request that names the agent is then missed.
Strengths and weaknesses
Strengths
- It stays quiet through people's talk. It answers 0.2% of lines meant for others and holds under 1%.
- It answers when called. Requests that name the agent are answered 99.6% of the time.
- It holds real requests that don't name it (91%), and answers if nobody else does.
- It is fast and local.
agent-1xtakes about 10 ms a line on half a CPU, with no API call.
Weaknesses
- Requests that only the context marks as for the agent are often missed: 29% answered by the default
build and 10% by
agent-1x. - A follow-up without the name, right after the agent answers, is hard to tell from talk to another person.
- About 1 in 100 of people's lines in real meetings is held for a pause, mostly questions people ask each other without a name.
- English text only.
Compared with Jev
Jev is a hosted classifier, asked as Pipecat's daily-audience demo asks it. Speakers are anonymous.
| Default | agent-1x |
Jev | |
|---|---|---|---|
| Duplex-MPE, decisions right (52,336 lines) | 96.4% | 95.7% | 83.1% |
| Social AI Ambient Bench, right (15,793) | 96.9% | 97.5% | 85.4% |
| AMI meetings, right (8,875) | 99.0% | 98.7% | 85.2% |
| Duplex-MPE, people's lines held for a pause | 0.8% | 0.7% | 16.0% |
| Duplex-MPE, requests only the context marks, answered | 29.3% | 10.2% | 26.6% |
| Real spoken requests without the name, held (2,705) | 90.9% | 86.6% | 82.1% |
| Time per line on 1 vCPU | 303 ms | 7 ms | 49 to 149 ms |
Labels follow the project's own guide, so read these as comparisons, not benchmarks.
Training
Both builds use Ettin encoders (MIT) with intent, respond
and unclear heads. agent-1x is distilled from the default build.
- Our data: generated multi-party conversations, labelled by kind under the project's guide. All of it is in pipecat-ai/sidechat-gen.
- Public data: AMI meetings (CC BY 4.0), and 5,000 requests to assistants from SLURP's text (CC BY 4.0) and CLINC150 (CC BY 3.0), each placed in an AMI meeting.
Duplex-MPE, the Social AI Ambient Bench and SLURP's recordings were used for evaluation only.
- Downloads last month
- 315
Model tree for pipecat-ai/sidechat
Base model
jhu-clsp/ettin-encoder-150m