sidechat / README.md
jptaylor's picture
Model card; training data moves to pipecat-ai/sidechat-gen
1a96c3c verified
|
Raw History Blame Contribute Delete
6.36 kB
metadata
license: bsd-2-clause
language: en
library_name: onnx
pipeline_tag: text-classification
base_model:
  - jhu-clsp/ettin-encoder-150m
  - jhu-clsp/ettin-encoder-32m
datasets:
  - pipecat-ai/sidechat-gen
  - ACloudCenter/ami-addressee
tags:
  - pipecat
  - onnx
  - addressee-detection
  - multi-party
  - turn-taking

Pipecat Sidechat (experimental, rc0.3.1)

⚠️ Experimental. Outputs, names and calibration may change between releases.

A small text classifier for a voice agent in a conversation with several people. For each line it says whether the agent should speak now, wait to see if anyone else does, or stay quiet. It runs on the CPU, and speakers can be anonymous ("Speaker 1").

Default agent-1x
Requests that name the agent, answered 99.6% 99.7%
People's lines answered by mistake 0.2% 0.3%
People's lines held for a pause 0.8% 0.7%
Real spoken requests without the name, held for a pause 91% 87%
Time per line, int8, on 1 vCPU 303 ms 7 ms
Download, int8 325 MB 195 MB

The first three rows are on Duplex-MPE, 52,336 lines. The fourth is on recorded requests placed in real meetings, as Deepgram heard them.

Code: github.com/pipecat-ai/sidechat. Training data: pipecat-ai/sidechat-gen.

Builds

Build Folder For
Default, 150M repo root 1 vCPU or more; the more accurate one
agent-1x, 32M two-tower agent-1x/ 0.5 vCPU, 1 GiB; encodes the earlier lines while the person talks

Each has fp32 and int8 files. int8 decides the same as fp32 on 98.5% to 99.9% of test lines.

Use

In Pipecat

PipecatSidechat goes between the user aggregator and the LLM, and lets the LLM answer only the turns meant for the bot:

from pipecat.turns.sidechat import PipecatSidechat

sidechat = PipecatSidechat(agent_name="Atlas")

pipeline = Pipeline([
    transport.input(), stt, user_aggregator,
    sidechat,
    llm, tts, transport.output(), assistant_aggregator,
])

It is in pipecat-ai/pipecat#6934 until that is released, with a full bot in examples/turn-management/turn-management-sidechat.py.

On its own

pip install git+https://github.com/pipecat-ai/sidechat
from huggingface_hub import snapshot_download
from sidechat.runtime import load
from sidechat.schema import Agent, SidechatRequest, Turn

path = snapshot_download("pipecat-ai/sidechat", allow_patterns=["agent-1x/*.json", "agent-1x/*-int8.onnx"])
model = load(f"{path}/agent-1x", threads=1, int8=True)

request = SidechatRequest(
    agent=Agent(name="Atlas"),
    speaker="Speaker 1",
    text="Can you check that for us?",
    history=[Turn(speaker="Atlas", text="Want me to look up flights?", agent=True)],
)
model.prepare(request)  # agent-1x only: encode the earlier lines while the person talks
answer = model.answer(request)
print(answer.respond, model.decide(answer))  # 0.97 respond
Output Meaning
respond probability the agent should speak now; about 0.5 for a question left open to the room
intent for the agent directly or indirectly, open to the room, about the agent, or other
unclear the line fits the agent and someone else equally

decide gives respond, wait (speak only if nobody else does) or ignore, at the thresholds in config.json. For the default build, load the repo root with model.onnx or model-int8.onnx.

Give your speech-to-text service the agent's name as a keyterm (Deepgram's keyterm). An unusual name is often misheard otherwise, and a request that names the agent is then missed.

Strengths and weaknesses

Strengths

  • It stays quiet through people's talk. It answers 0.2% of lines meant for others and holds under 1%.
  • It answers when called. Requests that name the agent are answered 99.6% of the time.
  • It holds real requests that don't name it (91%), and answers if nobody else does.
  • It is fast and local. agent-1x takes about 10 ms a line on half a CPU, with no API call.

Weaknesses

  • Requests that only the context marks as for the agent are often missed: 29% answered by the default build and 10% by agent-1x.
  • A follow-up without the name, right after the agent answers, is hard to tell from talk to another person.
  • About 1 in 100 of people's lines in real meetings is held for a pause, mostly questions people ask each other without a name.
  • English text only.

Compared with Jev

Jev is a hosted classifier, asked as Pipecat's daily-audience demo asks it. Speakers are anonymous.

Default agent-1x Jev
Duplex-MPE, decisions right (52,336 lines) 96.4% 95.7% 83.1%
Social AI Ambient Bench, right (15,793) 96.9% 97.5% 85.4%
AMI meetings, right (8,875) 99.0% 98.7% 85.2%
Duplex-MPE, people's lines held for a pause 0.8% 0.7% 16.0%
Duplex-MPE, requests only the context marks, answered 29.3% 10.2% 26.6%
Real spoken requests without the name, held (2,705) 90.9% 86.6% 82.1%
Time per line on 1 vCPU 303 ms 7 ms 49 to 149 ms

Labels follow the project's own guide, so read these as comparisons, not benchmarks.

Training

Both builds use Ettin encoders (MIT) with intent, respond and unclear heads. agent-1x is distilled from the default build.

  • Our data: generated multi-party conversations, labelled by kind under the project's guide. All of it is in pipecat-ai/sidechat-gen.
  • Public data: AMI meetings (CC BY 4.0), and 5,000 requests to assistants from SLURP's text (CC BY 4.0) and CLINC150 (CC BY 3.0), each placed in an AMI meeting.

Duplex-MPE, the Social AI Ambient Bench and SLURP's recordings were used for evaluation only.