File size: 6,359 Bytes
c9b59be
 
 
 
 
08c9b8c
 
 
c9b59be
1a96c3c
c9b59be
 
 
 
 
 
 
 
 
0a0cbce
c9b59be
08c9b8c
c9b59be
1a96c3c
 
 
c9b59be
1a96c3c
 
 
 
 
 
 
 
 
 
 
 
 
 
c9b59be
1a96c3c
08c9b8c
 
 
1a96c3c
 
08c9b8c
1a96c3c
08c9b8c
1a96c3c
05704f4
1a96c3c
0a0cbce
1a96c3c
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
c9b59be
 
 
 
 
 
 
 
 
 
1a96c3c
 
 
 
c9b59be
 
 
 
1a96c3c
 
 
 
0a0cbce
 
c9b59be
 
 
 
 
 
1a96c3c
 
c9b59be
1a96c3c
 
c9b59be
1a96c3c
c9b59be
1a96c3c
c9b59be
1a96c3c
 
 
 
c9b59be
1a96c3c
 
 
 
 
 
 
 
 
 
 
 
 
 
 
08c9b8c
1a96c3c
 
 
 
 
 
 
 
 
c9b59be
 
 
1a96c3c
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
---
license: bsd-2-clause
language: en
library_name: onnx
pipeline_tag: text-classification
base_model:
  - jhu-clsp/ettin-encoder-150m
  - jhu-clsp/ettin-encoder-32m
datasets:
  - pipecat-ai/sidechat-gen
  - ACloudCenter/ami-addressee
tags:
  - pipecat
  - onnx
  - addressee-detection
  - multi-party
  - turn-taking
---

# Pipecat Sidechat (experimental, rc0.3.1)

> **⚠️ Experimental.** Outputs, names and calibration may change between releases.

A small text classifier for a voice agent in a conversation with several people. For each line it says whether
the agent should speak now, wait to see if anyone else does, or stay quiet. It runs on the CPU, and speakers can
be anonymous ("Speaker 1").

| | Default | `agent-1x` |
|---|---|---|
| Requests that name the agent, answered | 99.6% | 99.7% |
| People's lines answered by mistake | 0.2% | 0.3% |
| People's lines held for a pause | 0.8% | 0.7% |
| Real spoken requests without the name, held for a pause | 91% | 87% |
| Time per line, int8, on 1 vCPU | 303 ms | 7 ms |
| Download, int8 | 325 MB | 195 MB |

The first three rows are on [Duplex-MPE](https://huggingface.co/datasets/ChengqianMa/Duplex-MPE), 52,336 lines.
The fourth is on recorded requests placed in real meetings, as Deepgram heard them.

Code: [github.com/pipecat-ai/sidechat](https://github.com/pipecat-ai/sidechat).
Training data: [pipecat-ai/sidechat-gen](https://huggingface.co/datasets/pipecat-ai/sidechat-gen).

## Builds

| Build | Folder | For |
|---|---|---|
| Default, 150M | repo root | 1 vCPU or more; the more accurate one |
| `agent-1x`, 32M two-tower | `agent-1x/` | 0.5 vCPU, 1 GiB; encodes the earlier lines while the person talks |

Each has fp32 and int8 files. int8 decides the same as fp32 on 98.5% to 99.9% of test lines.

## Use

### In Pipecat

`PipecatSidechat` goes between the user aggregator and the LLM, and lets the LLM answer only the turns meant
for the bot:

```python
from pipecat.turns.sidechat import PipecatSidechat

sidechat = PipecatSidechat(agent_name="Atlas")

pipeline = Pipeline([
    transport.input(), stt, user_aggregator,
    sidechat,
    llm, tts, transport.output(), assistant_aggregator,
])
```

It is in [pipecat-ai/pipecat#6934](https://github.com/pipecat-ai/pipecat/pull/6934) until that is released,
with a full bot in `examples/turn-management/turn-management-sidechat.py`.

### On its own

```bash
pip install git+https://github.com/pipecat-ai/sidechat
```

```python
from huggingface_hub import snapshot_download
from sidechat.runtime import load
from sidechat.schema import Agent, SidechatRequest, Turn

path = snapshot_download("pipecat-ai/sidechat", allow_patterns=["agent-1x/*.json", "agent-1x/*-int8.onnx"])
model = load(f"{path}/agent-1x", threads=1, int8=True)

request = SidechatRequest(
    agent=Agent(name="Atlas"),
    speaker="Speaker 1",
    text="Can you check that for us?",
    history=[Turn(speaker="Atlas", text="Want me to look up flights?", agent=True)],
)
model.prepare(request)  # agent-1x only: encode the earlier lines while the person talks
answer = model.answer(request)
print(answer.respond, model.decide(answer))  # 0.97 respond
```

| Output | Meaning |
|---|---|
| `respond` | probability the agent should speak now; about 0.5 for a question left open to the room |
| `intent` | for the agent directly or indirectly, open to the room, about the agent, or other |
| `unclear` | the line fits the agent and someone else equally |

`decide` gives `respond`, `wait` (speak only if nobody else does) or `ignore`, at the thresholds in
`config.json`. For the default build, load the repo root with `model.onnx` or `model-int8.onnx`.

Give your speech-to-text service the agent's name as a keyterm (Deepgram's `keyterm`). An unusual name is often
misheard otherwise, and a request that names the agent is then missed.

## Strengths and weaknesses

**Strengths**

- **It stays quiet through people's talk.** It answers 0.2% of lines meant for others and holds under 1%.
- **It answers when called.** Requests that name the agent are answered 99.6% of the time.
- **It holds real requests that don't name it** (91%), and answers if nobody else does.
- **It is fast and local.** `agent-1x` takes about 10 ms a line on half a CPU, with no API call.

**Weaknesses**

- **Requests that only the context marks as for the agent** are often missed: 29% answered by the default
  build and 10% by `agent-1x`.
- **A follow-up without the name, right after the agent answers,** is hard to tell from talk to another
  person.
- **About 1 in 100 of people's lines in real meetings is held for a pause,** mostly questions people ask each
  other without a name.
- **English text only.**

## Compared with Jev

Jev is a hosted classifier, asked as Pipecat's daily-audience demo asks it. Speakers are anonymous.

| | Default | `agent-1x` | Jev |
|---|---|---|---|
| Duplex-MPE, decisions right (52,336 lines) | 96.4% | 95.7% | 83.1% |
| [Social AI Ambient Bench](https://huggingface.co/datasets/text-ai/social-ai-ambient-bench), right (15,793) | 96.9% | 97.5% | 85.4% |
| [AMI](https://huggingface.co/datasets/ACloudCenter/ami-addressee) meetings, right (8,875) | 99.0% | 98.7% | 85.2% |
| Duplex-MPE, people's lines held for a pause | 0.8% | 0.7% | 16.0% |
| Duplex-MPE, requests only the context marks, answered | 29.3% | 10.2% | 26.6% |
| Real spoken requests without the name, held (2,705) | 90.9% | 86.6% | 82.1% |
| Time per line on 1 vCPU | 303 ms | 7 ms | 49 to 149 ms |

Labels follow the project's own guide, so read these as comparisons, not benchmarks.

## Training

Both builds use [Ettin](https://huggingface.co/jhu-clsp/ettin-encoder-150m) encoders (MIT) with intent, respond
and unclear heads. `agent-1x` is distilled from the default build.

- **Our data:** generated multi-party conversations, labelled by kind under the project's guide. All of it is
  in [pipecat-ai/sidechat-gen](https://huggingface.co/datasets/pipecat-ai/sidechat-gen).
- **Public data:** [AMI](https://huggingface.co/datasets/ACloudCenter/ami-addressee) meetings (CC BY 4.0), and
  5,000 requests to assistants from [SLURP](https://github.com/pswietojanski/slurp)'s text (CC BY 4.0) and
  [CLINC150](https://github.com/clinc/oos-eval) (CC BY 3.0), each placed in an AMI meeting.

Duplex-MPE, the Social AI Ambient Bench and SLURP's recordings were used for evaluation only.