jptaylor commited on
Commit
0a0cbce
·
verified ·
1 Parent(s): dafacac

side-r9-ettin-150m-s1: ONNX export, model card and training rows

Browse files
Files changed (6) hide show
  1. MANIFEST.json +10 -6
  2. README.md +32 -11
  3. config.json +17 -1
  4. data/dev.jsonl.gz +1 -1
  5. data/train.jsonl.gz +1 -1
  6. model-int8.onnx +3 -0
MANIFEST.json CHANGED
@@ -2,20 +2,24 @@
2
  "run_id": "side-r9-ettin-150m-s1",
3
  "files": {
4
  "README.md": {
5
- "bytes": 8295,
6
- "sha256": "a7ba36f8328ce9b0273005467ac68a8b422a25bf8c646a32fc857bb3596f1e60"
7
  },
8
  "config.json": {
9
- "bytes": 615,
10
- "sha256": "a59e4802438d7baf5191af6c6eb40a1179e8321b9d7e8b41f2b8419a6e606ebc"
11
  },
12
  "data/dev.jsonl.gz": {
13
  "bytes": 1937924,
14
- "sha256": "848508009d00e201eda3b7f27ecf6bd373d9526b2db90e2339a6249b95d23b2f"
15
  },
16
  "data/train.jsonl.gz": {
17
  "bytes": 18648841,
18
- "sha256": "72a9655886a78f5e4bbebab565238031f0006e9412a8237100a72ed6a1f276b7"
 
 
 
 
19
  },
20
  "model.onnx": {
21
  "bytes": 598295149,
 
2
  "run_id": "side-r9-ettin-150m-s1",
3
  "files": {
4
  "README.md": {
5
+ "bytes": 9704,
6
+ "sha256": "d962f78c592ee75579e22a4824a1a1df523b7a2e046a79c097eeab58e426dcf5"
7
  },
8
  "config.json": {
9
+ "bytes": 1027,
10
+ "sha256": "f12b2850bda3876d089d6376ca0998a3fa82d83704bd9bfc3f8125849ca1c23b"
11
  },
12
  "data/dev.jsonl.gz": {
13
  "bytes": 1937924,
14
+ "sha256": "d9db0aa09f9756f8aa26c888ec62caa5253391552c3efc4d4471b37a8a5db872"
15
  },
16
  "data/train.jsonl.gz": {
17
  "bytes": 18648841,
18
+ "sha256": "bb8c77d10fe57456d9146ff81b0adff887527a80d19812555dbd581929b8f0dd"
19
+ },
20
+ "model-int8.onnx": {
21
+ "bytes": 325445917,
22
+ "sha256": "91920b9ed982c5d2a7e1302d95c1f9f651fa53af36e5a1cac56290064e895929"
23
  },
24
  "model.onnx": {
25
  "bytes": 598295149,
README.md CHANGED
@@ -16,7 +16,7 @@ tags:
16
  - turn-taking
17
  ---
18
 
19
- # Pipecat Sidechat (experimental, rc0.3)
20
 
21
  > **⚠️ Experimental.** Outputs, names and calibration may change between releases.
22
 
@@ -41,6 +41,11 @@ A request without the agent's name ("set a timer for ten minutes"), said before
41
  for a pause about nine times in ten. In rc0.2 it was mostly ignored. The cost is more holds on people's own talk;
42
  see Limitations. rc0.2 stays at tag `rc0.2`.
43
 
 
 
 
 
 
44
  ## Use
45
 
46
  ```bash
@@ -62,12 +67,18 @@ a = model.answer(SidechatRequest(
62
  print(a.respond, model.decide(a)) # 0.97 respond
63
  ```
64
 
 
 
 
 
 
 
65
  For `agent-1x`, load the folder and call `prepare` with the next line's request (its history is what counts)
66
  as soon as the previous line ends; `answer` then encodes only the new line:
67
 
68
  ```python
69
  path = snapshot_download("pipecat-ai/sidechat", allow_patterns=["agent-1x/*.json", "agent-1x/*.onnx"])
70
- model = load(f"{path}/agent-1x", threads=1)
71
  model.prepare(request) # while the person talks
72
  a = model.answer(request) # when they stop
73
  ```
@@ -87,9 +98,11 @@ misheard otherwise ("Olly" as "Oli"), and a request that names the agent is then
87
 
88
  | File | |
89
  |---|---|
90
- | `model.onnx` | default build, fp32, 598 MB (rc0.3 has no int8 build: it failed its quality check) |
 
91
  | `agent-1x/context.onnx`, `agent-1x/line.onnx` | `agent-1x` build, fp32, 128 + 131 MB |
92
- | `tokenizer.json`, `config.json` (each folder) | tokenizer, calibration and thresholds |
 
93
  | `data/`, `agent-1x/data/` | the rows each build was trained on (train and dev) |
94
  | `MANIFEST.json` (each folder) | sha256 of every file, row counts |
95
 
@@ -119,18 +132,23 @@ trained on.
119
 
120
  ## Speed
121
 
122
- Median ms per line with 8 earlier lines, one thread, on containers capped like Pipecat Cloud's profiles. Jev is
123
- an API call from the same container, so its time is mostly the network.
 
124
 
125
  | | 0.5 vCPU, 1 GiB | 1 vCPU, 2 GiB | Memory |
126
  |---|---|---|---|
127
- | `agent-1x`, earlier lines encoded ahead | 14 | 14 | 487 MB |
128
- | `agent-1x`, earlier lines not ready | 108 | 68 | |
129
- | Default, fp32 | does not fit well (648, tail to 21 s) | 253 | about 0.9 GB |
 
 
 
130
  | Jev | 49 to 163 | 49 to 149 | |
131
 
132
- Encoding the earlier lines takes about 97 ms on `agent-1x`, well inside the time anyone takes to say a line. The
133
- default build's times were measured on rc0.2's default build, which has the same size and graph. On an M5 Max it
 
134
  takes 10 ms with a short history and 52 ms with 8 earlier lines.
135
 
136
  ## Limitations
@@ -157,6 +175,9 @@ unclear heads.
157
  - **`agent-1x`:** Ettin-32M as one shared encoder for the earlier lines and the new line, joined by a
158
  cross-attention layer. 3 epochs on 425,828 lines, with the default build's probabilities as 80% of each
159
  target.
 
 
 
160
 
161
  The lines come from three sources:
162
 
 
16
  - turn-taking
17
  ---
18
 
19
+ # Pipecat Sidechat (experimental, rc0.3.1)
20
 
21
  > **⚠️ Experimental.** Outputs, names and calibration may change between releases.
22
 
 
41
  for a pause about nine times in ten. In rc0.2 it was mostly ignored. The cost is more holds on people's own talk;
42
  see Limitations. rc0.2 stays at tag `rc0.2`.
43
 
44
+ **New in rc0.3.1:** int8 files for both builds, opt in with `int8=True`. The fp32 files are rc0.3's, unchanged,
45
+ and stay the default; rc0.3 stays at tag `rc0.3`. On every test set below, int8 at its own thresholds makes the
46
+ same decision as fp32 on 98.5% to 99.9% of lines and is within a point of it. It is smaller and faster; see Files
47
+ and Speed.
48
+
49
  ## Use
50
 
51
  ```bash
 
67
  print(a.respond, model.decide(a)) # 0.97 respond
68
  ```
69
 
70
+ For int8, download its file and pass `int8=True`; the thresholds in `config.json`'s `thresholds_int8` are used:
71
+
72
+ ```python
73
+ model = load(snapshot_download("pipecat-ai/sidechat", allow_patterns=["*.json", "model-int8.onnx"]), int8=True)
74
+ ```
75
+
76
  For `agent-1x`, load the folder and call `prepare` with the next line's request (its history is what counts)
77
  as soon as the previous line ends; `answer` then encodes only the new line:
78
 
79
  ```python
80
  path = snapshot_download("pipecat-ai/sidechat", allow_patterns=["agent-1x/*.json", "agent-1x/*.onnx"])
81
+ model = load(f"{path}/agent-1x", threads=1) # or int8=True, with "agent-1x/*-int8.onnx"
82
  model.prepare(request) # while the person talks
83
  a = model.answer(request) # when they stop
84
  ```
 
98
 
99
  | File | |
100
  |---|---|
101
+ | `model.onnx` | default build, fp32, 598 MB |
102
+ | `model-int8.onnx` | default build, int8, 325 MB |
103
  | `agent-1x/context.onnx`, `agent-1x/line.onnx` | `agent-1x` build, fp32, 128 + 131 MB |
104
+ | `agent-1x/context-int8.onnx`, `agent-1x/line-int8.onnx` | `agent-1x` build, int8, 97 + 98 MB |
105
+ | `tokenizer.json`, `config.json` (each folder) | tokenizer, calibration, and thresholds for fp32 and int8 |
106
  | `data/`, `agent-1x/data/` | the rows each build was trained on (train and dev) |
107
  | `MANIFEST.json` (each folder) | sha256 of every file, row counts |
108
 
 
132
 
133
  ## Speed
134
 
135
+ Median ms per line with 8 earlier lines, one thread, on containers capped like Pipecat Cloud's profiles, fp32 and
136
+ int8 in the same run. Jev is an API call from the same container, so its time is mostly the network; its row is
137
+ from earlier runs.
138
 
139
  | | 0.5 vCPU, 1 GiB | 1 vCPU, 2 GiB | Memory |
140
  |---|---|---|---|
141
+ | `agent-1x` int8, earlier lines encoded ahead | 10 | 7 | 428 MB |
142
+ | `agent-1x` fp32, earlier lines encoded ahead | 11 | 16 | 487 MB |
143
+ | `agent-1x` int8, earlier lines not ready | 95 | 32 | |
144
+ | `agent-1x` fp32, earlier lines not ready | 104 | 77 | |
145
+ | Default, int8 | 672 | 303 | 628 MB |
146
+ | Default, fp32 | 991 (near the memory limit; it took 3 minutes to load) | 465 | about 0.9 GB |
147
  | Jev | 49 to 163 | 49 to 149 | |
148
 
149
+ Times move with the host's CPU between runs: an earlier run on another host measured the default fp32 build at
150
+ 253 ms on 1 vCPU. Compare rows within the table. Encoding the earlier lines on `agent-1x` takes 86 ms in int8 and
151
+ 93 ms in fp32 at 0.5 vCPU, well inside the time anyone takes to say a line. On an M5 Max the default fp32 build
152
  takes 10 ms with a short history and 52 ms with 8 earlier lines.
153
 
154
  ## Limitations
 
175
  - **`agent-1x`:** Ettin-32M as one shared encoder for the earlier lines and the new line, joined by a
176
  cross-attention layer. 3 epochs on 425,828 lines, with the default build's probabilities as 80% of each
177
  target.
178
+ - **int8 (rc0.3.1):** dynamic int8 weights, per channel with reduced range, with each layer's MLP output kept in
179
+ fp32. Without reduced range, int8 lost about a third of its decisions on CPUs without VNNI. Each build's int8
180
+ thresholds were chosen on dev, where it is within 0.1 points of fp32.
181
 
182
  The lines come from three sources:
183
 
config.json CHANGED
@@ -27,5 +27,21 @@
27
  "max_wrong_now_real": 0.004,
28
  "max_held_real": 0.002
29
  },
30
- "exported_at": "2026-11-20T12:29:54+00:00"
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
31
  }
 
27
  "max_wrong_now_real": 0.004,
28
  "max_held_real": 0.002
29
  },
30
+ "exported_at": "2026-11-20T12:29:54+00:00",
31
+ "thresholds_int8": {
32
+ "respond": 0.6,
33
+ "wait": 0.45,
34
+ "open": 0.5,
35
+ "dev_right": 21611,
36
+ "dev_n": 21841,
37
+ "dev_wrong_now": 24,
38
+ "dev_quiet": 13074,
39
+ "max_wrong_now": 0.01,
40
+ "real_wrong_now": 0,
41
+ "real_held": 5,
42
+ "real_quiet": 4186,
43
+ "max_wrong_now_real": 0.004,
44
+ "max_held_real": 0.002
45
+ },
46
+ "int8_recipe": "per channel, reduce_range, MatMul with constant B only, each layer's mlp.Wo left in fp32"
47
  }
data/dev.jsonl.gz CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:848508009d00e201eda3b7f27ecf6bd373d9526b2db90e2339a6249b95d23b2f
3
  size 1937924
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d9db0aa09f9756f8aa26c888ec62caa5253391552c3efc4d4471b37a8a5db872
3
  size 1937924
data/train.jsonl.gz CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:72a9655886a78f5e4bbebab565238031f0006e9412a8237100a72ed6a1f276b7
3
  size 18648841
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:bb8c77d10fe57456d9146ff81b0adff887527a80d19812555dbd581929b8f0dd
3
  size 18648841
model-int8.onnx ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:91920b9ed982c5d2a7e1302d95c1f9f651fa53af36e5a1cac56290064e895929
3
+ size 325445917