side-r9-ettin-150m-s1: ONNX export, model card and training rows
Browse files- MANIFEST.json +10 -6
- README.md +32 -11
- config.json +17 -1
- data/dev.jsonl.gz +1 -1
- data/train.jsonl.gz +1 -1
- model-int8.onnx +3 -0
MANIFEST.json
CHANGED
|
@@ -2,20 +2,24 @@
|
|
| 2 |
"run_id": "side-r9-ettin-150m-s1",
|
| 3 |
"files": {
|
| 4 |
"README.md": {
|
| 5 |
-
"bytes":
|
| 6 |
-
"sha256": "
|
| 7 |
},
|
| 8 |
"config.json": {
|
| 9 |
-
"bytes":
|
| 10 |
-
"sha256": "
|
| 11 |
},
|
| 12 |
"data/dev.jsonl.gz": {
|
| 13 |
"bytes": 1937924,
|
| 14 |
-
"sha256": "
|
| 15 |
},
|
| 16 |
"data/train.jsonl.gz": {
|
| 17 |
"bytes": 18648841,
|
| 18 |
-
"sha256": "
|
|
|
|
|
|
|
|
|
|
|
|
|
| 19 |
},
|
| 20 |
"model.onnx": {
|
| 21 |
"bytes": 598295149,
|
|
|
|
| 2 |
"run_id": "side-r9-ettin-150m-s1",
|
| 3 |
"files": {
|
| 4 |
"README.md": {
|
| 5 |
+
"bytes": 9704,
|
| 6 |
+
"sha256": "d962f78c592ee75579e22a4824a1a1df523b7a2e046a79c097eeab58e426dcf5"
|
| 7 |
},
|
| 8 |
"config.json": {
|
| 9 |
+
"bytes": 1027,
|
| 10 |
+
"sha256": "f12b2850bda3876d089d6376ca0998a3fa82d83704bd9bfc3f8125849ca1c23b"
|
| 11 |
},
|
| 12 |
"data/dev.jsonl.gz": {
|
| 13 |
"bytes": 1937924,
|
| 14 |
+
"sha256": "d9db0aa09f9756f8aa26c888ec62caa5253391552c3efc4d4471b37a8a5db872"
|
| 15 |
},
|
| 16 |
"data/train.jsonl.gz": {
|
| 17 |
"bytes": 18648841,
|
| 18 |
+
"sha256": "bb8c77d10fe57456d9146ff81b0adff887527a80d19812555dbd581929b8f0dd"
|
| 19 |
+
},
|
| 20 |
+
"model-int8.onnx": {
|
| 21 |
+
"bytes": 325445917,
|
| 22 |
+
"sha256": "91920b9ed982c5d2a7e1302d95c1f9f651fa53af36e5a1cac56290064e895929"
|
| 23 |
},
|
| 24 |
"model.onnx": {
|
| 25 |
"bytes": 598295149,
|
README.md
CHANGED
|
@@ -16,7 +16,7 @@ tags:
|
|
| 16 |
- turn-taking
|
| 17 |
---
|
| 18 |
|
| 19 |
-
# Pipecat Sidechat (experimental, rc0.3)
|
| 20 |
|
| 21 |
> **⚠️ Experimental.** Outputs, names and calibration may change between releases.
|
| 22 |
|
|
@@ -41,6 +41,11 @@ A request without the agent's name ("set a timer for ten minutes"), said before
|
|
| 41 |
for a pause about nine times in ten. In rc0.2 it was mostly ignored. The cost is more holds on people's own talk;
|
| 42 |
see Limitations. rc0.2 stays at tag `rc0.2`.
|
| 43 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 44 |
## Use
|
| 45 |
|
| 46 |
```bash
|
|
@@ -62,12 +67,18 @@ a = model.answer(SidechatRequest(
|
|
| 62 |
print(a.respond, model.decide(a)) # 0.97 respond
|
| 63 |
```
|
| 64 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 65 |
For `agent-1x`, load the folder and call `prepare` with the next line's request (its history is what counts)
|
| 66 |
as soon as the previous line ends; `answer` then encodes only the new line:
|
| 67 |
|
| 68 |
```python
|
| 69 |
path = snapshot_download("pipecat-ai/sidechat", allow_patterns=["agent-1x/*.json", "agent-1x/*.onnx"])
|
| 70 |
-
model = load(f"{path}/agent-1x", threads=1)
|
| 71 |
model.prepare(request) # while the person talks
|
| 72 |
a = model.answer(request) # when they stop
|
| 73 |
```
|
|
@@ -87,9 +98,11 @@ misheard otherwise ("Olly" as "Oli"), and a request that names the agent is then
|
|
| 87 |
|
| 88 |
| File | |
|
| 89 |
|---|---|
|
| 90 |
-
| `model.onnx` | default build, fp32, 598 MB
|
|
|
|
| 91 |
| `agent-1x/context.onnx`, `agent-1x/line.onnx` | `agent-1x` build, fp32, 128 + 131 MB |
|
| 92 |
-
| `
|
|
|
|
| 93 |
| `data/`, `agent-1x/data/` | the rows each build was trained on (train and dev) |
|
| 94 |
| `MANIFEST.json` (each folder) | sha256 of every file, row counts |
|
| 95 |
|
|
@@ -119,18 +132,23 @@ trained on.
|
|
| 119 |
|
| 120 |
## Speed
|
| 121 |
|
| 122 |
-
Median ms per line with 8 earlier lines, one thread, on containers capped like Pipecat Cloud's profiles
|
| 123 |
-
an API call from the same container, so its time is mostly the network
|
|
|
|
| 124 |
|
| 125 |
| | 0.5 vCPU, 1 GiB | 1 vCPU, 2 GiB | Memory |
|
| 126 |
|---|---|---|---|
|
| 127 |
-
| `agent-1x`, earlier lines encoded ahead |
|
| 128 |
-
| `agent-1x`, earlier lines
|
| 129 |
-
|
|
|
|
|
|
|
|
|
|
|
| 130 |
| Jev | 49 to 163 | 49 to 149 | |
|
| 131 |
|
| 132 |
-
|
| 133 |
-
|
|
|
|
| 134 |
takes 10 ms with a short history and 52 ms with 8 earlier lines.
|
| 135 |
|
| 136 |
## Limitations
|
|
@@ -157,6 +175,9 @@ unclear heads.
|
|
| 157 |
- **`agent-1x`:** Ettin-32M as one shared encoder for the earlier lines and the new line, joined by a
|
| 158 |
cross-attention layer. 3 epochs on 425,828 lines, with the default build's probabilities as 80% of each
|
| 159 |
target.
|
|
|
|
|
|
|
|
|
|
| 160 |
|
| 161 |
The lines come from three sources:
|
| 162 |
|
|
|
|
| 16 |
- turn-taking
|
| 17 |
---
|
| 18 |
|
| 19 |
+
# Pipecat Sidechat (experimental, rc0.3.1)
|
| 20 |
|
| 21 |
> **⚠️ Experimental.** Outputs, names and calibration may change between releases.
|
| 22 |
|
|
|
|
| 41 |
for a pause about nine times in ten. In rc0.2 it was mostly ignored. The cost is more holds on people's own talk;
|
| 42 |
see Limitations. rc0.2 stays at tag `rc0.2`.
|
| 43 |
|
| 44 |
+
**New in rc0.3.1:** int8 files for both builds, opt in with `int8=True`. The fp32 files are rc0.3's, unchanged,
|
| 45 |
+
and stay the default; rc0.3 stays at tag `rc0.3`. On every test set below, int8 at its own thresholds makes the
|
| 46 |
+
same decision as fp32 on 98.5% to 99.9% of lines and is within a point of it. It is smaller and faster; see Files
|
| 47 |
+
and Speed.
|
| 48 |
+
|
| 49 |
## Use
|
| 50 |
|
| 51 |
```bash
|
|
|
|
| 67 |
print(a.respond, model.decide(a)) # 0.97 respond
|
| 68 |
```
|
| 69 |
|
| 70 |
+
For int8, download its file and pass `int8=True`; the thresholds in `config.json`'s `thresholds_int8` are used:
|
| 71 |
+
|
| 72 |
+
```python
|
| 73 |
+
model = load(snapshot_download("pipecat-ai/sidechat", allow_patterns=["*.json", "model-int8.onnx"]), int8=True)
|
| 74 |
+
```
|
| 75 |
+
|
| 76 |
For `agent-1x`, load the folder and call `prepare` with the next line's request (its history is what counts)
|
| 77 |
as soon as the previous line ends; `answer` then encodes only the new line:
|
| 78 |
|
| 79 |
```python
|
| 80 |
path = snapshot_download("pipecat-ai/sidechat", allow_patterns=["agent-1x/*.json", "agent-1x/*.onnx"])
|
| 81 |
+
model = load(f"{path}/agent-1x", threads=1) # or int8=True, with "agent-1x/*-int8.onnx"
|
| 82 |
model.prepare(request) # while the person talks
|
| 83 |
a = model.answer(request) # when they stop
|
| 84 |
```
|
|
|
|
| 98 |
|
| 99 |
| File | |
|
| 100 |
|---|---|
|
| 101 |
+
| `model.onnx` | default build, fp32, 598 MB |
|
| 102 |
+
| `model-int8.onnx` | default build, int8, 325 MB |
|
| 103 |
| `agent-1x/context.onnx`, `agent-1x/line.onnx` | `agent-1x` build, fp32, 128 + 131 MB |
|
| 104 |
+
| `agent-1x/context-int8.onnx`, `agent-1x/line-int8.onnx` | `agent-1x` build, int8, 97 + 98 MB |
|
| 105 |
+
| `tokenizer.json`, `config.json` (each folder) | tokenizer, calibration, and thresholds for fp32 and int8 |
|
| 106 |
| `data/`, `agent-1x/data/` | the rows each build was trained on (train and dev) |
|
| 107 |
| `MANIFEST.json` (each folder) | sha256 of every file, row counts |
|
| 108 |
|
|
|
|
| 132 |
|
| 133 |
## Speed
|
| 134 |
|
| 135 |
+
Median ms per line with 8 earlier lines, one thread, on containers capped like Pipecat Cloud's profiles, fp32 and
|
| 136 |
+
int8 in the same run. Jev is an API call from the same container, so its time is mostly the network; its row is
|
| 137 |
+
from earlier runs.
|
| 138 |
|
| 139 |
| | 0.5 vCPU, 1 GiB | 1 vCPU, 2 GiB | Memory |
|
| 140 |
|---|---|---|---|
|
| 141 |
+
| `agent-1x` int8, earlier lines encoded ahead | 10 | 7 | 428 MB |
|
| 142 |
+
| `agent-1x` fp32, earlier lines encoded ahead | 11 | 16 | 487 MB |
|
| 143 |
+
| `agent-1x` int8, earlier lines not ready | 95 | 32 | |
|
| 144 |
+
| `agent-1x` fp32, earlier lines not ready | 104 | 77 | |
|
| 145 |
+
| Default, int8 | 672 | 303 | 628 MB |
|
| 146 |
+
| Default, fp32 | 991 (near the memory limit; it took 3 minutes to load) | 465 | about 0.9 GB |
|
| 147 |
| Jev | 49 to 163 | 49 to 149 | |
|
| 148 |
|
| 149 |
+
Times move with the host's CPU between runs: an earlier run on another host measured the default fp32 build at
|
| 150 |
+
253 ms on 1 vCPU. Compare rows within the table. Encoding the earlier lines on `agent-1x` takes 86 ms in int8 and
|
| 151 |
+
93 ms in fp32 at 0.5 vCPU, well inside the time anyone takes to say a line. On an M5 Max the default fp32 build
|
| 152 |
takes 10 ms with a short history and 52 ms with 8 earlier lines.
|
| 153 |
|
| 154 |
## Limitations
|
|
|
|
| 175 |
- **`agent-1x`:** Ettin-32M as one shared encoder for the earlier lines and the new line, joined by a
|
| 176 |
cross-attention layer. 3 epochs on 425,828 lines, with the default build's probabilities as 80% of each
|
| 177 |
target.
|
| 178 |
+
- **int8 (rc0.3.1):** dynamic int8 weights, per channel with reduced range, with each layer's MLP output kept in
|
| 179 |
+
fp32. Without reduced range, int8 lost about a third of its decisions on CPUs without VNNI. Each build's int8
|
| 180 |
+
thresholds were chosen on dev, where it is within 0.1 points of fp32.
|
| 181 |
|
| 182 |
The lines come from three sources:
|
| 183 |
|
config.json
CHANGED
|
@@ -27,5 +27,21 @@
|
|
| 27 |
"max_wrong_now_real": 0.004,
|
| 28 |
"max_held_real": 0.002
|
| 29 |
},
|
| 30 |
-
"exported_at": "2026-11-20T12:29:54+00:00"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 31 |
}
|
|
|
|
| 27 |
"max_wrong_now_real": 0.004,
|
| 28 |
"max_held_real": 0.002
|
| 29 |
},
|
| 30 |
+
"exported_at": "2026-11-20T12:29:54+00:00",
|
| 31 |
+
"thresholds_int8": {
|
| 32 |
+
"respond": 0.6,
|
| 33 |
+
"wait": 0.45,
|
| 34 |
+
"open": 0.5,
|
| 35 |
+
"dev_right": 21611,
|
| 36 |
+
"dev_n": 21841,
|
| 37 |
+
"dev_wrong_now": 24,
|
| 38 |
+
"dev_quiet": 13074,
|
| 39 |
+
"max_wrong_now": 0.01,
|
| 40 |
+
"real_wrong_now": 0,
|
| 41 |
+
"real_held": 5,
|
| 42 |
+
"real_quiet": 4186,
|
| 43 |
+
"max_wrong_now_real": 0.004,
|
| 44 |
+
"max_held_real": 0.002
|
| 45 |
+
},
|
| 46 |
+
"int8_recipe": "per channel, reduce_range, MatMul with constant B only, each layer's mlp.Wo left in fp32"
|
| 47 |
}
|
data/dev.jsonl.gz
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
size 1937924
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:d9db0aa09f9756f8aa26c888ec62caa5253391552c3efc4d4471b37a8a5db872
|
| 3 |
size 1937924
|
data/train.jsonl.gz
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
size 18648841
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:bb8c77d10fe57456d9146ff81b0adff887527a80d19812555dbd581929b8f0dd
|
| 3 |
size 18648841
|
model-int8.onnx
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:91920b9ed982c5d2a7e1302d95c1f9f651fa53af36e5a1cac56290064e895929
|
| 3 |
+
size 325445917
|