Joshuant commited on
Commit
a6539d5
·
verified ·
1 Parent(s): 117d9d7

Apache-2.0 licence for the weights; banner

Browse files

The weights (cue.safetensors, config.json) are now Apache-2.0, like the encoder. NOTICE keeps the CC BY 4.0 attributions for the training data.

Files changed (5) hide show
  1. .gitattributes +1 -0
  2. LICENSE +202 -0
  3. NOTICE +4 -1
  4. README.md +143 -141
  5. banner.png +3 -0
.gitattributes CHANGED
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ banner.png filter=lfs diff=lfs merge=lfs -text
LICENSE ADDED
@@ -0,0 +1,202 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+
2
+ Apache License
3
+ Version 2.0, January 2004
4
+ http://www.apache.org/licenses/
5
+
6
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
7
+
8
+ 1. Definitions.
9
+
10
+ "License" shall mean the terms and conditions for use, reproduction,
11
+ and distribution as defined by Sections 1 through 9 of this document.
12
+
13
+ "Licensor" shall mean the copyright owner or entity authorized by
14
+ the copyright owner that is granting the License.
15
+
16
+ "Legal Entity" shall mean the union of the acting entity and all
17
+ other entities that control, are controlled by, or are under common
18
+ control with that entity. For the purposes of this definition,
19
+ "control" means (i) the power, direct or indirect, to cause the
20
+ direction or management of such entity, whether by contract or
21
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
22
+ outstanding shares, or (iii) beneficial ownership of such entity.
23
+
24
+ "You" (or "Your") shall mean an individual or Legal Entity
25
+ exercising permissions granted by this License.
26
+
27
+ "Source" form shall mean the preferred form for making modifications,
28
+ including but not limited to software source code, documentation
29
+ source, and configuration files.
30
+
31
+ "Object" form shall mean any form resulting from mechanical
32
+ transformation or translation of a Source form, including but
33
+ not limited to compiled object code, generated documentation,
34
+ and conversions to other media types.
35
+
36
+ "Work" shall mean the work of authorship, whether in Source or
37
+ Object form, made available under the License, as indicated by a
38
+ copyright notice that is included in or attached to the work
39
+ (an example is provided in the Appendix below).
40
+
41
+ "Derivative Works" shall mean any work, whether in Source or Object
42
+ form, that is based on (or derived from) the Work and for which the
43
+ editorial revisions, annotations, elaborations, or other modifications
44
+ represent, as a whole, an original work of authorship. For the purposes
45
+ of this License, Derivative Works shall not include works that remain
46
+ separable from, or merely link (or bind by name) to the interfaces of,
47
+ the Work and Derivative Works thereof.
48
+
49
+ "Contribution" shall mean any work of authorship, including
50
+ the original version of the Work and any modifications or additions
51
+ to that Work or Derivative Works thereof, that is intentionally
52
+ submitted to Licensor for inclusion in the Work by the copyright owner
53
+ or by an individual or Legal Entity authorized to submit on behalf of
54
+ the copyright owner. For the purposes of this definition, "submitted"
55
+ means any form of electronic, verbal, or written communication sent
56
+ to the Licensor or its representatives, including but not limited to
57
+ communication on electronic mailing lists, source code control systems,
58
+ and issue tracking systems that are managed by, or on behalf of, the
59
+ Licensor for the purpose of discussing and improving the Work, but
60
+ excluding communication that is conspicuously marked or otherwise
61
+ designated in writing by the copyright owner as "Not a Contribution."
62
+
63
+ "Contributor" shall mean Licensor and any individual or Legal Entity
64
+ on behalf of whom a Contribution has been received by Licensor and
65
+ subsequently incorporated within the Work.
66
+
67
+ 2. Grant of Copyright License. Subject to the terms and conditions of
68
+ this License, each Contributor hereby grants to You a perpetual,
69
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
70
+ copyright license to reproduce, prepare Derivative Works of,
71
+ publicly display, publicly perform, sublicense, and distribute the
72
+ Work and such Derivative Works in Source or Object form.
73
+
74
+ 3. Grant of Patent License. Subject to the terms and conditions of
75
+ this License, each Contributor hereby grants to You a perpetual,
76
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
77
+ (except as stated in this section) patent license to make, have made,
78
+ use, offer to sell, sell, import, and otherwise transfer the Work,
79
+ where such license applies only to those patent claims licensable
80
+ by such Contributor that are necessarily infringed by their
81
+ Contribution(s) alone or by combination of their Contribution(s)
82
+ with the Work to which such Contribution(s) was submitted. If You
83
+ institute patent litigation against any entity (including a
84
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
85
+ or a Contribution incorporated within the Work constitutes direct
86
+ or contributory patent infringement, then any patent licenses
87
+ granted to You under this License for that Work shall terminate
88
+ as of the date such litigation is filed.
89
+
90
+ 4. Redistribution. You may reproduce and distribute copies of the
91
+ Work or Derivative Works thereof in any medium, with or without
92
+ modifications, and in Source or Object form, provided that You
93
+ meet the following conditions:
94
+
95
+ (a) You must give any other recipients of the Work or
96
+ Derivative Works a copy of this License; and
97
+
98
+ (b) You must cause any modified files to carry prominent notices
99
+ stating that You changed the files; and
100
+
101
+ (c) You must retain, in the Source form of any Derivative Works
102
+ that You distribute, all copyright, patent, trademark, and
103
+ attribution notices from the Source form of the Work,
104
+ excluding those notices that do not pertain to any part of
105
+ the Derivative Works; and
106
+
107
+ (d) If the Work includes a "NOTICE" text file as part of its
108
+ distribution, then any Derivative Works that You distribute must
109
+ include a readable copy of the attribution notices contained
110
+ within such NOTICE file, excluding those notices that do not
111
+ pertain to any part of the Derivative Works, in at least one
112
+ of the following places: within a NOTICE text file distributed
113
+ as part of the Derivative Works; within the Source form or
114
+ documentation, if provided along with the Derivative Works; or,
115
+ within a display generated by the Derivative Works, if and
116
+ wherever such third-party notices normally appear. The contents
117
+ of the NOTICE file are for informational purposes only and
118
+ do not modify the License. You may add Your own attribution
119
+ notices within Derivative Works that You distribute, alongside
120
+ or as an addendum to the NOTICE text from the Work, provided
121
+ that such additional attribution notices cannot be construed
122
+ as modifying the License.
123
+
124
+ You may add Your own copyright statement to Your modifications and
125
+ may provide additional or different license terms and conditions
126
+ for use, reproduction, or distribution of Your modifications, or
127
+ for any such Derivative Works as a whole, provided Your use,
128
+ reproduction, and distribution of the Work otherwise complies with
129
+ the conditions stated in this License.
130
+
131
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
132
+ any Contribution intentionally submitted for inclusion in the Work
133
+ by You to the Licensor shall be under the terms and conditions of
134
+ this License, without any additional terms or conditions.
135
+ Notwithstanding the above, nothing herein shall supersede or modify
136
+ the terms of any separate license agreement you may have executed
137
+ with Licensor regarding such Contributions.
138
+
139
+ 6. Trademarks. This License does not grant permission to use the trade
140
+ names, trademarks, service marks, or product names of the Licensor,
141
+ except as required for reasonable and customary use in describing the
142
+ origin of the Work and reproducing the content of the NOTICE file.
143
+
144
+ 7. Disclaimer of Warranty. Unless required by applicable law or
145
+ agreed to in writing, Licensor provides the Work (and each
146
+ Contributor provides its Contributions) on an "AS IS" BASIS,
147
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
148
+ implied, including, without limitation, any warranties or conditions
149
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
150
+ PARTICULAR PURPOSE. You are solely responsible for determining the
151
+ appropriateness of using or redistributing the Work and assume any
152
+ risks associated with Your exercise of permissions under this License.
153
+
154
+ 8. Limitation of Liability. In no event and under no legal theory,
155
+ whether in tort (including negligence), contract, or otherwise,
156
+ unless required by applicable law (such as deliberate and grossly
157
+ negligent acts) or agreed to in writing, shall any Contributor be
158
+ liable to You for damages, including any direct, indirect, special,
159
+ incidental, or consequential damages of any character arising as a
160
+ result of this License or out of the use or inability to use the
161
+ Work (including but not limited to damages for loss of goodwill,
162
+ work stoppage, computer failure or malfunction, or any and all
163
+ other commercial damages or losses), even if such Contributor
164
+ has been advised of the possibility of such damages.
165
+
166
+ 9. Accepting Warranty or Additional Liability. While redistributing
167
+ the Work or Derivative Works thereof, You may choose to offer,
168
+ and charge a fee for, acceptance of support, warranty, indemnity,
169
+ or other liability obligations and/or rights consistent with this
170
+ License. However, in accepting such obligations, You may act only
171
+ on Your own behalf and on Your sole responsibility, not on behalf
172
+ of any other Contributor, and only if You agree to indemnify,
173
+ defend, and hold each Contributor harmless for any liability
174
+ incurred by, or claims asserted against, such Contributor by reason
175
+ of your accepting any such warranty or additional liability.
176
+
177
+ END OF TERMS AND CONDITIONS
178
+
179
+ APPENDIX: How to apply the Apache License to your work.
180
+
181
+ To apply the Apache License to your work, attach the following
182
+ boilerplate notice, with the fields enclosed by brackets "[]"
183
+ replaced with your own identifying information. (Don't include
184
+ the brackets!) The text should be enclosed in the appropriate
185
+ comment syntax for the file format. We also recommend that a
186
+ file or class name and description of purpose be included on the
187
+ same "printed page" as the copyright notice for easier
188
+ identification within third-party archives.
189
+
190
+ Copyright [yyyy] [name of copyright owner]
191
+
192
+ Licensed under the Apache License, Version 2.0 (the "License");
193
+ you may not use this file except in compliance with the License.
194
+ You may obtain a copy of the License at
195
+
196
+ http://www.apache.org/licenses/LICENSE-2.0
197
+
198
+ Unless required by applicable law or agreed to in writing, software
199
+ distributed under the License is distributed on an "AS IS" BASIS,
200
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
201
+ See the License for the specific language governing permissions and
202
+ limitations under the License.
NOTICE CHANGED
@@ -1,4 +1,7 @@
1
- Cue v5 - third-party components and data
 
 
 
2
 
3
  encoder/ The encoder of OpenAI Whisper small (openai/whisper-small), Apache License 2.0.
4
  Modification: only the encoder is kept and its positional-embedding table is cut to the
 
1
+ Cue v5
2
+ Copyright 2026 IOTEverythin. Licensed under the Apache License, Version 2.0 (see LICENSE).
3
+
4
+ Third-party components and data
5
 
6
  encoder/ The encoder of OpenAI Whisper small (openai/whisper-small), Apache License 2.0.
7
  Modification: only the encoder is kept and its positional-embedding table is cut to the
README.md CHANGED
@@ -1,141 +1,143 @@
1
- ---
2
- license: other
3
- license_name: ioteverythin-research
4
- library_name: transformers
5
- pipeline_tag: audio-classification
6
- base_model: openai/whisper-small
7
- tags:
8
- - voice-agent
9
- - turn-taking
10
- - barge-in
11
- - endpointing
12
- - full-duplex
13
- - indian-english
14
- ---
15
-
16
- # Cue v5: turn-taking for voice agents, trained on real conversation
17
-
18
- Cue listens to a call and decides, every 160 ms, what a voice agent should do: keep talking,
19
- **PAUSE** or **STOP** while it speaks; **RESPOND** or keep waiting after the caller stops.
20
- v5 is Cue v4 retrained on real human conversation as well as synthetic calls, and it can hear
21
- the agent's own audio.
22
-
23
- **What changed from v4**
24
-
25
- - **Real conversation.** Besides the synthetic Indian English calls, v5 was trained on the AMI and
26
- ICSI meeting corpora (CC BY 4.0): real backchannels, overlaps, interruptions and hesitations.
27
- On held-out real meetings it catches about twice as many interruptions as v4.
28
- - **The agent's own audio as a second input.** A live agent knows what it is playing; v5 hears it
29
- through a second encoder pass, which helps it tell the agent's own voice (echo) from the caller.
30
- It is optional: without it the input is zero, a condition v5 was also trained on.
31
- - **Phone conditions.** Meetings and calls were trained clean and through echo, reverb, recorded
32
- noise (MUSAN) and an 8 kHz phone codec.
33
- - **No shortcut through the agent's reply.** In recordings the other side starts talking right after
34
- a turn ends; a live agent only does after Cue decides. The agent audio is hidden in that window
35
- and meetings are trained without the other side's audio, so v5 does not learn that shortcut.
36
-
37
- **Status: research model.** Real meetings are not phone calls, and v5 has not been tested on
38
- production call traffic.
39
-
40
- ## Use
41
-
42
- ```python
43
- import cue_turn # pip install "cue-turn[full]"
44
- cue = cue_turn.load("v5") # or a folder with this repository's files
45
- stream = cue.stream(sample_rate=8000, agent_sample_rate=24000)
46
- for caller_chunk, agent_chunk in call: # the agent chunk: what it played over the same span
47
- for d in stream.feed(caller_chunk, assistant_speaking=bot_is_playing(), agent_audio=agent_chunk):
48
- handle(d) # {"t_ms": ..., "decision": "STOP" | "PAUSE" | "RESPOND"}
49
- ```
50
-
51
- In Pipecat, add `CueAgentAudioTap(session)` just before `transport.output()` so the session sees the
52
- bot's audio; see the cue-turn README. A GPU is recommended (two encoder passes while the agent
53
- speaks; one while it is quiet).
54
-
55
- ## Decision settings
56
-
57
- | Profile | Barge-in (stop_p, hold) | End of turn (p_done, turn_hold) |
58
- |---|---|---|
59
- | responsive | 0.7, 4 | 0.5, 6 |
60
- | **balanced (default)** | **0.8, 6** | **0.9, 12** |
61
- | cautious | 0.95, 6 | 0.95, 16 |
62
-
63
- All with cooldown 1500 ms, min_speech 10 frames, fallback 2000 ms. Defaults were chosen on AMI
64
- validation meetings and the synthetic test sets together: real speech wants lower thresholds for
65
- interruptions and longer holds for end of turn than synthetic speech does.
66
-
67
- | Setting | Synthetic barge-in, clean: hard / takeover / stops on backchannel | AMI validation: interruptions caught / false alarms |
68
- |---|---|---|
69
- | 0.7, 4 | 85.5% / 96.9% / 11.4% | 63% / 7.8% |
70
- | **0.8, 6** | **90.3% / 93.8% / 8.8%** | **39% / 4.1%** |
71
- | 0.95, 6 | 91.9% / 87.5% / 3.5% | 2% / 0% |
72
-
73
- | Setting | Synthetic end of turn, clean: mid-thought / answered / delay | AMI validation: turn ends caught / answers into pauses |
74
- |---|---|---|
75
- | 0.5, 6 | 4.5% / 95.6% / 137 ms | 76% / 37% |
76
- | **0.9, 12** | **3.7% / 96.7% / 256 ms** | **63% / 25%** |
77
- | 0.95, 16 | 3.0% / 96.6% / 336 ms | 51% / 14% |
78
-
79
- ## Results
80
-
81
- ### Real meetings (held-out AMI and ICSI test meetings; best setting with at most 10% false alarms)
82
-
83
- | | v4 | **v5** |
84
- |---|---|---|
85
- | AMI interruptions caught | 24% | **61%** |
86
- | ICSI interruptions caught | 37% | **63%** |
87
- | AMI end of turn | none under 10% false alarms | **27%** (1.2 s) |
88
- | ICSI end of turn | none under 10% false alarms | **59%** (1.06 s) |
89
-
90
- ### Synthetic test calls (each model's best setting under the v4 card's matched rules)
91
-
92
- | | Hard | Takeovers | Soft | Stops on backchannel | Stops on noise | Reaction p50 |
93
- |---|---|---|---|---|---|---|
94
- | v4, clean | 85.5% | 90.6% | 93.8% | 3.5% | 5.6% | 243 ms |
95
- | v5, clean | 90.3% | 93.8% | 92.5% | 7.0% | 7.4% | 241 ms |
96
- | v4, realistic | 75.8% | 75.0% | 91.2% | 3.5% | 8.0% | 311 ms |
97
- | v5, realistic | 85.5% | 81.2% | 93.8% | 4.4% | 11.7% | 281 ms |
98
-
99
- | End of turn | Answers mid-thought | Turn ends answered | Delay p50 |
100
- |---|---|---|---|
101
- | v4, clean / realistic | 5.2% / 7.5% | 95.5% / 94.4% | 177 / 185 ms |
102
- | v5, clean / realistic | 4.5% / 7.5% | 95.6% / 94.1% | 137 / 154 ms |
103
-
104
- ### TurnBench (dev set, cross-validated settings, official scorer)
105
-
106
- Both models without any TurnBench training data. Recall / false alarms / median delay.
107
-
108
- | | End of turn | End of turn, Cue's end-of-turn judgement | Interruption |
109
- |---|---|---|---|
110
- | v4 | 0.772 / 7.7% / 1.41 s | 0.608 / 8.7% / 1.16 s | 0.614 / 8.8% / 583 ms |
111
- | v5, without the other speaker's audio | 0.779 / 8.9% / 1.38 s | 0.605 / 7.0% / 947 ms | **0.804 / 7.9% / 547 ms** |
112
- | v5, with the other speaker's audio | **0.800 / 7.0% / 1.16 s** | **0.686 / 8.1% / 908 ms** | 0.712 / 9.2% / 653 ms |
113
-
114
- The best end-of-turn rule on TurnBench is a silence timer on Cue's speech detection (about 1 s of
115
- quiet): the benchmark ranks recall within a false-alarm budget and does not count delay. Published
116
- systems on the same dev set include VAP (0.841 / 0.957) and Kyutai's semantic VAD (0.803 / 0.934).
117
-
118
- ## Training
119
-
120
- | | |
121
- |---|---|
122
- | Start | Cue v4 (the agent-audio weights start at zero, so step 0 is v4) |
123
- | Data | 2,600 synthetic call tracks (1,300 calls, clean + realistic; 32 with Dia-1.6B caller voices), AMI 20 meetings (80 speaker tracks), ICSI 10 meetings (30 speaker tracks); sampled 50 / 30 / 20% |
124
- | Labels | synthetic: exact event timings from the generator; meetings: from the corpus dialogue acts (backchannels, floor-taking, interrupted utterances), word timings and vocal sounds |
125
- | Augmentation | echo, room reverb, MUSAN noise, 8 kHz mu-law codec, level, on half the tracks |
126
- | Agent input | the agent's audio, kept only while the agent is playing and hidden in the 1.5 s decision window; zeroed for 15% of synthetic segments and for all meetings |
127
- | Optimiser | AdamW, lr 1e-4, one-cycle, batch 16 x 20 s, 8,000 steps, bfloat16 on an RTX 4060 laptop GPU (79 min) |
128
-
129
- ## Limitations
130
-
131
- - Real meetings are not phone calls; v5 has not been validated on production calls.
132
- - End of turn on real speech is still slow (about 1 s) and catches about half of turn ends.
133
- - The default settings trade synthetic false stops for real-speech recall; tune per deployment.
134
- - Indian-English and Hinglish backchannels ("haan", "achha") are only in the synthetic data.
135
-
136
- ## Licences
137
-
138
- - `encoder/`: Whisper small's encoder, Apache-2.0 (position table shortened; see NOTICE).
139
- - `cue.safetensors`, `config.json`: IOTEverythin, research use.
140
- - Training data: AMI and ICSI (CC BY 4.0), MUSAN (CC BY 4.0), Chatterbox (MIT), Dia-1.6B (Apache-2.0),
141
- GLOBE (CC0), Svarah (CC BY 4.0). See NOTICE.
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ library_name: transformers
4
+ pipeline_tag: audio-classification
5
+ base_model: openai/whisper-small
6
+ tags:
7
+ - voice-agent
8
+ - turn-taking
9
+ - barge-in
10
+ - endpointing
11
+ - full-duplex
12
+ - indian-english
13
+ ---
14
+
15
+ <p align="center"><img src="https://huggingface.co/IOTEverythin/cue-v5/resolve/main/banner.png" alt="Cue v5: caller audio and the agent's own audio in; STOP, PAUSE or RESPOND out. 2.5x more real interruptions caught than Cue v4 on AMI meetings; 0.80 interruption recall on TurnBench." width="100%"></p>
16
+
17
+ # Cue v5: turn-taking for voice agents, trained on real conversation
18
+
19
+ Cue listens to a call and decides, every 160 ms, what a voice agent should do: keep talking,
20
+ **PAUSE** or **STOP** while it speaks; **RESPOND** or keep waiting after the caller stops.
21
+ v5 is Cue v4 retrained on real human conversation as well as synthetic calls, and it can hear
22
+ the agent's own audio.
23
+
24
+ **What changed from v4**
25
+
26
+ - **Real conversation.** Besides the synthetic Indian English calls, v5 was trained on the AMI and
27
+ ICSI meeting corpora (CC BY 4.0): real backchannels, overlaps, interruptions and hesitations.
28
+ On held-out real meetings it catches about twice as many interruptions as v4.
29
+ - **The agent's own audio as a second input.** A live agent knows what it is playing; v5 hears it
30
+ through a second encoder pass, which helps it tell the agent's own voice (echo) from the caller.
31
+ It is optional: without it the input is zero, a condition v5 was also trained on.
32
+ - **Phone conditions.** Meetings and calls were trained clean and through echo, reverb, recorded
33
+ noise (MUSAN) and an 8 kHz phone codec.
34
+ - **No shortcut through the agent's reply.** In recordings the other side starts talking right after
35
+ a turn ends; a live agent only does after Cue decides. The agent audio is hidden in that window
36
+ and meetings are trained without the other side's audio, so v5 does not learn that shortcut.
37
+
38
+ **Status: early release.** Real meetings are not phone calls, and v5 has not been tested on
39
+ production call traffic.
40
+
41
+ ## Use
42
+
43
+ ```python
44
+ import cue_turn # pip install "cue-turn[full]"
45
+ cue = cue_turn.load("v5") # or a folder with this repository's files
46
+ stream = cue.stream(sample_rate=8000, agent_sample_rate=24000)
47
+ for caller_chunk, agent_chunk in call: # the agent chunk: what it played over the same span
48
+ for d in stream.feed(caller_chunk, assistant_speaking=bot_is_playing(), agent_audio=agent_chunk):
49
+ handle(d) # {"t_ms": ..., "decision": "STOP" | "PAUSE" | "RESPOND"}
50
+ ```
51
+
52
+ In Pipecat, add `CueAgentAudioTap(session)` just before `transport.output()` so the session sees the
53
+ bot's audio; see the cue-turn README. A GPU is recommended (two encoder passes while the agent
54
+ speaks; one while it is quiet).
55
+
56
+ ## Decision settings
57
+
58
+ | Profile | Barge-in (stop_p, hold) | End of turn (p_done, turn_hold) |
59
+ |---|---|---|
60
+ | responsive | 0.7, 4 | 0.5, 6 |
61
+ | **balanced (default)** | **0.8, 6** | **0.9, 12** |
62
+ | cautious | 0.95, 6 | 0.95, 16 |
63
+
64
+ All with cooldown 1500 ms, min_speech 10 frames, fallback 2000 ms. Defaults were chosen on AMI
65
+ validation meetings and the synthetic test sets together: real speech wants lower thresholds for
66
+ interruptions and longer holds for end of turn than synthetic speech does.
67
+
68
+ | Setting | Synthetic barge-in, clean: hard / takeover / stops on backchannel | AMI validation: interruptions caught / false alarms |
69
+ |---|---|---|
70
+ | 0.7, 4 | 85.5% / 96.9% / 11.4% | 63% / 7.8% |
71
+ | **0.8, 6** | **90.3% / 93.8% / 8.8%** | **39% / 4.1%** |
72
+ | 0.95, 6 | 91.9% / 87.5% / 3.5% | 2% / 0% |
73
+
74
+ | Setting | Synthetic end of turn, clean: mid-thought / answered / delay | AMI validation: turn ends caught / answers into pauses |
75
+ |---|---|---|
76
+ | 0.5, 6 | 4.5% / 95.6% / 137 ms | 76% / 37% |
77
+ | **0.9, 12** | **3.7% / 96.7% / 256 ms** | **63% / 25%** |
78
+ | 0.95, 16 | 3.0% / 96.6% / 336 ms | 51% / 14% |
79
+
80
+ ## Results
81
+
82
+ ### Real meetings (held-out AMI and ICSI test meetings; best setting with at most 10% false alarms)
83
+
84
+ | | v4 | **v5** |
85
+ |---|---|---|
86
+ | AMI interruptions caught | 24% | **61%** |
87
+ | ICSI interruptions caught | 37% | **63%** |
88
+ | AMI end of turn | none under 10% false alarms | **27%** (1.2 s) |
89
+ | ICSI end of turn | none under 10% false alarms | **59%** (1.06 s) |
90
+
91
+ ### Synthetic test calls (each model's best setting under the v4 card's matched rules)
92
+
93
+ | | Hard | Takeovers | Soft | Stops on backchannel | Stops on noise | Reaction p50 |
94
+ |---|---|---|---|---|---|---|
95
+ | v4, clean | 85.5% | 90.6% | 93.8% | 3.5% | 5.6% | 243 ms |
96
+ | v5, clean | 90.3% | 93.8% | 92.5% | 7.0% | 7.4% | 241 ms |
97
+ | v4, realistic | 75.8% | 75.0% | 91.2% | 3.5% | 8.0% | 311 ms |
98
+ | v5, realistic | 85.5% | 81.2% | 93.8% | 4.4% | 11.7% | 281 ms |
99
+
100
+ | End of turn | Answers mid-thought | Turn ends answered | Delay p50 |
101
+ |---|---|---|---|
102
+ | v4, clean / realistic | 5.2% / 7.5% | 95.5% / 94.4% | 177 / 185 ms |
103
+ | v5, clean / realistic | 4.5% / 7.5% | 95.6% / 94.1% | 137 / 154 ms |
104
+
105
+ ### TurnBench (dev set, cross-validated settings, official scorer)
106
+
107
+ Both models without any TurnBench training data. Recall / false alarms / median delay.
108
+
109
+ | | End of turn | End of turn, Cue's end-of-turn judgement | Interruption |
110
+ |---|---|---|---|
111
+ | v4 | 0.772 / 7.7% / 1.41 s | 0.608 / 8.7% / 1.16 s | 0.614 / 8.8% / 583 ms |
112
+ | v5, without the other speaker's audio | 0.779 / 8.9% / 1.38 s | 0.605 / 7.0% / 947 ms | **0.804 / 7.9% / 547 ms** |
113
+ | v5, with the other speaker's audio | **0.800 / 7.0% / 1.16 s** | **0.686 / 8.1% / 908 ms** | 0.712 / 9.2% / 653 ms |
114
+
115
+ The best end-of-turn rule on TurnBench is a silence timer on Cue's speech detection (about 1 s of
116
+ quiet): the benchmark ranks recall within a false-alarm budget and does not count delay. Published
117
+ systems on the same dev set include VAP (0.841 / 0.957) and Kyutai's semantic VAD (0.803 / 0.934).
118
+
119
+ ## Training
120
+
121
+ | | |
122
+ |---|---|
123
+ | Start | Cue v4 (the agent-audio weights start at zero, so step 0 is v4) |
124
+ | Data | 2,600 synthetic call tracks (1,300 calls, clean + realistic; 32 with Dia-1.6B caller voices), AMI 20 meetings (80 speaker tracks), ICSI 10 meetings (30 speaker tracks); sampled 50 / 30 / 20% |
125
+ | Labels | synthetic: exact event timings from the generator; meetings: from the corpus dialogue acts (backchannels, floor-taking, interrupted utterances), word timings and vocal sounds |
126
+ | Augmentation | echo, room reverb, MUSAN noise, 8 kHz mu-law codec, level, on half the tracks |
127
+ | Agent input | the agent's audio, kept only while the agent is playing and hidden in the 1.5 s decision window; zeroed for 15% of synthetic segments and for all meetings |
128
+ | Optimiser | AdamW, lr 1e-4, one-cycle, batch 16 x 20 s, 8,000 steps, bfloat16 on an RTX 4060 laptop GPU (79 min) |
129
+
130
+ ## Limitations
131
+
132
+ - Real meetings are not phone calls; v5 has not been validated on production calls.
133
+ - End of turn on real speech is still slow (about 1 s) and catches about half of turn ends.
134
+ - The default settings trade synthetic false stops for real-speech recall; tune per deployment.
135
+ - Indian-English and Hinglish backchannels ("haan", "achha") are only in the synthetic data.
136
+
137
+ ## Licences
138
+
139
+ - `encoder/`: Whisper small's encoder, Apache-2.0 (position table shortened; see NOTICE).
140
+ - `cue.safetensors`, `config.json`: IOTEverythin, Apache-2.0 (see LICENSE). Commercial use is allowed;
141
+ keep the NOTICE file, which carries the attributions the CC BY 4.0 training data requires.
142
+ - Training data: AMI and ICSI (CC BY 4.0), MUSAN (CC BY 4.0), Chatterbox (MIT), Dia-1.6B (Apache-2.0),
143
+ GLOBE (CC0), Svarah (CC BY 4.0). See NOTICE.
banner.png ADDED

Git LFS Details

  • SHA256: 12e15667e6a2eb0681f9f05c6c51938e61d9e025fce90ce5c71df2453c640685
  • Pointer size: 132 Bytes
  • Size of remote file: 1.12 MB