R1: streaming end-of-turn heads on the Smart Turn v3.2 encoder

AMI_LICENCE.txt

@@ -0,0 +1,307 @@
+The AMI corpus and its annotations are released under the Creative
+Commons Attribution 4.0 International Public License agreement (CC BY
+4.0).  Use of this data implies agreement with the terms below. See
+also: https://creativecommons.org/licenses/by/4.0/
+
+
+THE WORK (AS DEFINED BELOW) IS PROVIDED UNDER THE TERMS OF THIS
+CREATIVE COMMONS PUBLIC LICENSE ("CCPL" OR "LICENSE"). THE WORK IS
+PROTECTED BY COPYRIGHT AND/OR OTHER APPLICABLE LAW. ANY USE OF THE
+WORK OTHER THAN AS AUTHORIZED UNDER THIS LICENSE OR COPYRIGHT LAW IS
+PROHIBITED.
+
+BY EXERCISING ANY RIGHTS TO THE WORK PROVIDED HERE, YOU ACCEPT AND
+AGREE TO BE BOUND BY THE TERMS OF THIS LICENSE. THE LICENSOR GRANTS
+YOU THE RIGHTS CONTAINED HERE IN CONSIDERATION OF YOUR ACCEPTANCE OF
+SUCH TERMS AND CONDITIONS.
+
+Section 1 – Definitions.
+
+a. Adapted Material means material subject to Copyright and Similar
+Rights that is derived from or based upon the Licensed Material and in
+which the Licensed Material is translated, altered, arranged,
+transformed, or otherwise modified in a manner requiring permission
+under the Copyright and Similar Rights held by the Licensor. For
+purposes of this Public License, where the Licensed Material is a
+musical work, performance, or sound recording, Adapted Material is
+always produced where the Licensed Material is synched in timed
+relation with a moving image.  
+
+b. Adapter's License means the license You apply to Your Copyright and
+Similar Rights in Your contributions to Adapted Material in accordance
+with the terms and conditions of this Public License.
+
+c. Copyright and Similar Rights means copyright and/or similar rights
+closely related to copyright including, without limitation,
+performance, broadcast, sound recording, and Sui Generis Database
+Rights, without regard to how the rights are labeled or
+categorized. For purposes of this Public License, the rights specified
+in Section 2(b)(1)-(2) are not Copyright and Similar Rights.
+
+d. Effective Technological Measures means those measures that, in the
+absence of proper authority, may not be circumvented under laws
+fulfilling obligations under Article 11 of the WIPO Copyright Treaty
+adopted on December 20, 1996, and/or similar international agreements.
+
+e. Exceptions and Limitations means fair use, fair dealing, and/or any
+other exception or limitation to Copyright and Similar Rights that
+applies to Your use of the Licensed Material.
+
+f. Licensed Material means the artistic or literary work, database, or
+other material to which the Licensor applied this Public License.
+
+g. Licensed Rights means the rights granted to You subject to the
+terms and conditions of this Public License, which are limited to all
+Copyright and Similar Rights that apply to Your use of the Licensed
+Material and that the Licensor has authority to license.
+
+h. Licensor means the individual(s) or entity(ies) granting rights
+under this Public License.
+
+i. Share means to provide material to the public by any means or
+process that requires permission under the Licensed Rights, such as
+reproduction, public display, public performance, distribution,
+dissemination, communication, or importation, and to make material
+available to the public including in ways that members of the public
+may access the material from a place and at a time individually chosen
+by them.
+
+j. Sui Generis Database Rights means rights other than copyright
+resulting from Directive 96/9/EC of the European Parliament and of the
+Council of 11 March 1996 on the legal protection of databases, as
+amended and/or succeeded, as well as other essentially equivalent
+rights anywhere in the world.
+
+k. You means the individual or entity exercising the Licensed Rights
+under this Public License. Your has a corresponding meaning.
+
+
+Section 2 – Scope.
+
+a. License grant.
+
+1. Subject to the terms and conditions of this Public License, the
+Licensor hereby grants You a worldwide, royalty-free,
+non-sublicensable, non-exclusive, irrevocable license to exercise the
+Licensed Rights in the Licensed Material to:
+  A. reproduce and Share the Licensed Material, in whole or in part; and
+  B. produce, reproduce, and Share Adapted Material.
+
+2. Exceptions and Limitations. For the avoidance of doubt, where
+Exceptions and Limitations apply to Your use, this Public License does
+not apply, and You do not need to comply with its terms and
+conditions.
+
+3. Term. The term of this Public License is specified in Section 6(a).
+
+4. Media and formats; technical modifications allowed. The Licensor
+authorizes You to exercise the Licensed Rights in all media and
+formats whether now known or hereafter created, and to make technical
+modifications necessary to do so. The Licensor waives and/or agrees
+not to assert any right or authority to forbid You from making
+technical modifications necessary to exercise the Licensed Rights,
+including technical modifications necessary to circumvent Effective
+Technological Measures. For purposes of this Public License, simply
+making modifications authorized by this Section 2(a)(4) never produces
+Adapted Material.
+
+5. Downstream recipients.
+
+  A. Offer from the Licensor – Licensed Material. Every recipient of
+  the Licensed Material automatically receives an offer from the
+  Licensor to exercise the Licensed Rights under the terms and
+  conditions of this Public License.
+
+  B. No downstream restrictions. You may not offer or impose any
+  additional or different terms or conditions on, or apply any
+  Effective Technological Measures to, the Licensed Material if doing
+  so restricts exercise of the Licensed Rights by any recipient of the
+  Licensed Material.
+
+6. No endorsement. Nothing in this Public License constitutes or may
+be construed as permission to assert or imply that You are, or that
+Your use of the Licensed Material is, connected with, or sponsored,
+endorsed, or granted official status by, the Licensor or others
+designated to receive attribution as provided in Section
+3(a)(1)(A)(i).
+
+b. Other rights.
+
+1. Moral rights, such as the right of integrity, are not licensed
+under this Public License, nor are publicity, privacy, and/or other
+similar personality rights; however, to the extent possible, the
+Licensor waives and/or agrees not to assert any such rights held by
+the Licensor to the limited extent necessary to allow You to exercise
+the Licensed Rights, but not otherwise.
+
+2. Patent and trademark rights are not licensed under this Public License.
+
+3. To the extent possible, the Licensor waives any right to collect
+royalties from You for the exercise of the Licensed Rights, whether
+directly or through a collecting society under any voluntary or
+waivable statutory or compulsory licensing scheme. In all other cases
+the Licensor expressly reserves any right to collect such royalties.
+
+
+Section 3 – License Conditions.
+
+Your exercise of the Licensed Rights is expressly made subject to the
+following conditions.
+
+a. Attribution.
+
+1. If You Share the Licensed Material (including in modified form),
+ You must:
+
+  A. retain the following if it is supplied by the Licensor with the
+  Licensed Material:
+
+   i. identification of the creator(s) of the Licensed Material and
+   any others designated to receive attribution, in any reasonable
+   manner requested by the Licensor (including by pseudonym if
+   designated);
+
+   ii. a copyright notice;
+   
+   iii. a notice that refers to this Public License;
+
+   iv. a notice that refers to the disclaimer of warranties;
+
+   v. a URI or hyperlink to the Licensed Material to the extent
+   reasonably practicable;
+
+  B. indicate if You modified the Licensed Material and retain an
+  indication of any previous modifications; and
+
+  C. indicate the Licensed Material is licensed under this Public
+  License, and include the text of, or the URI or hyperlink to, this
+  Public License.
+
+2. You may satisfy the conditions in Section 3(a)(1) in any reasonable
+manner based on the medium, means, and context in which You Share the
+Licensed Material. For example, it may be reasonable to satisfy the
+conditions by providing a URI or hyperlink to a resource that includes
+the required information.
+
+3. If requested by the Licensor, You must remove any of the
+information required by Section 3(a)(1)(A) to the extent reasonably
+practicable.
+
+4. If You Share Adapted Material You produce, the Adapter's License
+You apply must not prevent recipients of the Adapted Material from
+complying with this Public License.  
+
+
+Section 4 – Sui Generis Database Rights.
+
+Where the Licensed Rights include Sui Generis Database Rights that
+apply to Your use of the Licensed Material:
+
+a. for the avoidance of doubt, Section 2(a)(1) grants You the right to
+extract, reuse, reproduce, and Share all or a substantial portion of
+the contents of the database;
+
+b. if You include all or a substantial portion of the database
+contents in a database in which You have Sui Generis Database Rights,
+then the database in which You have Sui Generis Database Rights (but
+not its individual contents) is Adapted Material; and
+
+c. You must comply with the conditions in Section 3(a) if You Share
+all or a substantial portion of the contents of the database.
+
+For the avoidance of doubt, this Section 4 supplements and does not
+replace Your obligations under this Public License where the Licensed
+Rights include other Copyright and Similar Rights.
+
+
+Section 5 – Disclaimer of Warranties and Limitation of Liability.
+
+a. Unless otherwise separately undertaken by the Licensor, to the
+extent possible, the Licensor offers the Licensed Material as-is and
+as-available, and makes no representations or warranties of any kind
+concerning the Licensed Material, whether express, implied, statutory,
+or other. This includes, without limitation, warranties of title,
+merchantability, fitness for a particular purpose, non-infringement,
+absence of latent or other defects, accuracy, or the presence or
+absence of errors, whether or not known or discoverable. Where
+disclaimers of warranties are not allowed in full or in part, this
+disclaimer may not apply to You.
+
+b. To the extent possible, in no event will the Licensor be liable to
+You on any legal theory (including, without limitation, negligence) or
+otherwise for any direct, special, indirect, incidental,
+consequential, punitive, exemplary, or other losses, costs, expenses,
+or damages arising out of this Public License or use of the Licensed
+Material, even if the Licensor has been advised of the possibility of
+such losses, costs, expenses, or damages. Where a limitation of
+liability is not allowed in full or in part, this limitation may not
+apply to You.
+
+c. The disclaimer of warranties and limitation of liability provided
+above shall be interpreted in a manner that, to the extent possible,
+most closely approximates an absolute disclaimer and waiver of all
+liability.
+
+
+Section 6 – Term and Termination.
+
+a. This Public License applies for the term of the Copyright and
+Similar Rights licensed here. However, if You fail to comply with this
+Public License, then Your rights under this Public License terminate
+automatically.
+
+b. Where Your right to use the Licensed Material has terminated under
+Section 6(a), it reinstates:
+
+  1. automatically as of the date the violation is cured, provided it
+  is cured within 30 days of Your discovery of the violation; or
+
+  2. upon express reinstatement by the Licensor.
+
+For the avoidance of doubt, this Section 6(b) does not affect any
+right the Licensor may have to seek remedies for Your violations of
+this Public License.
+
+
+c. For the avoidance of doubt, the Licensor may also offer the
+Licensed Material under separate terms or conditions or stop
+distributing the Licensed Material at any time; however, doing so will
+not terminate this Public License.
+
+d. Sections 1, 5, 6, 7, and 8 survive termination of this Public
+License.  
+
+
+Section 7 – Other Terms and Conditions.
+
+a. The Licensor shall not be bound by any additional or different
+terms or conditions communicated by You unless expressly agreed.
+
+b. Any arrangements, understandings, or agreements regarding the
+Licensed Material not stated herein are separate from and independent
+of the terms and conditions of this Public License.
+
+
+Section 8 – Interpretation.
+
+a. For the avoidance of doubt, this Public License does not, and shall
+not be interpreted to, reduce, limit, restrict, or impose conditions
+on any use of the Licensed Material that could lawfully be made
+without permission under this Public License.
+
+b. To the extent possible, if any provision of this Public License is
+deemed unenforceable, it shall be automatically reformed to the
+minimum extent necessary to make it enforceable. If the provision
+cannot be reformed, it shall be severed from this Public License
+without affecting the enforceability of the remaining terms and
+conditions.
+
+c. No term or condition of this Public License will be waived and no
+failure to comply consented to unless expressly agreed to by the
+Licensor.
+
+d. Nothing in this Public License constitutes or may be interpreted as
+a limitation upon, or waiver of, any privileges and immunities that
+apply to the Licensor or You, including from the legal processes of
+any jurisdiction or authority.
+

ATTRIBUTION.md

@@ -0,0 +1,47 @@
+# Attribution and licences
+
+## This release
+
+The trained heads (`eot-heads-v32.onnx`), the decision-rule tables (`rules/`), the evaluation
+script and manifests (`eval/`) and the model card are released by susuROBO under the BSD 2-Clause
+License (`LICENSE`).
+
+## Encoder: Smart Turn v3.2 (Daily / Pipecat)
+
+`smart-turn-v3.2-encoder-pooled.onnx` is Pipecat's `smart-turn-v3.2-gpu.onnx`
+(https://huggingface.co/pipecat-ai/smart-turn-v3, BSD 2-Clause, Copyright (c) 2024–2025 Daily)
+with one change: the attention-pooled 384-d vector that feeds the model's own classifier is added
+as a second graph output (`sum_1`). Weights are unchanged, so the first output (`logits`) is
+bit-exact with the original model. The encoder architecture is OpenAI's Whisper-tiny encoder
+(MIT), as fine-tuned by Pipecat on `pipecat-ai/smart-turn-data-v3.2-train`.
+
+## Training data of the heads
+
+- `pipecat-ai/smart-turn-data-v3.2-train` (Pipecat; the τ-padded and pre-offset variants described
+  in the model card are derived from it on the fly and are not redistributed).
+- **AMI Meeting Corpus** (University of Edinburgh and the AMI consortium),
+  https://groups.inf.ed.ac.uk/ami/corpus/, released under the Creative Commons Attribution 4.0
+  International licence (`AMI_LICENCE.txt`, the licence text shipped with the corpus annotations,
+  release 1.7). The per-speaker close-talking headset recordings and the NXT word alignments were
+  used; the heads were trained on the 137 Kaldi-recipe training meetings.
+
+  Citation: J. Carletta et al., "The AMI Meeting Corpus: A Pre-announcement", in *Machine Learning
+  for Multimodal Interaction (MLMI 2005)*, LNCS 3869, Springer, 2006, pp. 28–39.
+
+## Evaluation manifests (`eval/ami_dev.csv`, `eval/ami_test.csv`)
+
+Derived from the AMI Meeting Corpus (CC BY 4.0) and redistributed under the same licence with
+attribution to the corpus authors above. Each row is one labelled silence of one speaker:
+`meeting`, `speaker` (NXT agent letter), `channel` (headset number in the corpus audio), `t` (the
+speaker's VAD stop, seconds into the meeting), `label` (`complete`: another participant took the
+floor within 3 s; `incomplete`: the speaker resumed), `gap_s` (incomplete: length of the pause),
+`b_onset_s` (complete: the next speaker's onset after `t`), `a_turn_s` (speech in the speaker's turn
+so far). The VAD comes from Silero run on the headset audio and gated by the transcript; the
+labelling rules are documented in the public repository (`docs/a2_ami.md`). The dev split is the
+Kaldi AMI recipe's dev set (18 meetings), the test split its eval set (16 meetings). Audio is not
+redistributed: `eval/ami_eval.py` downloads the headset files it needs from the corpus mirror.
+
+## Not used
+
+Nothing in this release was trained or fitted on Meta's Seamless Interaction dataset (CC BY-NC) or
+on Meta's Casual Conversations dataset.

LICENSE

@@ -0,0 +1,25 @@
+BSD 2-Clause License
+
+Copyright (c) 2026, susuROBO
+Copyright (c) 2024–2025, Daily
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this
+   list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice,
+   this list of conditions and the following disclaimer in the documentation
+   and/or other materials provided with the distribution.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

README.md

@@ -0,0 +1,225 @@
+---
+license: bsd-2-clause
+library_name: onnx
+pipeline_tag: audio-classification
+tags:
+  - turn-detection
+  - end-of-turn
+  - voice-agents
+  - pipecat
+  - streaming
+  - onnx
+datasets:
+  - pipecat-ai/smart-turn-data-v3.2-train
+language:
+  - en
+  - multilingual
+base_model: pipecat-ai/smart-turn-v3
+---
+
+# Smart Turn streaming end-of-turn heads (R1, audio-only)
+
+A silence-aware, streaming end-of-turn detector for voice agents, built on Pipecat's frozen
+[Smart Turn v3.2](https://huggingface.co/pipecat-ai/smart-turn-v3) encoder. Instead of one query
+at the VAD stop followed by a 3 s timeout, it scores the conversation every 100 ms on the CPU and
+decides at 0.2 / 0.5 / 1.2 s of silence with thresholds fitted to a false-alarm budget. On real
+multi-party conversation (AMI Meeting Corpus) it ends the turn on 5 % of mid-turn pauses where the
+stock rule fires on 42 %, and brings the 90th-percentile response time from 3.2 s to 1.3 s. It runs
+at about 9 % of one Apple M4 core single-threaded, with numpy and onnxruntime only.
+
+Code (Pipecat 1.14 turn analyzer, eager-then-commit stop strategy, training and evaluation):
+[github.com/OpenHRIai/smart-turn-streaming](https://github.com/OpenHRIai/smart-turn-streaming).
+
+![Wins and costs against stock Smart Turn v3.2](https://raw.githubusercontent.com/OpenHRIai/smart-turn-streaming/main/docs/figures/wins_and_costs.png)
+
+![One turn end under the three rules](https://raw.githubusercontent.com/OpenHRIai/smart-turn-streaming/main/docs/figures/timeline_turn_end.png)
+
+![One mid-turn pause under the three rules](https://raw.githubusercontent.com/OpenHRIai/smart-turn-streaming/main/docs/figures/timeline_pause.png)
+
+## Files
+
+| file | size | what |
+|---|---|---|
+| `smart-turn-v3.2-encoder-pooled.onnx` | 31 MB | Pipecat's Smart Turn v3.2 (Whisper-tiny encoder + its classifier), weights unchanged, with the pooled 384-d vector exposed as a second output. Input `input_features` (1, 80, 800) Whisper log-mel of the last 8 s; outputs `logits` (the stock model's P(complete)) and `sum_1` (the pooled vector). |
+| `eot-heads-v32.onnx` | 0.7 MB | The trained heads: LayerNorm + Linear on the pooled vector, a GRU over the last second of log-mel (20 × 80 at 50 ms), the silence length, a fusion MLP. Inputs `pooled` (1, 384), `mel20` (1, 20, 80), `tau_feat` (1, 3) = [τ / 2.5, log1p(τ), speaking]; output `probs` (1, 2) = [P(turn complete), P(turn ends within 0.4 s)]. 172k parameters. |
+| `rules/theta_tables.csv` | | Commit thresholds θ(0.2 s), θ(0.5 s), θ(1.2 s) fitted on AMI dev at false-alarm budgets of 5 / 10 / 15 %, with their AMI test numbers. |
+| `rules/eager_tables.csv` | | Eager thresholds for Pipecat's speculation path, fitted on AMI dev at budgets of LLM generations spent on pauses (20–50 %). |
+| `eval/ami_eval.py`, `eval/ami_{dev,test}.csv`, `eval/ami_eval_reference.md` | | Reproduces the AMI tables below from the public corpus audio (numpy, onnxruntime, soundfile); the reference file is its output on 2026-10-25. |
+| `LICENSE`, `AMI_LICENCE.txt`, `ATTRIBUTION.md` | | BSD 2-Clause for this release; CC BY 4.0 for the AMI-derived manifests; sources and citations. |
+
+## How it works
+
+Two cadences in one object on one CPU thread, plus one worker thread for the encoder:
+
+| path | when | work | cost (Apple M4, 1 ORT thread) |
+|---|---|---|---|
+| mel | every 20 ms audio frame | incremental Whisper log-mel, one 10 ms column at a time (exact to 0.003 against the reference extractor on real audio) | 0.02 ms |
+| heads | every 100 ms, and at each silence-clock point | last 1 s of mel → GRU, cached pooled vector, τ → P(complete) | 0.7 ms |
+| encoder | every 320 ms while the user speaks; at τ = 0, 0.2, 0.5, 0.8, 1.2, 1.8, 2.5 s of silence (the grid the heads were trained on), where the decision waits for it | 8 s window → Whisper-tiny → pooled vector, on a worker thread | 22 ms (46 ms when the core is otherwise idle) |
+
+The decision rule queries P(complete) at 0.2, 0.5 and 1.2 s of silence while the pause lasts and
+commits at the first offset where it exceeds θ(offset); Pipecat's `stop_secs` (3 s) stays as the
+backstop. The budget is the product knob: `far10` for an agent that must not interrupt, `far15` when
+a fast reply matters more. A second, looser table (`eager_tables.csv`) on the same decision points
+drives Pipecat's speculation path (`UserTurnSpeculation` → `SpeculationGate`): the LLM starts
+generating at the eager point, the response is held until the commit and discarded if the user
+resumes.
+
+Why silence awareness has to be trained: 79 % of the clips in `smart-turn-data-v3.2` end within
+0.2 s of the speech, so the published models never saw a pause, and their false-alarm rate on
+incomplete utterances rises from 5 % at the offset to 19 % after 2.5 s of silence. The heads were
+trained on windows ending τ ∈ {0 … 2.5} s after the acoustic offset (clips padded with their own
+room tone) plus 43k real conversational pauses from AMI. The GRU over the last second of mel is the
+component that carries the silence awareness (AUC at 1.2 s of silence 0.74 with it, 0.66 without).
+
+## Results
+
+### AMI Meeting Corpus, test split (511 turn ends, 4,304 mid-turn pauses, 16 meetings, 4-party)
+
+Offline, windows ending at τ seconds of silence (`eval/ami_eval.py` reproduces this table):
+
+| τ (s) | AUC heads | AUC frozen Smart Turn v3.2 |
+|---|---|---|
+| 0.2 | 0.775 | 0.671 |
+| 0.5 | 0.776 | 0.638 |
+| 1.2 | 0.742 | 0.575 |
+| 2.5 | 0.735 | 0.619 |
+
+Over three training seeds the AUC at 0.2 s is 0.78 ± 0.01 and at 1.2 s 0.73 ± 0.01.
+
+Decision rule, thresholds fitted on AMI dev, scored on AMI test; a turn end with no commit is
+charged the 3 s timeout:
+
+| rule | θ at 0.2 / 0.5 / 1.2 s | recall | false alarms on pauses | mean latency | FAR on pauses ≥ 1.5 s |
+|---|---|---|---|---|---|
+| stock rule (one query at the VAD stop, θ 0.5), frozen v3.2 | 0.50 / – / – | 0.51 | 30.4 % | 1.56 s | 0.45 |
+| stock rule, heads | 0.50 / – / – | 0.64 | 25.5 % | 1.21 s | 0.46 |
+| re-query, 10 % budget, frozen v3.2 | 1.00 / 1.00 / 0.28 | 0.70 | 10.6 % | 1.74 s | 0.61 |
+| **re-query, 10 % budget, heads (`far10`)** | 0.92 / 0.98 / 0.86 | **0.80** | **9.4 %** | **1.48 s** | 0.50 |
+| re-query, 15 % budget, frozen v3.2 | 1.00 / 1.00 / 0.02 | 0.87 | 13.7 % | 1.44 s | 0.78 |
+| **re-query, 15 % budget, heads (`far15`)** | 0.98 / 0.88 / 0.80 | **0.86** | **13.3 %** | **1.21 s** | 0.61 |
+| no model: end every turn at 1.2 s of silence | – | 1.00 | 18.0 % | 1.20 s | 1.00 |
+
+### Regression guard: `smart-turn-data-v3.2` test (6,000 clips, balanced)
+
+| τ (s) | AUC heads / frozen v3.2 | recall at 5 % FAR, heads / frozen |
+|---|---|---|
+| 0.0 | 0.975 / 0.978 | 0.87 / 0.89 |
+| 0.2 | 0.976 / 0.976 | 0.87 / 0.86 |
+| 1.2 | 0.972 / 0.964 | 0.84 / 0.77 |
+| 2.5 | 0.965 / 0.955 | 0.81 / 0.72 |
+
+Per-language and per-source AUC at τ = 0.2 s is unchanged within noise. The heads are calibrated
+differently from the frozen model (class-weighted loss), so use the fitted tables, not θ = 0.5.
+
+### End to end in Pipecat 1.14 (real-time replay of 160 AMI test silences through Silero VAD and `UserTurnProcessor`)
+
+| analyzer and rule | false alarms on pauses | turn ends committed by the model (not the 3 s backstop) | commit latency p50 / p90 | response time p50 / p90 / mean at 0.8 s of LLM latency | LLM generations spent on pauses |
+|---|---|---|---|---|---|
+| stock `LocalSmartTurnAnalyzerV3`, v3.2 fp32 | 42.5 % | 0.70 | 0.23 / 3.20 s | 1.03 / 4.00 / 1.89 s | 42.5 % |
+| streaming, `far10` | 2.5 % | 0.78 | 1.21 / 3.00 s | 2.01 / 3.80 / 2.36 s | 2.5 % |
+| streaming, `far15` | 5.0 % | 0.93 | 1.19 / 1.27 s | 1.99 / 2.07 / 1.92 s | 5.0 % |
+| **streaming, `far15` + eager 30 %** | **5.0 %** | **0.93** | 1.19 / 1.26 s | **1.31 / 2.04 / 1.55 s** | 18.8 % |
+
+With the eager stage the median response time matches the stock analyzer within 0.3 s, the p90 is
+halved, an eighth of the user turns are split, and only 2.5 % of pauses are audibly interrupted
+(stock: 7.5 %).
+
+### Cost (Apple M4, single ORT thread, numpy + onnxruntime)
+
+| | |
+|---|---|
+| detector share of one core, tight loop | 9.1 % (encoder refreshes included) |
+| whole Pipecat process (Silero VAD, pipeline, analyzer) | 16–19 % of one core at real-time pace (stock analyzer: 9 %) |
+| heads hop | 0.7 ms |
+| encoder refresh | 22 ms (46 ms at real-time pace on an otherwise idle machine) |
+| pipeline-task wall time per audio frame, encoder on the worker thread | p99 4–5 ms (inline: 45 ms); the 0.5 s and 1.2 s decisions 0.2 ms |
+| resident memory added by the analyzer | +114 MB (stock: +76 MB) |
+| model files | 32 MB |
+
+A cloud vCPU is about 3× slower than an M4 core for this encoder (Pipecat's own figure for the same
+architecture on Pipecat Cloud), so expect roughly 25–30 % of one vCPU for the detector alone.
+
+## Usage
+
+### In Pipecat (recommended)
+
+```python
+from eot.pipecat_analyzer import StreamingEOTAnalyzer, StreamingEOTParams
+from eot.pipecat_eager_strategy import TurnAnalyzerEagerUserTurnStopStrategy
+from pipecat.turns.user_turn_strategies import UserTurnStrategies
+from pipecat.turns.user_start import VADUserTurnStartStrategy
+
+analyzer = StreamingEOTAnalyzer(
+    encoder_onnx="smart-turn-v3.2-encoder-pooled.onnx",
+    heads_onnx="eot-heads-v32.onnx",
+    params=StreamingEOTParams.from_rule("far15", eager="far15_gen30"),   # commit and eager tables
+)
+strategies = UserTurnStrategies(
+    start=[VADUserTurnStartStrategy()],
+    stop=[TurnAnalyzerEagerUserTurnStopStrategy(turn_analyzer=analyzer)],   # or pipecat's TurnAnalyzerUserTurnStopStrategy
+)
+```
+
+`eot/` is in the repository linked above; the analyzer needs 16 kHz audio and Pipecat's default VAD
+`stop_secs` of 0.2 s. `StreamingEOTParams.from_rule("far10")` without an eager table gives the
+one-stage rule.
+
+### Standalone streaming (numpy + onnxruntime)
+
+```python
+from eot.streaming import StreamingEOT
+
+eot = StreamingEOT("smart-turn-v3.2-encoder-pooled.onnx", "eot-heads-v32.onnx", worker=True)
+for frame, is_speech in audio_frames():        # 20 ms float32 frames in [-1, 1] and the VAD's verdict
+    out = eot.push(frame, is_speech)          # None, or {"p_end", "p_eager", "tau", "stale_s", ...} at a hop
+    if out and not out["speaking"] and out["tau"] >= 1.2 and out["p_end"] >= 0.80:
+        ...                                    # end of turn (the far15 table's 1.2 s threshold)
+```
+
+### One window, offline
+
+```python
+import numpy as np, onnxruntime as ort
+enc = ort.InferenceSession("smart-turn-v3.2-encoder-pooled.onnx")
+heads = ort.InferenceSession("eot-heads-v32.onnx")
+feats = whisper_logmel(last_8_seconds)          # (80, 800); eval/ami_eval.py has a numpy implementation
+logits, pooled = enc.run(None, {"input_features": feats[None]})
+mel20 = feats[:, -100:].reshape(80, 20, 5).mean(-1).T[None]       # last 1 s at 50 ms steps
+tau = 0.5                                                          # seconds of silence at the window end
+tau_feat = np.array([[tau / 2.5, np.log1p(tau), 0.0]], np.float32)
+p_end, p_eager = heads.run(None, {"pooled": pooled, "mel20": mel20, "tau_feat": tau_feat})[0][0]
+```
+
+## Limitations
+
+- **Conversational evaluation is English-only.** The silence-aware behaviour was trained and measured on
+  English meetings (AMI); the per-language check covers 23 languages but only on isolated utterances
+  (`smart-turn-data`), where the heads stay within noise of the frozen model in every language. Whether
+  the pause thresholds transfer to other languages' conversational timing is not measured.
+- Meetings have a ceiling set by the labels: ending every turn at 1.2 s of silence
+  gives recall 1.0 at 18 % false alarms, and below that budget about a quarter of turn ends still
+  wait for the backstop, for every model tried.
+- Long pauses (≥ 1.5 s) remain the weak spot: 50–60 % false alarms at the 10–15 % budgets.
+- The second output (`p_eager`, "the turn ends within 0.4 s") is not usable as a free-running
+  trigger during speech (AUC 0.70 on AMI; ~22 false fires per minute of speech). The eager tables use
+  P(complete) at the silence-clock points instead.
+- Every decision waits for an encoder refresh of its own window: the 0.2 s decision at the VAD stop for up
+  to 50 ms (`wait_s`), the 0.5 s and 1.2 s decisions for up to 80 ms (`max_defer_s`); the encoder takes
+  22–46 ms on an M4 and about 3× that on a cloud vCPU, so raise the allowances there. If the refresh is
+  late the decision uses the cached vector, up to 320 ms old, and the analyzer records the staleness of
+  every decision. With the refresh in time the streaming decisions equal the offline evaluation's within
+  0.001 AUC on AMI.
+- Labels are by outcome, not intent: a speaker who was cut off counts as a complete turn.
+
+## Reproduce the AMI tables
+
+```
+pip install numpy onnxruntime soundfile
+python eval/ami_eval.py --audio-dir ./ami_audio          # downloads 34 headset files (~3 GB), ~10 min on an M4
+```
+
+## Citation
+
+If you use this model, please cite Pipecat's Smart Turn (the encoder) and the AMI Meeting Corpus
+(`ATTRIBUTION.md`).

SHA256SUMS

@@ -0,0 +1,12 @@
+8f5e4b1f7e4dfff70198bb2867f14b75f2e3afd94944334de21ac4b9712fa29e  README.md
+dd4613e925f3d6770eb3705863dc30d0cd6427456445ae533d5906bf6687ad12  LICENSE
+d52fd9c7c19eec9f0bec3107554c10937203e261259d73b01637a61501fec7f7  AMI_LICENCE.txt
+2418f9016f3ba767fc400e9b6732bb3b6a31c8e2c72672861a084f44a7499c87  ATTRIBUTION.md
+c9c1d5905088965e754fcf718765c897a779658b5fdc6ae5de471eeab41c49c0  rules/theta_tables.csv
+3a63a46477243d1ff6e7620a96aa8cde257209b40506202312f6aff139c4e5ad  rules/eager_tables.csv
+ccdfe3ad97d9ff204644c53a027610bfaf50c20b00ee991931109a016fb3e942  eval/ami_eval.py
+9f4fb10b3f0488e5bb2faa0b3247364c68b8e074db5e77be39045008e05049cc  eval/ami_dev.csv
+77ca3e93c35c951d2cc337154d70d94ff1f9ba71e06b3d9e08cc958f0b1034a0  eval/ami_test.csv
+8fccc07d5262daa94989fb7f5ff92a993a38038291e33c5cb217e24b8d53e4ec  eval/ami_eval_reference.md
+37e2a4fe22e1c4efaeba4df40141e422a3b96519db9160ae9134962a936ba4ea  smart-turn-v3.2-encoder-pooled.onnx
+8d5b4039c9d7c27e22e84d372c343077489f2fe291ccd67caeeb5ce45b05ce1a  eot-heads-v32.onnx

eot-heads-v32.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8d5b4039c9d7c27e22e84d372c343077489f2fe291ccd67caeeb5ce45b05ce1a
+size 696344

eval/ami_eval.py

@@ -0,0 +1,451 @@
+#!/usr/bin/env python3
+"""Reproduce the AMI end-of-turn evaluation of the Smart Turn v2 streaming heads (release R1).
+
+    python ami_eval.py --audio-dir ./ami_audio            # downloads the 34 test (and dev) headset files, ~3 GB
+    python ami_eval.py --audio-dir ./ami_audio --splits test --no-fit    # test only, shipped thresholds, ~5 min
+
+What it computes, from the public AMI Meeting Corpus audio and the labelled-silence manifests shipped
+next to this script (``ami_dev.csv``, ``ami_test.csv``; CC BY 4.0, see ../ATTRIBUTION.md):
+
+1. For every labelled silence (a VAD stop of speaker A at time ``t``) and every silence offset tau in
+   ``--taus`` the 8 s window of A's headset audio ending at ``t + tau``. For complete turns the audio after
+   the next speaker's onset (``t + b_onset_s``) is replaced by A's own room tone, because a deployed
+   microphone never hears the agent's reply; for incomplete turns a window exists only while the pause is
+   still going on (``tau < gap_s``).
+2. Whisper log-mel features (80 x 800, as ``WhisperFeatureExtractor(chunk_length=8)`` with
+   ``do_normalize=True``), the frozen Smart Turn v3.2 encoder with its pooled 384-d vector exposed, and
+   the trained heads (pooled vector + last 1 s of mel + tau) -> P(complete). The frozen model's own
+   probability is kept as the baseline.
+3. AUC (complete vs incomplete) per tau for the heads and the frozen model, and the deployed decision
+   rule: query at 0.2 / 0.5 / 1.2 s of silence while the pause lasts, commit COMPLETE at the first offset
+   with P >= theta(offset); a complete with no commit is charged the 3 s timeout. Thresholds are the
+   shipped tables (``../rules/theta_tables.csv``) and, unless ``--no-fit``, re-fitted on the dev split under
+   false-alarm budgets of 5 / 10 / 15 % (exhaustive grid, min mean latency), then scored on test.
+
+Dependencies: numpy, onnxruntime, soundfile. CPU only; about 16k encoder windows for dev + test at the
+three rule offsets (~7 min single-threaded on an Apple M4). Scores are cached in ``--cache`` so the
+tables can be re-made without re-running the models.
+"""
+
+from __future__ import annotations
+
+import argparse
+import csv
+import itertools
+import json
+import math
+import os
+import sys
+import time
+import urllib.request
+from pathlib import Path
+
+import numpy as np
+
+SR = 16000
+WIN_S = 8.0
+N_FFT, HOP, N_MELS, WIN_FRAMES = 400, 160, 80, 800
+MEL20_FRAMES, MEL20_GROUP = 100, 5
+OFFSETS = (0.2, 0.5, 1.2)
+TAU_GRID = (0.0, 0.1, 0.2, 0.3, 0.5, 0.8, 1.2, 1.8, 2.5)
+TIMEOUT_S = 3.0
+MIRROR = "https://groups.inf.ed.ac.uk/ami/AMICorpusMirror/amicorpus"
+HERE = Path(__file__).resolve().parent
+
+
+# --- Whisper log-mel in numpy ---------------------------------------------------------------------
+
+def _hz_to_mel(f):
+    f = np.asarray(f, dtype=np.float64)
+    m = 3.0 * f / 200.0
+    hi = f >= 1000.0
+    m[hi] = 15.0 + np.log(f[hi] / 1000.0) * (27.0 / np.log(6.4))
+    return m
+
+
+def _mel_to_hz(m):
+    m = np.asarray(m, dtype=np.float64)
+    f = 200.0 * m / 3.0
+    hi = m >= 15.0
+    f[hi] = 1000.0 * np.exp(np.log(6.4) / 27.0 * (m[hi] - 15.0))
+    return f
+
+
+def mel_filters(n_freq: int = N_FFT // 2 + 1, n_mels: int = N_MELS, sr: int = SR) -> np.ndarray:
+    """(201, 80) Slaney-normalised triangular filterbank, identical to transformers' ``mel_filter_bank``."""
+    fft_freqs = np.linspace(0.0, sr / 2, n_freq)
+    mel_pts = np.linspace(_hz_to_mel(np.array([0.0]))[0], _hz_to_mel(np.array([sr / 2]))[0], n_mels + 2)
+    f = _mel_to_hz(mel_pts)
+    d = np.diff(f)
+    slopes = f[None, :] - fft_freqs[:, None]
+    fb = np.maximum(0.0, np.minimum(-slopes[:, :-2] / d[None, :-1], slopes[:, 2:] / d[None, 1:]))
+    enorm = 2.0 / (f[2: n_mels + 2] - f[:n_mels])
+    return (fb * enorm[None, :]).astype(np.float32)
+
+
+_FILTERS = mel_filters()
+_WINDOW = np.hanning(N_FFT + 1)[:-1].astype(np.float32)
+
+
+def whisper_logmel(x: np.ndarray) -> np.ndarray:
+    """(80, 800) features of exactly 8 s of 16 kHz audio, as WhisperFeatureExtractor(chunk_length=8)
+    with padding='max_length', truncation=True, do_normalize=True gives them."""
+    x = np.asarray(x, dtype=np.float64)
+    assert len(x) == int(WIN_S * SR), len(x)
+    x = (x - x.mean()) / np.sqrt(x.var() + 1e-7)                       # do_normalize
+    pad = N_FFT // 2
+    xp = np.concatenate([x[1: pad + 1][::-1], x, x[-pad - 1: -1][::-1]])   # reflect padding, centre=True
+    n_frames = 1 + (len(xp) - N_FFT) // HOP                               # 801; the last one is dropped
+    idx = np.arange(N_FFT)[None, :] + HOP * np.arange(n_frames)[:, None]
+    frames = xp[idx] * _WINDOW
+    spec = np.abs(np.fft.rfft(frames, axis=1)) ** 2                       # (801, 201)
+    mel = spec[:-1] @ _FILTERS                                            # (800, 80)
+    ls = np.log10(np.maximum(mel, 1e-10))
+    ls = np.maximum(ls, ls.max() - 8.0)
+    return ((ls + 4.0) / 4.0).T.astype(np.float32)
+
+
+def mel20(feats: np.ndarray) -> np.ndarray:
+    """(20, 80): the last second of the features averaged to 50 ms steps (the heads' fresh-mel input)."""
+    return feats[:, -MEL20_FRAMES:].reshape(N_MELS, MEL20_FRAMES // MEL20_GROUP, MEL20_GROUP).mean(-1).T.astype(np.float32)
+
+
+# --- windows ----------------------------------------------------------------------------------------
+
+def room_tone(x: np.ndarray, sr: int, t_off: float, n: int, rng: np.random.Generator) -> np.ndarray:
+    """``n`` samples of room tone tiled from the audio after ``t_off`` (0.5 s tiles, random order, some
+    reversed, 10 ms crossfades), as the training windows were built."""
+    tail = x[int(round(t_off * sr)):]
+    if n <= 0:
+        return np.zeros(0, np.float32)
+    if len(tail) < int(0.05 * sr):
+        nf = len(x) // 320
+        if nf >= 5:
+            fr = x[: nf * 320].reshape(nf, 320).astype(np.float64)
+            level = float(np.percentile(np.sqrt(np.mean(fr * fr, axis=1)), 5))
+        else:
+            level = float(np.sqrt(np.mean(np.square(x, dtype=np.float64)))) if len(x) else 0.0
+        level = float(np.clip(level, 10 ** (-70 / 20), 10 ** (-45 / 20)))
+        return (rng.standard_normal(n) * level).astype(np.float32)
+    fade = int(10 * sr / 1000)
+    tile = min(len(tail), int(0.5 * sr))
+    out = np.zeros(n + tile, np.float32)
+    ramp = np.linspace(0.0, 1.0, fade, dtype=np.float32)
+    pos = 0
+    while pos < n:
+        start = int(rng.integers(0, len(tail) - tile + 1))
+        seg = tail[start: start + tile].copy()
+        if rng.random() < 0.5:
+            seg = seg[::-1]
+        if pos > 0 and tile > fade:
+            seg[:fade] *= ramp
+            out[pos - fade: pos] *= ramp[::-1]
+            out[pos - fade: pos - fade + tile] += seg
+            pos += tile - fade
+        else:
+            out[pos: pos + tile] = seg
+            pos += tile
+    return out[:n]
+
+
+def cut_window(x: np.ndarray, end_s: float, t_off: float, cut_s: float | None, rng: np.random.Generator) -> np.ndarray:
+    """8 s ending at ``end_s``: zeros before the recording, A's room tone after ``cut_s`` (the next
+    speaker's onset) or after the end of the recording."""
+    if cut_s is not None:
+        x = x[: int(round(cut_s * SR))]
+    n = int(WIN_S * SR)
+    end = int(round(end_s * SR))
+    start = end - n
+    out = np.zeros(n, np.float32)
+    s0, s1 = max(start, 0), min(end, len(x))
+    if s1 > s0:
+        out[s0 - start: s1 - start] = x[s0:s1]
+    if end > len(x):
+        pad_from = max(len(x), start)
+        out[pad_from - start:] = room_tone(x, SR, t_off, end - pad_from, rng)
+    return out
+
+
+# --- data -------------------------------------------------------------------------------------------
+
+def read_manifest(path: Path) -> list[dict]:
+    rows = []
+    with open(path, newline="") as f:
+        for r in csv.DictReader(f):
+            rows.append(dict(meeting=r["meeting"], speaker=r["speaker"], channel=int(r["channel"]), t=float(r["t"]),
+                             label=int(r["label"] == "complete"),
+                             gap_s=float(r["gap_s"]) if r["gap_s"] else math.nan,
+                             b_onset_s=float(r["b_onset_s"]) if r["b_onset_s"] else math.nan))
+    return rows
+
+
+def headset_path(audio_dir: Path, meeting: str, channel: int) -> Path:
+    return audio_dir / meeting / f"{meeting}.Headset-{channel}.wav"
+
+
+def fetch_audio(audio_dir: Path, rows: list[dict], retries: int = 4) -> None:
+    need = sorted({(r["meeting"], r["channel"]) for r in rows})
+    missing = [(m, c) for m, c in need if not headset_path(audio_dir, m, c).exists()]
+    if not missing:
+        return
+    print(f"downloading {len(missing)} headset files (~40 MB each) from {MIRROR} into {audio_dir}", flush=True)
+    for m, c in missing:
+        url = f"{MIRROR}/{m}/audio/{m}.Headset-{c}.wav"
+        dest = headset_path(audio_dir, m, c)
+        dest.parent.mkdir(parents=True, exist_ok=True)
+        for attempt in range(retries):
+            try:
+                tmp = dest.with_suffix(".part")
+                with urllib.request.urlopen(url, timeout=120) as r, open(tmp, "wb") as f:
+                    size = int(r.headers.get("Content-Length", "0"))
+                    while True:
+                        chunk = r.read(1 << 20)
+                        if not chunk:
+                            break
+                        f.write(chunk)
+                if size and tmp.stat().st_size != size:
+                    raise OSError(f"size mismatch {tmp.stat().st_size} != {size}")
+                tmp.replace(dest)
+                print(f"  got {dest.name}", flush=True)
+                break
+            except Exception as e:  # noqa: BLE001
+                if attempt == retries - 1:
+                    raise
+                print(f"  retry {dest.name}: {e}", flush=True)
+                time.sleep(5 * (attempt + 1))
+
+
+# --- models -----------------------------------------------------------------------------------------
+
+class Models:
+    def __init__(self, encoder: Path, heads: Path, threads: int):
+        import onnxruntime as ort
+        so = ort.SessionOptions()
+        so.intra_op_num_threads = threads
+        so.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL
+        self.enc = ort.InferenceSession(str(encoder), sess_options=so, providers=["CPUExecutionProvider"])
+        self.heads = ort.InferenceSession(str(heads), sess_options=so, providers=["CPUExecutionProvider"])
+        self.enc_in = self.enc.get_inputs()[0].name
+        outs = [o.name for o in self.enc.get_outputs()]
+        self.enc_prob, self.enc_pooled = outs[0], outs[-1]
+
+    def score(self, feats: np.ndarray, taus: np.ndarray) -> tuple[np.ndarray, np.ndarray]:
+        """feats (B, 80, 800), taus (B,) -> (p_end of the heads (B,), p of the frozen model (B,))."""
+        prob, pooled = self.enc.run([self.enc_prob, self.enc_pooled], {self.enc_in: feats.astype(np.float32)})
+        m20 = np.stack([mel20(f) for f in feats])
+        tau_feat = np.stack([taus / 2.5, np.log1p(taus), np.zeros_like(taus)], axis=1).astype(np.float32)
+        p = self.heads.run(None, {"pooled": pooled.astype(np.float32), "mel20": m20, "tau_feat": tau_feat})[0]
+        return p[:, 0].astype(np.float64), prob.reshape(-1).astype(np.float64)
+
+
+def score_split(rows: list[dict], taus: tuple, models: Models, audio_dir: Path, seed: int, batch: int = 16, tag: str = "") -> dict:
+    """Per silence and tau: P(complete) of the heads and of the frozen model (nan where no window)."""
+    rng = np.random.default_rng(seed)
+    P = np.full((len(rows), len(taus)), np.nan)
+    PB = np.full((len(rows), len(taus)), np.nan)
+    import soundfile as sf
+    by_file: dict[tuple, list[int]] = {}
+    for i, r in enumerate(rows):
+        by_file.setdefault((r["meeting"], r["channel"]), []).append(i)
+    jobs = []        # (row index, tau index, window)
+    t0, done, total = time.time(), 0, sum(len(v) for v in by_file.values())
+
+    def flush():
+        if not jobs:
+            return
+        feats = np.stack([whisper_logmel(w) for _, _, w in jobs])
+        p, pb = models.score(feats, np.array([taus[k] for _, k, _ in jobs], np.float64))
+        for (i, k, _), a, b in zip(jobs, p, pb):
+            P[i, k], PB[i, k] = a, b
+        jobs.clear()
+
+    for (meeting, channel), idx in by_file.items():
+        x, sr = sf.read(headset_path(audio_dir, meeting, channel), dtype="float32")
+        if sr != SR:
+            raise ValueError(f"{meeting} headset {channel}: {sr} Hz, expected 16 kHz")
+        if x.ndim > 1:
+            x = x[:, 0]
+        for i in idx:
+            r = rows[i]
+            for k, tau in enumerate(taus):
+                if not r["label"] and not (tau < r["gap_s"]):
+                    continue                                   # the pause ended before this offset
+                cut = r["t"] + r["b_onset_s"] if (r["label"] and not math.isnan(r["b_onset_s"]) and tau > r["b_onset_s"]) else None
+                jobs.append((i, k, cut_window(x, r["t"] + tau, r["t"], cut, rng)))
+                if len(jobs) >= batch:
+                    flush()
+        done += len(idx)
+        print(f"  {tag}{done}/{total} silences, {time.time() - t0:.0f} s", end="\r", file=sys.stderr, flush=True)
+    flush()
+    print(file=sys.stderr)
+    return dict(p=P, pb=PB, label=np.array([r["label"] for r in rows]), gap=np.array([r["gap_s"] for r in rows]), taus=np.array(taus))
+
+
+# --- metrics ----------------------------------------------------------------------------------------
+
+def auc(y: np.ndarray, s: np.ndarray) -> float:
+    """ROC AUC by the rank statistic (ties counted half)."""
+    y, s = np.asarray(y), np.asarray(s)
+    pos, neg = s[y == 1], s[y == 0]
+    if not len(pos) or not len(neg):
+        return math.nan
+    order = np.argsort(np.concatenate([pos, neg]), kind="mergesort")
+    ranks = np.empty(len(order)); ranks[order] = np.arange(1, len(order) + 1)
+    allv = np.concatenate([pos, neg])
+    # average ranks for ties
+    uniq, inv, counts = np.unique(allv, return_inverse=True, return_counts=True)
+    sums = np.bincount(inv, weights=ranks)
+    ranks = (sums / counts)[inv]
+    return float((ranks[: len(pos)].sum() - len(pos) * (len(pos) + 1) / 2) / (len(pos) * len(neg)))
+
+
+def simulate(p, avail, label, gap, theta, offsets=OFFSETS) -> dict:
+    th = np.asarray(theta)[None, :]
+    hit = avail & (np.nan_to_num(p, nan=-1.0) >= th)
+    first = np.where(hit.any(1), hit.argmax(1), -1)
+    comp, inc = label == 1, label == 0
+    lat = np.full(comp.sum(), TIMEOUT_S)
+    ok = first[comp] >= 0
+    lat[ok] = np.asarray(offsets)[first[comp][ok]]
+    fa = first[inc] >= 0
+    bins = {"<0.5": gap[inc] < 0.5, "0.5-1.5": (gap[inc] >= 0.5) & (gap[inc] < 1.5), ">=1.5": gap[inc] >= 1.5}
+    return dict(theta=tuple(float(t) for t in theta), recall=float(ok.mean()), far=float(fa.mean()),
+                p50=float(np.percentile(lat, 50)), p90=float(np.percentile(lat, 90)), mean=float(lat.mean()),
+                far_by_bin={k: (float(fa[m].mean()) if m.any() else math.nan) for k, m in bins.items()},
+                n_complete=int(comp.sum()), n_incomplete=int(inc.sum()))
+
+
+def fit(p, avail, label, gap, target_far: float, offsets=OFFSETS, grid=np.round(np.arange(0.0, 1.0001, 0.02), 2)) -> dict:
+    """Exhaustive grid over theta per offset with FAR <= target, minimising the mean latency (misses charged 3 s)."""
+    inc, comp = label == 0, label == 1
+    K = len(offsets)
+    hits = [np.stack([avail[:, k] & (np.nan_to_num(p[:, k], nan=-1.0) >= t) for t in grid]) for k in range(K)]
+    best = None
+    for combo in itertools.product(range(len(grid)), repeat=K):
+        h = np.stack([hits[k][combo[k]] for k in range(K)], axis=1)
+        first = np.where(h.any(1), h.argmax(1), -1)
+        if (first[inc] >= 0).mean() > target_far:
+            continue
+        rec = (first[comp] >= 0).mean()
+        lat = np.where(first[comp] >= 0, np.asarray(offsets)[np.maximum(first[comp], 0)], TIMEOUT_S).mean()
+        key = (lat, -rec)
+        if best is None or key < best[0]:
+            best = (key, tuple(grid[i] for i in combo))
+    theta = best[1] if best else tuple([1.01] * K)
+    return simulate(p, avail, label, gap, theta, offsets)
+
+
+def rule_inputs(sc: dict, offsets=OFFSETS):
+    cols = [int(np.where(np.isclose(sc["taus"], o))[0][0]) for o in offsets]
+    p, pb = sc["p"][:, cols], sc["pb"][:, cols]
+    label, gap = sc["label"], sc["gap"]
+    avail = (label[:, None] == 1) | (np.nan_to_num(gap, nan=np.inf)[:, None] > np.asarray(offsets)[None, :])
+    avail &= ~np.isnan(p)
+    # the rule is scored on silences that have a decision point before the resumption
+    keep = avail.any(1)
+    return p[keep], pb[keep], avail[keep], label[keep], gap[keep]
+
+
+def rule_row(name: str, r: dict) -> str:
+    th = " / ".join(f"{t:.2f}" if t <= 1 else "-" for t in r["theta"])
+    fb = " / ".join(f"{r['far_by_bin'][k]:.2f}" for k in ("<0.5", "0.5-1.5", ">=1.5"))
+    return f"| {name} | {th} | {r['recall']:.2f} | {r['far']:.3f} | {r['p50']:.2f} s | {r['p90']:.2f} s | {r['mean']:.2f} s | {fb} |"
+
+
+def read_theta_tables(path: Path) -> list[tuple[str, tuple]]:
+    out = []
+    if not path.exists():
+        return out
+    with open(path, newline="") as f:
+        for r in csv.DictReader(f):
+            if r["model"] == "heads":
+                out.append((f"shipped table, {float(r['far_budget']):.0%} budget", tuple(float(t) for t in json.loads(r["theta"].replace("(", "[").replace(")", "]")))))
+    return out
+
+
+# --- main -------------------------------------------------------------------------------------------
+
+def main():
+    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+    ap.add_argument("--audio-dir", default="./ami_audio", help="AMI headset audio, <meeting>/<meeting>.Headset-<n>.wav (downloaded when missing)")
+    ap.add_argument("--no-download", action="store_true")
+    ap.add_argument("--manifest-dir", default=str(HERE), help="directory with ami_dev.csv / ami_test.csv")
+    ap.add_argument("--encoder", default=str(HERE.parent / "smart-turn-v3.2-encoder-pooled.onnx"))
+    ap.add_argument("--heads", default=str(HERE.parent / "eot-heads-v32.onnx"))
+    ap.add_argument("--theta-tables", default=str(HERE.parent / "rules" / "theta_tables.csv"))
+    ap.add_argument("--splits", nargs="*", default=["dev", "test"])
+    ap.add_argument("--taus", nargs="*", default=None, help="silence offsets to score (default: the rule offsets 0.2 0.5 1.2; 'all' = the trained grid)")
+    ap.add_argument("--budgets", nargs="*", type=float, default=[0.05, 0.10, 0.15])
+    ap.add_argument("--no-fit", action="store_true", help="skip the threshold fit on dev (score the shipped tables only)")
+    ap.add_argument("--threads", type=int, default=1)
+    ap.add_argument("--seed", type=int, default=0)
+    ap.add_argument("--cache", default="./ami_eval_scores.npz")
+    ap.add_argument("--out", default="./ami_eval.md")
+    a = ap.parse_args()
+
+    taus = TAU_GRID if (a.taus and a.taus[0] == "all") else tuple(float(t) for t in (a.taus or OFFSETS))
+    for o in OFFSETS:
+        if not any(np.isclose(o, taus)):
+            sys.exit(f"--taus must include the rule offsets {OFFSETS}")
+    audio_dir = Path(a.audio_dir)
+    rows = {sp: read_manifest(Path(a.manifest_dir) / f"ami_{sp}.csv") for sp in a.splits}
+    if not a.no_download:
+        fetch_audio(audio_dir, [r for rs in rows.values() for r in rs])
+
+    cache = dict(np.load(a.cache, allow_pickle=True)) if os.path.exists(a.cache) else {}
+    scores = {}
+    models = None
+    for sp in a.splits:
+        key = f"{sp}:{','.join(f'{t:g}' for t in taus)}:{a.seed}"
+        if key in cache:
+            scores[sp] = cache[key].item()
+            print(f"{sp}: scores from {a.cache}", file=sys.stderr)
+            continue
+        if models is None:
+            models = Models(Path(a.encoder), Path(a.heads), a.threads)
+        print(f"{sp}: scoring {len(rows[sp])} silences at tau = {taus}", file=sys.stderr, flush=True)
+        scores[sp] = score_split(rows[sp], taus, models, audio_dir, a.seed, tag=f"{sp} ")
+        cache[key] = np.array(scores[sp], dtype=object)
+        np.savez(a.cache, **cache)
+
+    lines = [f"# AMI end-of-turn evaluation ({', '.join(a.splits)}; thresholds {'shipped' if a.no_fit else 'shipped and re-fitted on dev'})", ""]
+    for sp in a.splits:
+        sc = scores[sp]
+        lines += [f"## AMI {sp}: AUC of P(complete) per silence offset", "",
+                  "| tau (s) | n complete / incomplete | AUC heads | AUC frozen Smart Turn v3.2 | mean P on incompletes: heads / frozen |", "|---|---|---|---|---|"]
+        for k, tau in enumerate(sc["taus"]):
+            ok = ~np.isnan(sc["p"][:, k])
+            y, p, pb = sc["label"][ok], sc["p"][ok, k], sc["pb"][ok, k]
+            lines.append(f"| {tau:g} | {int((y == 1).sum())} / {int((y == 0).sum())} | {auc(y, p):.3f} | {auc(y, pb):.3f} | "
+                         f"{p[y == 0].mean():.3f} / {pb[y == 0].mean():.3f} |")
+        lines.append("")
+    test_sp = "test" if "test" in scores else a.splits[-1]
+    P, PB, A, L, G = rule_inputs(scores[test_sp])
+    hdr = ["| rule | theta at 0.2 / 0.5 / 1.2 s | recall | FAR | p50 | p90 | mean latency | FAR by pause length <0.5 / 0.5-1.5 / >=1.5 s |",
+           "|---|---|---|---|---|---|---|---|"]
+    lines += [f"## Decision rule on AMI {test_sp} ({int((L == 1).sum())} complete / {int((L == 0).sum())} incomplete silences with a decision point; "
+              f"misses charged {TIMEOUT_S:g} s)", "", "**Heads (this release)**", ""] + hdr
+    lines.append(rule_row("one-shot at the VAD stop, theta 0.5 (stock rule)", simulate(P, A, L, G, (0.5, 1.01, 1.01))))
+    for name, th in read_theta_tables(Path(a.theta_tables)):
+        lines.append(rule_row(name, simulate(P, A, L, G, th)))
+    fitted = {}
+    if not a.no_fit and "dev" in scores:
+        Pd, PBd, Ad, Ld, Gd = rule_inputs(scores["dev"])
+        for b in a.budgets:
+            t_fit = time.time()
+            fitted[b] = (fit(Pd, Ad, Ld, Gd, b), fit(PBd, Ad, Ld, Gd, b))
+            print(f"fit at {b:.0%}: {time.time() - t_fit:.0f} s", file=sys.stderr)
+            lines.append(rule_row(f"re-fitted on dev, {b:.0%} budget", simulate(P, A, L, G, fitted[b][0]["theta"])))
+    lines += ["", "**Frozen Smart Turn v3.2 (baseline, same protocol)**", ""] + hdr
+    lines.append(rule_row("one-shot at the VAD stop, theta 0.5 (stock rule)", simulate(PB, A, L, G, (0.5, 1.01, 1.01))))
+    for b, (_, rb) in fitted.items():
+        lines.append(rule_row(f"re-fitted on dev, {b:.0%} budget", simulate(PB, A, L, G, rb["theta"])))
+    lines += ["", "**No model**", ""] + hdr
+    lines.append(rule_row("silence only, commit at 1.2 s", simulate(np.ones_like(P), A, L, G, (1.01, 1.01, 0.0))))
+    lines += ["", f"Scores cached in `{a.cache}`; audio from {MIRROR} (AMI Meeting Corpus, CC BY 4.0)."]
+    Path(a.out).write_text("\n".join(lines) + "\n")
+    print("\n".join(lines))
+
+
+if __name__ == "__main__":
+    main()

eval/ami_eval_reference.md

@@ -0,0 +1,48 @@
+# AMI end-of-turn evaluation (dev, test; thresholds shipped and re-fitted on dev)
+
+## AMI dev: AUC of P(complete) per silence offset
+
+| tau (s) | n complete / incomplete | AUC heads | AUC frozen Smart Turn v3.2 | mean P on incompletes: heads / frozen |
+|---|---|---|---|---|
+| 0.2 | 734 / 3195 | 0.748 | 0.675 | 0.251 / 0.299 |
+| 0.5 | 734 / 1827 | 0.722 | 0.640 | 0.398 / 0.414 |
+| 1.2 | 734 / 624 | 0.698 | 0.616 | 0.727 / 0.450 |
+
+## AMI test: AUC of P(complete) per silence offset
+
+| tau (s) | n complete / incomplete | AUC heads | AUC frozen Smart Turn v3.2 | mean P on incompletes: heads / frozen |
+|---|---|---|---|---|
+| 0.2 | 511 / 3820 | 0.775 | 0.671 | 0.256 / 0.309 |
+| 0.5 | 511 / 2256 | 0.776 | 0.638 | 0.357 / 0.427 |
+| 1.2 | 511 / 688 | 0.738 | 0.577 | 0.713 / 0.533 |
+
+## Decision rule on AMI test (511 complete / 3820 incomplete silences with a decision point; misses charged 3 s)
+
+**Heads (this release)**
+
+| rule | theta at 0.2 / 0.5 / 1.2 s | recall | FAR | p50 | p90 | mean latency | FAR by pause length <0.5 / 0.5-1.5 / >=1.5 s |
+|---|---|---|---|---|---|---|---|
+| one-shot at the VAD stop, theta 0.5 (stock rule) | 0.50 / - / - | 0.64 | 0.254 | 0.20 s | 3.00 s | 1.21 s | 0.19 / 0.24 / 0.46 |
+| shipped table, 5% budget | 0.94 / 0.94 / 0.94 | 0.47 | 0.046 | 3.00 s | 3.00 s | 2.05 s | 0.00 / 0.03 / 0.21 |
+| shipped table, 10% budget | 0.92 / 0.98 / 0.86 | 0.79 | 0.094 | 1.20 s | 3.00 s | 1.49 s | 0.01 / 0.04 / 0.50 |
+| shipped table, 15% budget | 0.98 / 0.88 / 0.80 | 0.86 | 0.133 | 1.20 s | 3.00 s | 1.21 s | 0.00 / 0.09 / 0.61 |
+| re-fitted on dev, 5% budget | 0.94 / 0.94 / 0.94 | 0.47 | 0.046 | 3.00 s | 3.00 s | 2.05 s | 0.00 / 0.03 / 0.21 |
+| re-fitted on dev, 10% budget | 1.00 / 0.94 / 0.86 | 0.79 | 0.090 | 1.20 s | 3.00 s | 1.49 s | 0.00 / 0.04 / 0.50 |
+| re-fitted on dev, 15% budget | 0.98 / 0.88 / 0.80 | 0.86 | 0.133 | 1.20 s | 3.00 s | 1.21 s | 0.00 / 0.09 / 0.61 |
+
+**Frozen Smart Turn v3.2 (baseline, same protocol)**
+
+| rule | theta at 0.2 / 0.5 / 1.2 s | recall | FAR | p50 | p90 | mean latency | FAR by pause length <0.5 / 0.5-1.5 / >=1.5 s |
+|---|---|---|---|---|---|---|---|
+| one-shot at the VAD stop, theta 0.5 (stock rule) | 0.50 / - / - | 0.51 | 0.303 | 0.20 s | 3.00 s | 1.56 s | 0.23 / 0.30 / 0.45 |
+| re-fitted on dev, 5% budget | 1.00 / 1.00 / 0.96 | 0.37 | 0.056 | 3.00 s | 3.00 s | 2.34 s | 0.00 / 0.02 / 0.31 |
+| re-fitted on dev, 10% budget | 1.00 / 1.00 / 0.26 | 0.71 | 0.106 | 1.20 s | 3.00 s | 1.72 s | 0.00 / 0.04 / 0.61 |
+| re-fitted on dev, 15% budget | 1.00 / 1.00 / 0.02 | 0.87 | 0.137 | 1.20 s | 3.00 s | 1.44 s | 0.00 / 0.05 / 0.78 |
+
+**No model**
+
+| rule | theta at 0.2 / 0.5 / 1.2 s | recall | FAR | p50 | p90 | mean latency | FAR by pause length <0.5 / 0.5-1.5 / >=1.5 s |
+|---|---|---|---|---|---|---|---|
+| silence only, commit at 1.2 s | - / - / 0.00 | 1.00 | 0.180 | 1.20 s | 1.20 s | 1.20 s | 0.00 / 0.08 / 1.00 |
+
+Scores cached in `./ami_eval_scores.npz`; audio from https://groups.inf.ed.ac.uk/ami/AMICorpusMirror/amicorpus (AMI Meeting Corpus, CC BY 4.0).

rules/eager_tables.csv

@@ -0,0 +1,9 @@
+commit,budget,theta_e,theta_c
+far10,0.2,"(0.92, 0.84, 0.68)","(0.92, 0.98, 0.86)"
+far10,0.3,"(0.92, 0.54, 0.2)","(0.92, 0.98, 0.86)"
+far10,0.4,"(0.74, 0.24, 0.02)","(0.92, 0.98, 0.86)"
+far10,0.5,"(0.42, 0.1, 0.02)","(0.92, 0.98, 0.86)"
+far15,0.2,"(0.92, 0.84, 0.68)","(0.98, 0.88, 0.8)"
+far15,0.3,"(0.84, 0.56, 0.42)","(0.98, 0.88, 0.8)"
+far15,0.4,"(0.72, 0.24, 0.04)","(0.98, 0.88, 0.8)"
+far15,0.5,"(0.42, 0.1, 0.02)","(0.98, 0.88, 0.8)"

rules/theta_tables.csv

@@ -0,0 +1,7 @@
+model,far_budget,theta,test_recall,test_far,test_mean,test_p90,test_far_ge15
+heads,0.05,"(0.94, 0.94, 0.94)",0.472,0.046,2.05,3.0,0.21
+frozen_v32,0.05,"(1.0, 1.0, 0.96)",0.368,0.055,2.34,3.0,0.31
+heads,0.1,"(0.92, 0.98, 0.86)",0.796,0.094,1.48,3.0,0.5
+frozen_v32,0.1,"(1.0, 1.0, 0.28)",0.699,0.106,1.74,3.0,0.61
+heads,0.15,"(0.98, 0.88, 0.8)",0.863,0.133,1.21,3.0,0.61
+frozen_v32,0.15,"(1.0, 1.0, 0.02)",0.867,0.137,1.44,3.0,0.78

smart-turn-v3.2-encoder-pooled.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:37e2a4fe22e1c4efaeba4df40141e422a3b96519db9160ae9134962a936ba4ea
+size 32411234