Kuroko: weights, the file that runs them, settings, dev-split predictions, training log and card
879acfa | license: apache-2.0 | |
| language: | |
| - en | |
| pipeline_tag: voice-activity-detection | |
| library_name: pytorch | |
| tags: | |
| - turn-taking | |
| - end-of-turn | |
| - interruption | |
| - backchannel | |
| - turnbench | |
| - audio | |
| # Kuroko | |
| Kuroko is a turn-taking system for two-channel English conversation, built by PatientDesk AI for | |
| [TurnBench](https://turnbench.sesame.com). It decides when a speaker's turn has ended, and whether a sound from the | |
| other speaker is an interruption or an "mhm". | |
| **Write-up:** <https://spero.patientdesk.ai/blog/kuroko> | |
| This repository holds Kuroko's weights, the one file needed to run them, its settings and its predictions for | |
| TurnBench's development split. The training code and the training data are not public, and neither are the | |
| predictions for the test split, at the benchmark's request. | |
| ## Run the model | |
| The model is a 14.9M-parameter causal transformer. It hears both speakers and answers 25 times a second for each | |
| of them, from the past only. | |
| ``` | |
| pip install torch numpy scipy safetensors soundfile huggingface_hub | |
| ``` | |
| ```python | |
| import sys | |
| import soundfile as sf | |
| from huggingface_hub import snapshot_download | |
| path = snapshot_download("cloud0day3/kuroko") | |
| sys.path.insert(0, path) | |
| import kuroko | |
| model = kuroko.load(path) | |
| audio, rate = sf.read("conversation.wav", dtype="int16") # (samples, 2): one speaker on each channel | |
| answers = kuroko.answers(model, kuroko.features(audio, rate)) # (2, 11, n): speaker, answer, 25 a second | |
| claims = answers[:, kuroko.ANSWERS.index("claim")] | |
| ``` | |
| `answers[s, a, k]` is a probability for speaker `s` that has heard the audio up to `(k + 1) / 25` seconds. The 11 | |
| answers are named in `kuroko.ANSWERS`: | |
| | Answer | Meaning | | |
| |---|---| | |
| | `voice` | The speaker is making a sound | | |
| | `claim` | That sound claims the floor. This is the answer the events are read from | | |
| | `hold` | The speaker holds the floor, pauses included | | |
| | 8 more | Whether the speaker, and whether the partner, will talk for half or more of 0 to 0.2 s, 0.2 to 0.6 s, 0.6 to 1.2 s and 1.2 to 2 s ahead | | |
| The model was trained on 16-bit recordings at 48 kHz with one speaker per channel, brought to 16 kHz as | |
| `kuroko.features` does it. Read that way, on the GPU the predictions in this repository were made on, the released | |
| files give the answers behind them on all 38 dev conversations. On a CPU an answer differs by less than 0.001, | |
| which can move a threshold crossing by one answer, 40 ms. Audio of more bits read as floats moves an answer by up | |
| to 0.02. | |
| Only `claim` has been measured against annotations. Offline, a recorded 12.6-minute conversation is answered in 11 s | |
| on four CPU threads, in 40-second stretches. The model has not been run as a live stream. | |
| ## From answers to events | |
| The model is one of three parts, and this repository has the code of the model only. | |
| 1. **Detector.** [Silero VAD](https://github.com/snakers4/silero-vad) 6.2.3 runs on each channel in 32 ms steps and | |
| marks stretches of sound. It is certain that a stretch has ended 0.26 s after its last sound. | |
| 2. **Rules.** The timing rules p99lab published for [turn-1-mini](https://github.com/P99Lab/turn-1-mini), without that | |
| system's learned parts, with the numbers chosen again on the dev split (`policy.json`). Every stop opens a | |
| candidate end of turn, which is committed once the partner has taken over or 2.45 s have passed, unless the | |
| speaker is back. | |
| 3. **Model.** `claim` commits an interruption when it reaches 0.9, read from 0.32 s after a sound begins, before | |
| the sound has lasted the 1.3 s the length rule needs. It also tells the end-of-turn rules that the partner has | |
| taken over, when it reaches 0.5 for the partner's sound. | |
| The write-up gives the rules in full. | |
| ## Results | |
| TurnBench development split: 38 conversations, 1,904 ends of turn, 347 interruptions, scored by the benchmark's | |
| scorer. The settings were chosen on this split within a false-alarm budget of 8%. "Held out" is the same choice made | |
| on half of the conversations and scored on the other half, over 500 random halves. Kuroko was submitted to TurnBench | |
| on November 13, 2026. The test split is scored by the benchmark's maintainers, and its results are theirs to | |
| publish on the [leaderboard](https://turnbench.sesame.com). | |
| | | Recall | False-alarm rate | Median delay | Held-out recall | Held-out false-alarm rate | | |
| |---|---|---|---|---|---| | |
| | End of turn | 0.987 (1,879 of 1,904) | 0.079 (84 of 1,063) | 630 ms | 0.983 | 0.082 | | |
| | Interruption | 1.000 (347 of 347) | 0.073 (272 of 3,733) | 545 ms | 0.988 | 0.061 | | |
| To check these numbers, drop `predictions-dev.json` on the benchmark's [dev scorer](https://turnbench.sesame.com/dev). | |
| It scores the file in the browser. | |
| What each part adds, on the same split (recall / false-alarm rate / median delay): | |
| | System | End of turn | Interruption | | |
| |---|---|---| | |
| | Rules, turn-1-mini's published settings, on our detector | 0.968 / 0.051 / 879 ms | 0.899 / 0.026 / 1,579 ms | | |
| | Rules, settings chosen on dev | 0.983 / 0.080 / 692 ms | 0.977 / 0.076 / 1,078 ms | | |
| | **Kuroko**: rules and the model | 0.987 / 0.079 / 630 ms | 1.000 / 0.073 / 545 ms | | |
| ## Training data | |
| 1,025 hours of two-speaker conversation: 786 hours of [Open Yap 1K](https://theagenticdatacompany.com/open-yap-1k) | |
| and about 240 hours of PatientDesk's own recordings, made by contributors who agreed to this use. The labels come | |
| from a timing rule applied in hindsight. None of the data is in this repository. | |
| No audio and no label from TurnBench's dev or test split was trained on, and otoSpeech was not used. The dev split | |
| chose the settings, checked the label rule and compared the systems above. | |
| ## Files | |
| | File | Contents | | |
| |---|---| | |
| | `model.safetensors` | The weights, kept at step 12,000 | | |
| | `config.json` | The model's shape | | |
| | `kuroko.py` | The model, its features and how a conversation is answered. Nothing of the training | | |
| | `policy.json` | The settings of the rules, in seconds and answer levels: `eot` for ends of turn, `int` for interruptions | | |
| | `predictions-dev.json` | Kuroko's events for the 38 dev conversations, in TurnBench's submission format | | |
| | `training-log.jsonl` | Training loss and the loss of each answer on held-out conversations, every 600 steps | | |
| Each block of `policy.json` is a full settings object, and each rule reads only part of it. The end-of-turn rules | |
| read `wait_s`, `partner_s`, `deadline_s`, `resumed_s`, `held_s`, `confirm_s`, `partner_claim` and `claim_from_s` from | |
| `eot`. The interruption rule reads `sustained_s`, `claim_fire`, `claim_from_s` and `claim_rise` from `int`. | |
| `known_s` and `begun_s` are the detector's delays. | |
| ## Limits | |
| - These are development numbers on 38 conversations, with settings chosen on them. Zero missed interruptions is an | |
| in-sample number; held out, recall is 0.988. | |
| - TurnBench records two people, so the end-of-turn rules can wait for the second person to start talking. A voice | |
| agent is the second speaker, and in an agent these rules would be left with the 2.45 s timer. | |
| - A turn's end can be committed a second time, and an interruption is committed for every sound taken as a claim. | |
| The scorer counts at most one false alarm per pause, and looks for interruptions only near annotated interruptions | |
| and backchannels. Kept at least 1 s apart, the events score 0.975 at 0.074 on ends of turn and 1.000 at 0.070 on | |
| interruptions. | |
| - Studio audio with one clean channel per speaker, English only. Not measured on phone audio, with echo or as a live | |
| stream. | |
| The write-up has the rest: the label rule and its check against the annotators, the ablations, the causality test | |
| and the credits. | |
| ## License | |
| `model.safetensors`, `config.json` and `kuroko.py` are released under Apache-2.0. | |
| ## Credits | |
| TurnBench is by Sesame AI Labs, with the evaluation corpus from Mundo AI and the otoSpeech training set from Oto | |
| ([Jiang et al., 2026](https://arxiv.org/abs/2608.25218)). The timing rules are p99lab's. The detector is Silero VAD. | |
| The model's future-activity targets follow voice activity projection | |
| ([Ekstedt and Skantze, 2022](https://arxiv.org/abs/2205.09812)). | |
| Training data: The Agentic Data Company. Open Yap 1K: channel-separated English natural two-speaker conversations. | |
| 2026. Version 1.0. <https://theagenticdatacompany.com/open-yap-1k> | |