Blame for README.md

---
license: apache-2.0
language:
- en
pipeline_tag: voice-activity-detection
library_name: pytorch
tags:
- turn-taking
- end-of-turn
- interruption
- backchannel
- turnbench
- audio
---

# Kuroko

Kuroko is a turn-taking system for two-channel English conversation, built by PatientDesk AI for
[TurnBench](https://turnbench.sesame.com). It decides when a speaker's turn has ended, and whether a sound from the
other speaker is an interruption or an "mhm".

**Write-up:** <https://spero.patientdesk.ai/blog/kuroko>

This repository holds Kuroko's weights, the one file needed to run them, its settings and its predictions for
TurnBench's development split. The training code and the training data are not public, and neither are the
predictions for the test split, at the benchmark's request.

## Run the model

The model is a 14.9M-parameter causal transformer. It hears both speakers and answers 25 times a second for each
of them, from the past only.

```
pip install torch numpy scipy safetensors soundfile huggingface_hub
```

```python
import sys

import soundfile as sf
from huggingface_hub import snapshot_download

path = snapshot_download("cloud0day3/kuroko")
sys.path.insert(0, path)
import kuroko

model = kuroko.load(path)
audio, rate = sf.read("conversation.wav", dtype="int16")  # (samples, 2): one speaker on each channel
answers = kuroko.answers(model, kuroko.features(audio, rate))  # (2, 11, n): speaker, answer, 25 a second
claims = answers[:, kuroko.ANSWERS.index("claim")]
```

`answers[s, a, k]` is a probability for speaker `s` that has heard the audio up to `(k + 1) / 25` seconds. The 11
answers are named in `kuroko.ANSWERS`:

| Answer | Meaning |
|---|---|
| `voice` | The speaker is making a sound |
| `claim` | That sound claims the floor. This is the answer the events are read from |
| `hold` | The speaker holds the floor, pauses included |
| 8 more | Whether the speaker, and whether the partner, will talk for half or more of 0 to 0.2 s, 0.2 to 0.6 s, 0.6 to 1.2 s and 1.2 to 2 s ahead |

The model was trained on 16-bit recordings at 48 kHz with one speaker per channel, brought to 16 kHz as
`kuroko.features` does it. Read that way, on the GPU the predictions in this repository were made on, the released
files give the answers behind them on all 38 dev conversations. On a CPU an answer differs by less than 0.001,
which can move a threshold crossing by one answer, 40 ms. Audio of more bits read as floats moves an answer by up
to 0.02.

Only `claim` has been measured against annotations. Offline, a recorded 12.6-minute conversation is answered in 11 s
on four CPU threads, in 40-second stretches. The model has not been run as a live stream.

## From answers to events

The model is one of three parts, and this repository has the code of the model only.

1. **Detector.** [Silero VAD](https://github.com/snakers4/silero-vad) 6.2.3 runs on each channel in 32 ms steps and
   marks stretches of sound. It is certain that a stretch has ended 0.26 s after its last sound.
2. **Rules.** The timing rules p99lab published for [turn-1-mini](https://github.com/P99Lab/turn-1-mini), without that
   system's learned parts, with the numbers chosen again on the dev split (`policy.json`). Every stop opens a
   candidate end of turn, which is committed once the partner has taken over or 2.45 s have passed, unless the
   speaker is back.
3. **Model.** `claim` commits an interruption when it reaches 0.9, read from 0.32 s after a sound begins, before
   the sound has lasted the 1.3 s the length rule needs. It also tells the end-of-turn rules that the partner has
   taken over, when it reaches 0.5 for the partner's sound.

The write-up gives the rules in full.

## Results

TurnBench development split: 38 conversations, 1,904 ends of turn, 347 interruptions, scored by the benchmark's
scorer. The settings were chosen on this split within a false-alarm budget of 8%. "Held out" is the same choice made
on half of the conversations and scored on the other half, over 500 random halves. Kuroko was submitted to TurnBench
on November 13, 2026. The test split is scored by the benchmark's maintainers, and its results are theirs to
publish on the [leaderboard](https://turnbench.sesame.com).

| | Recall | False-alarm rate | Median delay | Held-out recall | Held-out false-alarm rate |
|---|---|---|---|---|---|
| End of turn | 0.987 (1,879 of 1,904) | 0.079 (84 of 1,063) | 630 ms | 0.983 | 0.082 |
| Interruption | 1.000 (347 of 347) | 0.073 (272 of 3,733) | 545 ms | 0.988 | 0.061 |

To check these numbers, drop `predictions-dev.json` on the benchmark's [dev scorer](https://turnbench.sesame.com/dev).
It scores the file in the browser.

What each part adds, on the same split (recall / false-alarm rate / median delay):

| System | End of turn | Interruption |
|---|---|---|
| Rules, turn-1-mini's published settings, on our detector | 0.968 / 0.051 / 879 ms | 0.899 / 0.026 / 1,579 ms |
| Rules, settings chosen on dev | 0.983 / 0.080 / 692 ms | 0.977 / 0.076 / 1,078 ms |
| **Kuroko**: rules and the model | 0.987 / 0.079 / 630 ms | 1.000 / 0.073 / 545 ms |

## Training data

1,025 hours of two-speaker conversation: 786 hours of [Open Yap 1K](https://theagenticdatacompany.com/open-yap-1k)
and about 240 hours of PatientDesk's own recordings, made by contributors who agreed to this use. The labels come
from a timing rule applied in hindsight. None of the data is in this repository.

No audio and no label from TurnBench's dev or test split was trained on, and otoSpeech was not used. The dev split
chose the settings, checked the label rule and compared the systems above.

## Files

| File | Contents |
|---|---|
| `model.safetensors` | The weights, kept at step 12,000 |
| `config.json` | The model's shape |
| `kuroko.py` | The model, its features and how a conversation is answered. Nothing of the training |
| `policy.json` | The settings of the rules, in seconds and answer levels: `eot` for ends of turn, `int` for interruptions |
| `predictions-dev.json` | Kuroko's events for the 38 dev conversations, in TurnBench's submission format |
| `training-log.jsonl` | Training loss and the loss of each answer on held-out conversations, every 600 steps |

Each block of `policy.json` is a full settings object, and each rule reads only part of it. The end-of-turn rules
read `wait_s`, `partner_s`, `deadline_s`, `resumed_s`, `held_s`, `confirm_s`, `partner_claim` and `claim_from_s` from
`eot`. The interruption rule reads `sustained_s`, `claim_fire`, `claim_from_s` and `claim_rise` from `int`.
`known_s` and `begun_s` are the detector's delays.

## Limits

- These are development numbers on 38 conversations, with settings chosen on them. Zero missed interruptions is an
  in-sample number; held out, recall is 0.988.
- TurnBench records two people, so the end-of-turn rules can wait for the second person to start talking. A voice
  agent is the second speaker, and in an agent these rules would be left with the 2.45 s timer.
- A turn's end can be committed a second time, and an interruption is committed for every sound taken as a claim.
  The scorer counts at most one false alarm per pause, and looks for interruptions only near annotated interruptions
  and backchannels. Kept at least 1 s apart, the events score 0.975 at 0.074 on ends of turn and 1.000 at 0.070 on
  interruptions.
- Studio audio with one clean channel per speaker, English only. Not measured on phone audio, with echo or as a live
  stream.

The write-up has the rest: the label rule and its check against the annotators, the ablations, the causality test
and the credits.

## License

`model.safetensors`, `config.json` and `kuroko.py` are released under Apache-2.0.

## Credits

TurnBench is by Sesame AI Labs, with the evaluation corpus from Mundo AI and the otoSpeech training set from Oto
([Jiang et al., 2026](https://arxiv.org/abs/2608.25218)). The timing rules are p99lab's. The detector is Silero VAD.
The model's future-activity targets follow voice activity projection
([Ekstedt and Skantze, 2022](https://arxiv.org/abs/2205.09812)).

Training data: The Agentic Data Company. Open Yap 1K: channel-separated English natural two-speaker conversations.
2026. Version 1.0. <https://theagenticdatacompany.com/open-yap-1k>