Upload folder using huggingface_hub

f7c4e02 verified 2 months ago

6.74 kB

---
license: apache-2.0
library_name: rubato
pipeline_tag: voice-activity-detection
tags:
- turn-taking
- endpointing
- spoken-dialogue
- survival-analysis
- realtime
- rubato
---
# Rubato — turn-taking timing model

**Predict the silence as a distribution, then decide when to speak.**

A 3,137-parameter model that answers one question at 50 Hz: *given
everything heard so far, how much longer will this silence last?*  A dynamic
program turns that distribution into a start/wait decision under an explicit,
tunable asymmetry between talking over someone and answering late.

It decides **when** to talk. It does not decide **what** to say, and it contains
no speech recogniser, no language model and no synthesiser. It is a middle layer
you drop into an existing pipeline.

## Three lines

```python
from rubato import load_pretrained, CostWeights
taker = load_pretrained(hf_repo="NagaYu/rubato-timing", weights=CostWeights.from_seconds_per_collision(2.0))
should_speak = taker.push_audio(chunk).should_speak     # one 20 ms frame of float audio
```

`seconds_per_collision` is the whole configuration surface: *how many seconds of
extra latency is one talk-over worth to you?*  Small values give an eager agent,
large values a patient one, and the sweep between them is the Pareto front below.

Framework adapters: `rubato.integrations.pipecat_processor.RubatoTurnGate` and
`rubato.integrations.livekit_plugin.RubatoTurnDetector`.

Source, benchmark protocol, ablations and sensitivity analyses: **https://github.com/NagaYu/rubato**

## What it was measured to do

![Talk-over rate against response latency: Rubato's frontier lies inside the fixed-threshold frontier](./pareto_maptask.png)

*Both axes are minimised. A fixed threshold can only ever trace the outer curve -- one threshold buys one point. The inset is the band real systems ship in.*

Held out on **5988 silences** from speakers never seen in training (maptask, CC-BY-4.0).

| | |
|---|---|
| covers the fixed-threshold frontier | **86%** of its points |
| covers the semantic-completeness frontier | **100%** of its points |
| latency saved at matched talk-over | **84 ms** vs fixed, 96 ms vs semantic |
| talk-over removed at matched latency | **2.8 pp** vs fixed, 2.1 pp vs semantic |
| hazard calibration (ECE) | 0.0007 |
| CRPS skill over a covariate-free hazard | +0.173 |
| layer cost per 20 ms frame | **0.26 ms** median, 0.49 ms p99 |

At the operating point matched to a 1000 ms threshold: latency **1000 → 772 ms** (95 % CI 707–831), talk-over **6.3% → 6.5%** (95 % CI 5.6%–7.6%). Intervals are a conversation-level cluster bootstrap.

## How it works

1. **Discrete-time hazard.** `h_k = P(partner resumes in frame k+1 | silent
   through k, evidence up to k)`, from a one-hidden-layer network over a causal
   feature vector: a radial-basis expansion of log elapsed silence, turn-so-far
   duration and pause count, transcript completeness (gated by ASR lag, because a
   real recogniser has not delivered the last word yet), terminal prosody where
   audio exists, acoustic precursor cues such as in-breaths, and a running
   per-partner posterior.
2. **Optimal stopping.** A backward dynamic program over the forecast survival
   curve minimises `alpha * P(collision) + beta * latency`. Being pre-empted --
   the human carries on while the agent is still silent -- costs nothing, which
   is what produces the human-like behaviour: when a resumption looks likely,
   waiting is nearly free; when the floor is clearly open, the agent can start at
   zero gap.
3. **Online entrainment.** Conjugate posteriors over each partner's pause and gap
   distributions, and an EMA of their speech rate, shrinking to a population
   prior so the first thirty seconds are never worse than not adapting.
## Inputs, outputs, and what it needs from you
| you provide | per frame |
|---|---|
| 20 ms of mono audio (any rate; 16 kHz assumed) | required |
| the ASR's partial transcript | optional, improves accuracy |
| a partner id | optional, enables entrainment |
| you get back | |
|---|---|
| `should_speak` | the decision |
| `decision.planned_onset_s` | when it currently intends to start |
| `decision.p_overlap_now` | collision probability if it started this instant |
| `prediction.future_hazards` | the full predicted silence distribution |

## Limitations

- **English lexical features.** The temporal and acoustic half transfers; the
  completeness features were fitted on English and will mislead elsewhere.
  Retrain on target-language data.
- **Task-oriented training data.** Trained on the HCRC Map Task Corpus: two people collaborating on a route-following task.  Open-domain conversation has
  longer, more variable gaps; expect to retune `seconds_per_collision`.
- **Two-party only.** Multi-party floor management is a different problem.
- **The offline protocol makes an assumption.** On a floor transfer we assume the
  floor would have stayed open had the agent not spoken. The alternative is
  worse (informative censoring); both are reported in the repository README.
- **It is a timing layer, not a turn-taking oracle.** It cannot know that you
  paused because you were about to say something difficult.

## Responsible use

Better timing makes an assistant less irritating. It also makes a synthetic voice
harder to distinguish from a person, and that is a use this model is **not** for.

- **Disclose that the system is an AI.** This model is released on the assumption
  that the deployment does so, in the modality the user is actually attending to
  -- spoken, not buried in a settings page.
- **Do not use it to impersonate anyone**, to make an automated caller pass as
  human, or in any deployment whose value depends on the person not knowing.
  Several jurisdictions require disclosure for automated calls; treat that as a
  floor, not a target.
- **Do not use it to time interruptions strategically** -- to cut a person off at
  a moment calculated to stop them finishing a thought. The asymmetric cost makes
  that easy to configure, which is exactly why it is named here.
- The entrainment state is a behavioural profile of an individual. Keep it in
  session scope unless you have a reason and consent to persist it.

## Training data

**maptask** (CC-BY-4.0). Anderson et al. (1991), The HCRC Map Task Corpus. Language and Speech 34(4). Annotations (c) 2007 HCRC, Univ. of Edinburgh & Univ. of Glasgow. CC BY 4.0. https://groups.inf.ed.ac.uk/maptask/

No audio was redistributed in building this model.

## Citation

```bibtex
@software{rubato,
  title  = {Rubato: predicting silence distributions for spoken-dialogue turn-taking},
  year   = {2026},
  url    = {https://huggingface.co/NagaYu/rubato-timing}
}
```

	---
	license: apache-2.0
	library_name: rubato
	pipeline_tag: voice-activity-detection
	tags:
	- turn-taking
	- endpointing
	- spoken-dialogue
	- survival-analysis
	- realtime
	- rubato
	---

	# Rubato — turn-taking timing model

	Predict the silence as a distribution, then decide when to speak.

	A 3,137-parameter model that answers one question at 50 Hz: *given
	everything heard so far, how much longer will this silence last?* A dynamic
	program turns that distribution into a start/wait decision under an explicit,
	tunable asymmetry between talking over someone and answering late.

	It decides when to talk. It does not decide what to say, and it contains
	no speech recogniser, no language model and no synthesiser. It is a middle layer
	you drop into an existing pipeline.

	## Three lines

	```python
	from rubato import load_pretrained, CostWeights

	taker = load_pretrained(hf_repo="NagaYu/rubato-timing", weights=CostWeights.from_seconds_per_collision(2.0))
	should_speak = taker.push_audio(chunk).should_speak # one 20 ms frame of float audio
	```

	`seconds_per_collision` is the whole configuration surface: *how many seconds of
	extra latency is one talk-over worth to you?* Small values give an eager agent,
	large values a patient one, and the sweep between them is the Pareto front below.

	Framework adapters: `rubato.integrations.pipecat_processor.RubatoTurnGate` and
	`rubato.integrations.livekit_plugin.RubatoTurnDetector`.

	Source, benchmark protocol, ablations and sensitivity analyses: https://github.com/NagaYu/rubato

	## What it was measured to do

	![Talk-over rate against response latency: Rubato's frontier lies inside the fixed-threshold frontier](./pareto_maptask.png)

	Both axes are minimised. A fixed threshold can only ever trace the outer curve -- one threshold buys one point. The inset is the band real systems ship in.

	Held out on 5988 silences from speakers never seen in training (maptask, CC-BY-4.0).

	\| \| \|
	\|---\|---\|
	\| covers the fixed-threshold frontier \| 86% of its points \|
	\| covers the semantic-completeness frontier \| 100% of its points \|
	\| latency saved at matched talk-over \| 84 ms vs fixed, 96 ms vs semantic \|
	\| talk-over removed at matched latency \| 2.8 pp vs fixed, 2.1 pp vs semantic \|
	\| hazard calibration (ECE) \| 0.0007 \|
	\| CRPS skill over a covariate-free hazard \| +0.173 \|
	\| layer cost per 20 ms frame \| 0.26 ms median, 0.49 ms p99 \|

	At the operating point matched to a 1000 ms threshold: latency 1000 → 772 ms (95 % CI 707–831), talk-over 6.3% → 6.5% (95 % CI 5.6%–7.6%). Intervals are a conversation-level cluster bootstrap.

	## How it works

	1. Discrete-time hazard. `h_k = P(partner resumes in frame k+1 \| silent
	through k, evidence up to k)`, from a one-hidden-layer network over a causal
	feature vector: a radial-basis expansion of log elapsed silence, turn-so-far
	duration and pause count, transcript completeness (gated by ASR lag, because a
	real recogniser has not delivered the last word yet), terminal prosody where
	audio exists, acoustic precursor cues such as in-breaths, and a running
	per-partner posterior.
	2. Optimal stopping. A backward dynamic program over the forecast survival
	curve minimises `alpha * P(collision) + beta * latency`. Being pre-empted --
	the human carries on while the agent is still silent -- costs nothing, which
	is what produces the human-like behaviour: when a resumption looks likely,
	waiting is nearly free; when the floor is clearly open, the agent can start at
	zero gap.
	3. Online entrainment. Conjugate posteriors over each partner's pause and gap
	distributions, and an EMA of their speech rate, shrinking to a population
	prior so the first thirty seconds are never worse than not adapting.

	## Inputs, outputs, and what it needs from you

	\| you provide \| per frame \|
	\|---\|---\|
	\| 20 ms of mono audio (any rate; 16 kHz assumed) \| required \|
	\| the ASR's partial transcript \| optional, improves accuracy \|
	\| a partner id \| optional, enables entrainment \|

	\| you get back \| \|
	\|---\|---\|
	\| `should_speak` \| the decision \|
	\| `decision.planned_onset_s` \| when it currently intends to start \|
	\| `decision.p_overlap_now` \| collision probability if it started this instant \|
	\| `prediction.future_hazards` \| the full predicted silence distribution \|

	## Limitations

	- English lexical features. The temporal and acoustic half transfers; the
	completeness features were fitted on English and will mislead elsewhere.
	Retrain on target-language data.
	- Task-oriented training data. Trained on the HCRC Map Task Corpus: two people collaborating on a route-following task. Open-domain conversation has
	longer, more variable gaps; expect to retune `seconds_per_collision`.
	- Two-party only. Multi-party floor management is a different problem.
	- The offline protocol makes an assumption. On a floor transfer we assume the
	floor would have stayed open had the agent not spoken. The alternative is
	worse (informative censoring); both are reported in the repository README.
	- It is a timing layer, not a turn-taking oracle. It cannot know that you
	paused because you were about to say something difficult.

	## Responsible use

	Better timing makes an assistant less irritating. It also makes a synthetic voice
	harder to distinguish from a person, and that is a use this model is not for.

	- Disclose that the system is an AI. This model is released on the assumption
	that the deployment does so, in the modality the user is actually attending to
	-- spoken, not buried in a settings page.
	- Do not use it to impersonate anyone, to make an automated caller pass as
	human, or in any deployment whose value depends on the person not knowing.
	Several jurisdictions require disclosure for automated calls; treat that as a
	floor, not a target.
	- Do not use it to time interruptions strategically -- to cut a person off at
	a moment calculated to stop them finishing a thought. The asymmetric cost makes
	that easy to configure, which is exactly why it is named here.
	- The entrainment state is a behavioural profile of an individual. Keep it in
	session scope unless you have a reason and consent to persist it.

	## Training data

	maptask (CC-BY-4.0). Anderson et al. (1991), The HCRC Map Task Corpus. Language and Speech 34(4). Annotations (c) 2007 HCRC, Univ. of Edinburgh & Univ. of Glasgow. CC BY 4.0. https://groups.inf.ed.ac.uk/maptask/

	No audio was redistributed in building this model.

	## Citation

	```bibtex
	@software{rubato,
	title = {Rubato: predicting silence distributions for spoken-dialogue turn-taking},
	year = {2026},
	url = {https://huggingface.co/NagaYu/rubato-timing}
	}
	```