File size: 5,761 Bytes

---
license: mit
language:
- en
library_name: chatterbox-flash
pipeline_tag: text-to-speech
tags:
- text-to-speech
- zero-shot-tts
- block-diffusion
- speech-synthesis
---
<img width="800" alt="Resemble_AI_Chatterbox_Flash" src="https://cdn-uploads.huggingface.co/production/uploads/68af35bc5928acf3617150db/lCegiC8uo_eA3RHOVC6go.png" />

<h1 style="font-size: 32px">Chatterbox-Flash</h1>

<div style="display: flex; align-items: center; gap: 12px">
  <a href="https://huggingface.co/spaces/ResembleAI/chatterbox-flash-demo">
    <img src="https://huggingface.co/datasets/huggingface/badges/resolve/main/open-in-hf-spaces-sm.svg" alt="Open in HF" />
  </a>
</div>
<div style="display: flex; align-items: center; gap: 8px;">
  <span style="font-style: italic;white-space: pre-wrap">Made with  ❤️  by</span>
  <img width="100" alt="resemble-logo-horizontal" src="https://github.com/user-attachments/assets/35cf756b-3506-4943-9c72-c05ddfa4e525" />
</div>

Chatterbox-Flash is a **block-diffusion zero-shot TTS model** that extends the [Chatterbox-TTS](https://github.com/resemble-ai/chatterbox) pipeline with a parallel masked decoder while preserving streaming generation. It unmasks multiple speech tokens **in parallel within each block** while keeping **native block-by-block streaming**, delivering autoregressive-class quality at a fraction of the latency.

# Streaming & Speed Highlights
- **Native block-by-block streaming** — emits audio as each block is committed, no full-sequence wait.
- **~9× real-time** synthesis at the default config (`D = 16`, `α = 0.5`), up to **~13× real-time** at `D = 32`, `α = 0.75`.
- **Time-to-first-packet from 103 ms**, on par with streaming AR systems.
- **RTF as low as 0.076** — substantially faster than autoregressive streaming TTS at the same scale.
- **Early-decoding schedule** adaptively ends denoising early, cutting average steps per block by **~20%** at negligible quality cost.
- Built on a **FlashInfer paged KV cache + CUDA-graph** inference engine for low per-step overhead.

The released weights cover the four checkpoints needed at inference time:

| File                        | What it is                                                         |
| --------------------------- | ------------------------------------------------------------------ |
| `t3_flash.safetensors`      | Block-diffusion T3 decoder (Llama-520M + 1 extra `[MASK]` token).  |
| `s3gen.safetensors`         | Flow-matching S3Gen vocoder.                                       |
| `ve.safetensors`            | GE2E voice encoder (taken verbatim from `ResembleAI/chatterbox`).  |
| `tokenizer.json`            | English BPE tokenizer (taken verbatim from `ResembleAI/chatterbox`).|

# Quick start
```bash
pip install chatterbox-flash
```
```python
import torchaudio as ta
from chatterbox_flash import ChatterboxFlashTTS
tts = ChatterboxFlashTTS.from_pretrained("ResembleAI/chatterbox-flash", device="cuda")
wav = tts.generate(
    "Hello, world.",
    audio_prompt_path="reference.wav",
)
ta.save("out.wav", wav.unsqueeze(0).cpu(), tts.sr)
```

# Inference defaults (paper configuration)
- Block size `D = 16`
- Maximum `K = 10` denoising steps per block
- Sampling temperature `0.2`
- `shift` outlier schedule with `tau = 0.5`
- CFG with `w = 1.0`, `pmi_cfg` combination
- FlashInfer paged KV cache + CUDA graph capture

# Streaming Efficiency
Latency and throughput at concurrency 1, measured over 50 utterances. **TTFP** is the wall-clock time from request to the first emitted audio packet; **RTF** (real-time factor) is generation time divided by synthesized audio duration — lower is faster, and `RTF < 1` means faster than real time.

| Config (25 Hz, 0.5B) | TTFP (ms) ↓ | RTF ↓ |
| --- | :---: | :---: |
| `D = 16`, `α = 0.5` (default) | 118 | 0.107 |
| `D = 16`, `α = 0.75` | 106 | 0.091 |
| `D = 24`, `α = 0.5` | 119 | 0.100 |
| `D = 24`, `α = 0.75` | 105 | 0.084 |
| `D = 32`, `α = 0.5` | 115 | 0.090 |
| `D = 32`, `α = 0.75` | **103** | **0.076** |

Even on a single concurrent request, Chatterbox-Flash sustains roughly **9× real-time** synthesis at the default setting and **~13× real-time** at `D = 32`, `α = 0.75`, while keeping time-to-first-packet low enough for interactive streaming.

# Apple Silicon (MLX)
Chatterbox-Flash also runs locally on Apple Silicon via [MLX](https://github.com/ml-explore/mlx). The numbers below were measured on a **Mac M4** at the default configuration; both stay comfortably under real time (`RTF < 1`), and 4-bit quantization gives a further speedup.

| Backend | RTF ↓ |
| --- | :---: |
| MLX | 0.778 |
| MLX (4-bit quantized) | **0.665** |

# Quality (Seed-TTS test-en)
Zero-shot TTS quality on the Seed-TTS English benchmark, under the canonical configuration (`D = 16`). **SIM-o** is speaker similarity to the reference (higher is better), **WER** is word error rate from ASR transcription (lower is better), and **UTMOS** is a predicted naturalness score (higher is better). Results are shown for our two main decoding settings against the Chatterbox backbone and ground-truth audio for reference.

| System | SIM-o ↑ | WER ↓ | UTMOS ↑ |
| --- | :---: | :---: | :---: |
| Ground-truth | 0.734 | 2.14 | 3.52 |
| Chatterbox (AR backbone) | 0.685 | 2.20 | 4.10 |
| Chatterbox-Flash (`α = 0`) | **0.704** | **1.96** | **4.09** |
| Chatterbox-Flash (`α = 0.5`, early decoding) | **0.704** | 2.04 | 4.08 |

Converting the autoregressive backbone into a block-diffusion decoder improves both speaker similarity (`0.685 → 0.704`) and intelligibility (`2.20 → 1.96` WER) while keeping naturalness essentially unchanged — all while unlocking parallel, streaming-friendly decoding.

# License
MIT — see `LICENSE` in the source repository.