Update README.md
Browse files
README.md
CHANGED
|
@@ -24,7 +24,15 @@ tags:
|
|
| 24 |
<img width="100" alt="resemble-logo-horizontal" src="https://github.com/user-attachments/assets/35cf756b-3506-4943-9c72-c05ddfa4e525" />
|
| 25 |
</div>
|
| 26 |
|
| 27 |
-
Chatterbox-Flash is a **block-diffusion zero-shot TTS model** that extends the [Chatterbox-TTS](https://github.com/resemble-ai/chatterbox) pipeline with a parallel masked decoder while preserving streaming generation.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 28 |
|
| 29 |
The released weights cover the four checkpoints needed at inference time:
|
| 30 |
|
|
@@ -58,5 +66,31 @@ ta.save("out.wav", wav.unsqueeze(0).cpu(), tts.sr)
|
|
| 58 |
- CFG with `w = 1.0`, `pmi_cfg` combination
|
| 59 |
- FlashInfer paged KV cache + CUDA graph capture
|
| 60 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 61 |
# License
|
| 62 |
MIT — see `LICENSE` in the source repository.
|
|
|
|
| 24 |
<img width="100" alt="resemble-logo-horizontal" src="https://github.com/user-attachments/assets/35cf756b-3506-4943-9c72-c05ddfa4e525" />
|
| 25 |
</div>
|
| 26 |
|
| 27 |
+
Chatterbox-Flash is a **block-diffusion zero-shot TTS model** that extends the [Chatterbox-TTS](https://github.com/resemble-ai/chatterbox) pipeline with a parallel masked decoder while preserving streaming generation. It unmasks multiple speech tokens **in parallel within each block** while keeping **native block-by-block streaming**, delivering autoregressive-class quality at a fraction of the latency.
|
| 28 |
+
|
| 29 |
+
# Streaming & Speed Highlights
|
| 30 |
+
- **Native block-by-block streaming** — emits audio as each block is committed, no full-sequence wait.
|
| 31 |
+
- **~9× real-time** synthesis at the default config (`D = 16`, `α = 0.5`), up to **~13× real-time** at `D = 32`, `α = 0.75`.
|
| 32 |
+
- **Time-to-first-packet from 103 ms**, on par with streaming AR systems.
|
| 33 |
+
- **RTF as low as 0.076** — substantially faster than autoregressive streaming TTS at the same scale.
|
| 34 |
+
- **Early-decoding schedule** adaptively ends denoising early, cutting average steps per block by **~20%** at negligible quality cost.
|
| 35 |
+
- Built on a **FlashInfer paged KV cache + CUDA-graph** inference engine for low per-step overhead.
|
| 36 |
|
| 37 |
The released weights cover the four checkpoints needed at inference time:
|
| 38 |
|
|
|
|
| 66 |
- CFG with `w = 1.0`, `pmi_cfg` combination
|
| 67 |
- FlashInfer paged KV cache + CUDA graph capture
|
| 68 |
|
| 69 |
+
# Streaming Efficiency
|
| 70 |
+
Latency and throughput at concurrency 1, measured over 50 utterances. **TTFP** is the wall-clock time from request to the first emitted audio packet; **RTF** (real-time factor) is generation time divided by synthesized audio duration — lower is faster, and `RTF < 1` means faster than real time.
|
| 71 |
+
|
| 72 |
+
| Config (25 Hz, 0.5B) | TTFP (ms) ↓ | RTF ↓ |
|
| 73 |
+
| --- | :---: | :---: |
|
| 74 |
+
| `D = 16`, `α = 0.5` (default) | 118 | 0.107 |
|
| 75 |
+
| `D = 16`, `α = 0.75` | 106 | 0.091 |
|
| 76 |
+
| `D = 24`, `α = 0.5` | 119 | 0.100 |
|
| 77 |
+
| `D = 24`, `α = 0.75` | 105 | 0.084 |
|
| 78 |
+
| `D = 32`, `α = 0.5` | 115 | 0.090 |
|
| 79 |
+
| `D = 32`, `α = 0.75` | **103** | **0.076** |
|
| 80 |
+
|
| 81 |
+
Even on a single concurrent request, Chatterbox-Flash sustains roughly **9× real-time** synthesis at the default setting and **~13× real-time** at `D = 32`, `α = 0.75`, while keeping time-to-first-packet low enough for interactive streaming.
|
| 82 |
+
|
| 83 |
+
# Quality (Seed-TTS test-en)
|
| 84 |
+
Zero-shot TTS quality on the Seed-TTS English benchmark, under the canonical configuration (`D = 16`). **SIM-o** is speaker similarity to the reference (higher is better), **WER** is word error rate from ASR transcription (lower is better), and **UTMOS** is a predicted naturalness score (higher is better). Results are shown for our two main decoding settings against the Chatterbox backbone and ground-truth audio for reference.
|
| 85 |
+
|
| 86 |
+
| System | SIM-o ↑ | WER ↓ | UTMOS ↑ |
|
| 87 |
+
| --- | :---: | :---: | :---: |
|
| 88 |
+
| Ground-truth | 0.734 | 2.14 | 3.52 |
|
| 89 |
+
| Chatterbox (AR backbone) | 0.685 | 2.20 | 4.10 |
|
| 90 |
+
| Chatterbox-Flash (`α = 0`) | **0.704** | **1.96** | **4.09** |
|
| 91 |
+
| Chatterbox-Flash (`α = 0.5`, early decoding) | **0.704** | 2.04 | 4.08 |
|
| 92 |
+
|
| 93 |
+
Converting the autoregressive backbone into a block-diffusion decoder improves both speaker similarity (`0.685 → 0.704`) and intelligibility (`2.20 → 1.96` WER) while keeping naturalness essentially unchanged — all while unlocking parallel, streaming-friendly decoding.
|
| 94 |
+
|
| 95 |
# License
|
| 96 |
MIT — see `LICENSE` in the source repository.
|