Update README.md

b3aac15 verified 1 day ago

6.2 kB

---
license: cc-by-nc-4.0
language:
- en
- de
- fr
- es
- it
- pl
- pt
base_model: k2-fsa/OmniVoice
pipeline_tag: text-to-speech
tags:
- text-to-speech
- voice-cloning
- zero-shot
---
# Reverb-TTS

<!-- TODO: model graphic -->

**Reverb** is a **text-to-speech model** designed for high-quality **voice cloning**. Built on top of one of the leading open-weight TTS models,
it combines an exceptional quality of mimicking any reference speaker, including prosody, emotions and voice timbre,
while staying lightweight and fast enough to run locally on mobile and desktop devices.

## What is Reverb?

Reverb starts from [OmniVoice](https://github.com/k2-fsa/OmniVoice), a 0.6B masked-diffusion TTS model that fills in
a grid of audio-codec tokens over several passes, conditioned on the text and on a reference speaker's audio. We made
Reverb from it in three stages:

1. **Pruning with distillation.** 4 of the 28 transformer layers are removed, the feed-forward layers shrink from 3072
   to 2048, and the text vocabulary from 151,676 to 62,035 tokens. The pruned model learns to match OmniVoice's
   predictions.
2. **Supervised retraining** in Reverb's seven languages, on cleaned and selected subsets of open speech datasets such
   as [Emilia](https://huggingface.co/datasets/amphion/Emilia-Dataset) and variants of YODAS2, such as
   [yodas2-sidon](https://huggingface.co/datasets/sarulab-speech/yodas2_sidon).
3. **RLHF-style post-training:** several rounds of iterative DPO with an automatic judge fitted on human preference
   rankings.

The result has 297M parameters, less than half of OmniVoice's 613M. Additional performance improvements let it
generate speech 10× to 20× faster than the original model.

![How Reverb is made: OmniVoice is pruned and distilled, retrained on seven languages, then refined with iterative DPO against a critic fitted on human preference rankings.](assets/training_scheme.svg)

### Quality, size and speed

We compare Reverb with open-weight voice-cloning models on the
[MiniMax Multilingual Test Set](https://huggingface.co/datasets/k2-fsa/TTS_eval_datasets), restricted to Reverb's seven
languages: 700 sentences, 100 per language, each spoken in the voice of one of two reference speakers of that
language. Every model generates one take per sentence with its default settings. WER (Whisper large-v3) and speaker
similarity (SIM-o) come from OmniVoice's evaluation scripts and are averaged over the languages.

| Model | Parameters (model + codec decoder) | WER ↓ | SIM-o ↑ |
|:---|---:|---:|---:|
| **Reverb** | **319M** | **1.35%** | 0.818 |
| [OmniVoice](https://huggingface.co/k2-fsa/OmniVoice) | 634M | 1.64% | **0.835** |
| [Audio8-TTS-Realistic](https://huggingface.co/Edge0/Audio8-TTS-Realistic-0.6b) | 732M | 1.97% | 0.750 |
| [Qwen3-TTS 0.6B Base](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-0.6B-Base) | 1,020M | 11.96%¹ | 0.781 |
| [Qwen3-TTS 1.7B Base](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-Base) | 2,031M | 1.52%¹ | 0.781 |
| [VoxCPM2](https://huggingface.co/openbmb/VoxCPM2) | 2,336M | 1.55% | 0.810 |

Parameters count the generating model and the decoder of its audio codec; encoders used only to process the reference
recording are not counted.<br>
¹ Qwen3-TTS does not support Polish, so its WER is averaged over the other six languages.

Generation speed, in × real time (seconds of audio per second of compute; higher is better), on MiniMax English
sentences. Timing runs from the text and an already-encoded reference voice to the waveform, codec decoding included.
Every model runs its default settings in bf16.

| Hardware | Reverb | OmniVoice | Audio8 | Qwen3-TTS 0.6B | Qwen3-TTS 1.7B | VoxCPM2 | Fish S2 Pro |
|:---|---:|---:|---:|---:|---:|---:|---:|
| NVIDIA RTX 5090, batch 32 | **164.9×** | 13.2× | 4.2× | 3.3× | 4.0× | 3.0×² | 0.18×² |
| Apple M4 Pro, batch 1 | **21.6×**³ | 0.84× | 0.76× | 0.51× | 0.52× | 0.69× | – |

² No batched generation: sentences run one at a time.<br>
³ Reverb's ExecuTorch MLX program (fp16) with the Core ML codec decoder. In PyTorch on the GPU (MPS), as the other models run
on the M4 Pro: 12.3×.

## Usage

```bash
pip install reverb-tts
```

```python
import soundfile as sf
from reverb_tts import TTS
tts = TTS.from_pretrained("software-mansion/Reverb-TTS")
voice = tts.voice("reference.wav", "The exact transcript of the reference recording.", language="en")
audio = tts.generate("Hello! This is my cloned voice.", voice, language="en")
sf.write("output.wav", audio, tts.sample_rate)
```

The reference should be 3 to 20 seconds of clean speech from one speaker, with its exact transcript. Languages: `en`,
`de`, `fr`, `es`, `it`, `pl`, `pt`.

## Model variants

| Variant | Weights | Activations | Size |
|:---|:---|:---|---:|
| `fp32` | fp32 | full precision | 1.19 GB |
| `int8` (default) | int8, one scale per output channel | full precision | 307 MB |
| `int8a8` | int8, one scale per output channel | int8, one scale per token | 307 MB |
| `int4` | int4, one scale per group of 32 inputs | full precision | 191 MB |

Select one with `TTS.from_pretrained(..., variant="int4")`. The weights are in `model/safetensors/<variant>/`.
In automatic scoring, `int8` and `int8a8` show no measurable loss against `fp32`, and `int4` a small one.
ExecuTorch programs for on-device use are in `model/executorch/`: MLX (Apple GPUs) in fp16, int8 and int4, and Vulkan
(Android GPUs) in int4. `voices/` holds ready-made reference voices.

## License and acknowledgments

- **Code:** Apache 2.0; see [LICENSE](https://github.com/software-mansion-labs/Echo-TTS/blob/main/LICENSE).
- **Model weights:** [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/), for non-commercial use with
  attribution.

Reverb's weights were initialized from [OmniVoice](https://github.com/k2-fsa/OmniVoice) by the k2-fsa team, and its
target-length rule and prompt format come from OmniVoice's code. We thank its authors for releasing the model and code
openly. Reverb uses the [Higgs Audio v2 tokenizer](https://huggingface.co/eustlb/higgs-audio-v2-tokenizer) as its audio
codec, and [Verbalis](https://github.com/software-mansion-labs/Verbalis) prepares its text.

	---
	license: cc-by-nc-4.0
	language:
	- en
	- de
	- fr
	- es
	- it
	- pl
	- pt
	base_model: k2-fsa/OmniVoice
	pipeline_tag: text-to-speech
	tags:
	- text-to-speech
	- voice-cloning
	- zero-shot
	---

	# Reverb-TTS

	<!-- TODO: model graphic -->

	Reverb is a text-to-speech model designed for high-quality voice cloning. Built on top of one of the leading open-weight TTS models,
	it combines an exceptional quality of mimicking any reference speaker, including prosody, emotions and voice timbre,
	while staying lightweight and fast enough to run locally on mobile and desktop devices.

	## What is Reverb?

	Reverb starts from [OmniVoice](https://github.com/k2-fsa/OmniVoice), a 0.6B masked-diffusion TTS model that fills in
	a grid of audio-codec tokens over several passes, conditioned on the text and on a reference speaker's audio. We made
	Reverb from it in three stages:

	1. Pruning with distillation. 4 of the 28 transformer layers are removed, the feed-forward layers shrink from 3072
	to 2048, and the text vocabulary from 151,676 to 62,035 tokens. The pruned model learns to match OmniVoice's
	predictions.
	2. Supervised retraining in Reverb's seven languages, on cleaned and selected subsets of open speech datasets such
	as [Emilia](https://huggingface.co/datasets/amphion/Emilia-Dataset) and variants of YODAS2, such as
	[yodas2-sidon](https://huggingface.co/datasets/sarulab-speech/yodas2_sidon).
	3. RLHF-style post-training: several rounds of iterative DPO with an automatic judge fitted on human preference
	rankings.

	The result has 297M parameters, less than half of OmniVoice's 613M. Additional performance improvements let it
	generate speech 10× to 20× faster than the original model.

	![How Reverb is made: OmniVoice is pruned and distilled, retrained on seven languages, then refined with iterative DPO against a critic fitted on human preference rankings.](assets/training_scheme.svg)

	### Quality, size and speed

	We compare Reverb with open-weight voice-cloning models on the
	[MiniMax Multilingual Test Set](https://huggingface.co/datasets/k2-fsa/TTS_eval_datasets), restricted to Reverb's seven
	languages: 700 sentences, 100 per language, each spoken in the voice of one of two reference speakers of that
	language. Every model generates one take per sentence with its default settings. WER (Whisper large-v3) and speaker
	similarity (SIM-o) come from OmniVoice's evaluation scripts and are averaged over the languages.

	\| Model \| Parameters (model + codec decoder) \| WER ↓ \| SIM-o ↑ \|
	\|:---\|---:\|---:\|---:\|
	\| Reverb \| 319M \| 1.35% \| 0.818 \|
	\| [OmniVoice](https://huggingface.co/k2-fsa/OmniVoice) \| 634M \| 1.64% \| 0.835 \|
	\| [Audio8-TTS-Realistic](https://huggingface.co/Edge0/Audio8-TTS-Realistic-0.6b) \| 732M \| 1.97% \| 0.750 \|
	\| [Qwen3-TTS 0.6B Base](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-0.6B-Base) \| 1,020M \| 11.96%¹ \| 0.781 \|
	\| [Qwen3-TTS 1.7B Base](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-Base) \| 2,031M \| 1.52%¹ \| 0.781 \|
	\| [VoxCPM2](https://huggingface.co/openbmb/VoxCPM2) \| 2,336M \| 1.55% \| 0.810 \|

	Parameters count the generating model and the decoder of its audio codec; encoders used only to process the reference
	recording are not counted.<br>
	¹ Qwen3-TTS does not support Polish, so its WER is averaged over the other six languages.

	Generation speed, in × real time (seconds of audio per second of compute; higher is better), on MiniMax English
	sentences. Timing runs from the text and an already-encoded reference voice to the waveform, codec decoding included.
	Every model runs its default settings in bf16.

	\| Hardware \| Reverb \| OmniVoice \| Audio8 \| Qwen3-TTS 0.6B \| Qwen3-TTS 1.7B \| VoxCPM2 \| Fish S2 Pro \|
	\|:---\|---:\|---:\|---:\|---:\|---:\|---:\|---:\|
	\| NVIDIA RTX 5090, batch 32 \| 164.9× \| 13.2× \| 4.2× \| 3.3× \| 4.0× \| 3.0×² \| 0.18×² \|
	\| Apple M4 Pro, batch 1 \| 21.6×³ \| 0.84× \| 0.76× \| 0.51× \| 0.52× \| 0.69× \| – \|

	² No batched generation: sentences run one at a time.<br>
	³ Reverb's ExecuTorch MLX program (fp16) with the Core ML codec decoder. In PyTorch on the GPU (MPS), as the other models run
	on the M4 Pro: 12.3×.

	## Usage

	```bash
	pip install reverb-tts
	```

	```python
	import soundfile as sf
	from reverb_tts import TTS

	tts = TTS.from_pretrained("software-mansion/Reverb-TTS")
	voice = tts.voice("reference.wav", "The exact transcript of the reference recording.", language="en")
	audio = tts.generate("Hello! This is my cloned voice.", voice, language="en")
	sf.write("output.wav", audio, tts.sample_rate)
	```

	The reference should be 3 to 20 seconds of clean speech from one speaker, with its exact transcript. Languages: `en`,
	`de`, `fr`, `es`, `it`, `pl`, `pt`.

	## Model variants

	\| Variant \| Weights \| Activations \| Size \|
	\|:---\|:---\|:---\|---:\|
	\| `fp32` \| fp32 \| full precision \| 1.19 GB \|
	\| `int8` (default) \| int8, one scale per output channel \| full precision \| 307 MB \|
	\| `int8a8` \| int8, one scale per output channel \| int8, one scale per token \| 307 MB \|
	\| `int4` \| int4, one scale per group of 32 inputs \| full precision \| 191 MB \|

	Select one with `TTS.from_pretrained(..., variant="int4")`. The weights are in `model/safetensors/<variant>/`.
	In automatic scoring, `int8` and `int8a8` show no measurable loss against `fp32`, and `int4` a small one.
	ExecuTorch programs for on-device use are in `model/executorch/`: MLX (Apple GPUs) in fp16, int8 and int4, and Vulkan
	(Android GPUs) in int4. `voices/` holds ready-made reference voices.

	## License and acknowledgments

	- Code: Apache 2.0; see [LICENSE](https://github.com/software-mansion-labs/Echo-TTS/blob/main/LICENSE).
	- Model weights: [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/), for non-commercial use with
	attribution.

	Reverb's weights were initialized from [OmniVoice](https://github.com/k2-fsa/OmniVoice) by the k2-fsa team, and its
	target-length rule and prompt format come from OmniVoice's code. We thank its authors for releasing the model and code
	openly. Reverb uses the [Higgs Audio v2 tokenizer](https://huggingface.co/eustlb/higgs-audio-v2-tokenizer) as its audio
	codec, and [Verbalis](https://github.com/software-mansion-labs/Verbalis) prepares its text.