WaveGen / README.md
LeeSangHoon's picture
Add the WaveGen arXiv paper link
0711d66 verified
|
Raw History Blame Contribute Delete
12.2 kB
---
license: cc-by-nc-4.0
pipeline_tag: text-to-speech
library_name: wavegen
language:
- en
- zh
datasets:
- amphion/Emilia-Dataset
tags:
- WaveGen
- TTS
- zero-shot
- voice-cloning
- audio
- multilingual
- pytorch
---
# WaveGen [NeurIPS 2026]
**WaveGen: Truly End-to-End Waveform Generation via Internal Spectral Trajectory Learning**
Sang-Hoon Lee and Ha-Yeong Choi
Ajou University and KT
[Paper](https://arxiv.org/abs/2611.12527) · [Source Code](https://github.com/sh-lee-prml/WaveGen) · [Live Demo](https://huggingface.co/spaces/LeeSangHoon/WaveGen) · [Demo Page](https://wavegen26.github.io/demo) · [Benchmark Demo](https://sh-lee-prml.github.io/wavegen-tts-demo/) · [Spectral Forcing Demo](https://wavegen26.github.io/demo/spectral_forcing_visualization/)
WaveGen generates speech directly from text and reference audio using Spectral Forcing. The models are trained on English and Mandarin Chinese speech from Emilia, using **24 kHz audio** and **50 Hz waveform patches** (480 samples per patch).
## Models
This repository contains the pre-trained, V-Spec, and four-step Flash checkpoints in separate folders.
| Model | Checkpoint | Folder | Inference steps |
|---|---|---|---|
| WaveGen (Pre-trained) | 1300k EMA | `WaveGen/` | 50 |
| **WaveGen (Fine-tuned with v-spec losses) — Recommended** | **1400k EMA** | **`WaveGen_VSpec/`** | **50** |
| WaveGen-Flash | 48k EMA | `WaveGen_Flash/` | **4 (fixed)** |
**WaveGen (Fine-tuned with v-spec losses)** is the **recommended model for inference** and the default in the source repository. All checkpoints contain **FP32 EMA weights only**, in Safetensors format. Optimizer state and other training state are excluded; these files are intended for inference, not for resuming a training run.
**WaveGen-Flash** is optimized for four-step speech generation. Its complete merged
weights load from one Safetensors file, together with its YAML and vocabulary.
The Flash release includes inference code and model weights; its training recipe
is not included.
## Seed-TTS evaluation
Results on the complete standard Seed-TTS test sets for the EMA weights released here.
The pre-trained and V-Spec models use **Prompt Anchor**, **Scheduled K3** (base CFG 1.25), **50 Euler steps**,
Mel temperature 0, inference bottleneck dropout 0.2, and seed 0, with FP32 model
weights and BF16 CUDA inference. The original test prompts are used.
Flash uses **four Euler steps**, the time grid **`[0, 0.1, 0.25, 0.5, 1.0]`**,
CFG 0, inference bottleneck dropout 0, waveform temperature 1, **initial Mel
temperature 0**, and seed 0. It uses the same original test prompts and
FP32-weight/BF16-inference precision. Prompt states interpolate from their
initial noise toward the reference at each step; the initial Mel noise is
scaled by the Mel temperature. The Flash results below use the new default
temperature 0. WER/CER and UTMOS are truncated to two decimal places,
and SIM to three decimal places (not rounded).
The complete test sets contain 1,088 English and 2,020 Chinese utterances.
<table>
<colgroup span="1"></colgroup>
<colgroup span="3"></colgroup>
<colgroup span="3"></colgroup>
<thead>
<tr>
<th rowspan="2" scope="col" align="left">Model</th>
<th colspan="3" scope="colgroup" align="center">EN</th>
<th colspan="3" scope="colgroup" align="center">ZH</th>
</tr>
<tr>
<th scope="col" align="center">WER (%) ↓</th>
<th scope="col" align="center">SIM ↑</th>
<th scope="col" align="center">UTMOS ↑</th>
<th scope="col" align="center">CER (%) ↓</th>
<th scope="col" align="center">SIM ↑</th>
<th scope="col" align="center">UTMOS ↑</th>
</tr>
</thead>
<tbody>
<tr>
<th scope="row" align="left"><strong>WaveGen</strong><br><sub>Pre-trained · 1300k EMA</sub></th>
<td align="center">1.47</td>
<td align="center">0.755</td>
<td align="center">3.68</td>
<td align="center">1.75</td>
<td align="center">0.805</td>
<td align="center">2.87</td>
</tr>
<tr>
<th scope="row" align="left"><strong>WaveGen V-Spec</strong><br><sub>1400k EMA · Recommended</sub></th>
<td align="center">1.39</td>
<td align="center">0.743</td>
<td align="center">3.59</td>
<td align="center">1.65</td>
<td align="center">0.783</td>
<td align="center">2.79</td>
</tr>
<tr>
<th scope="row" align="left"><strong>WaveGen-Flash</strong><br><sub>48k EMA · 4 steps</sub></th>
<td align="center">1.49</td>
<td align="center">0.727</td>
<td align="center">3.53</td>
<td align="center">1.59</td>
<td align="center">0.768</td>
<td align="center">2.74</td>
</tr>
</tbody>
</table>
**WaveGen achieves state-of-the-art performance among TTS models trained on the Emilia dataset.**
WER is reported for English and CER for Chinese. Error rates are percentages;
all metrics are macro-averaged over utterances. The Safetensors exports preserve
the FP32 EMA inference tensor values.
## Download
Download the checkpoint, YAML, and vocabulary from the desired model folder and place that folder under `ckpts/` in the WaveGen source repository:
```text
ckpts/
├── WaveGen/
│ ├── model_1300000.safetensors
│ ├── WaveGen_SF_Emilia_24k_50Hz_IBDrop02.yaml
│ └── vocab.txt
├── WaveGen_VSpec/
│ ├── model_1400000.safetensors
│ ├── WaveGen_VSpec.yaml
│ └── vocab.txt
└── WaveGen_Flash/
├── model.safetensors
├── WaveGen_Flash.yaml
└── vocab.txt
```
Keep each checkpoint together with its matching configuration and vocabulary. This Hub repository uses `WaveGen/`, `WaveGen_VSpec/`, and `WaveGen_Flash/` at its root; `ckpts/` is their destination in your local source checkout.
## Usage
Follow the [installation instructions](https://github.com/sh-lee-prml/WaveGen#installation), then run from the WaveGen source repository root. Supply a reference recording, its exact transcript, and the text to synthesize. Automatic transcription is not required.
### Fine-tuned model (recommended default)
```bash
python -m wavegen_v_spec.infer.infer_cli \
--ckpt_file ckpts/WaveGen_VSpec/model_1400000.safetensors \
--model_cfg ckpts/WaveGen_VSpec/WaveGen_VSpec.yaml \
--vocab_file ckpts/WaveGen_VSpec/vocab.txt \
--ref_audio reference.wav \
--ref_text "The transcript of the reference recording." \
--gen_text "The text to synthesize."
```
### Pre-trained model
```bash
python -m wavegen.infer.infer_cli \
--ckpt_file ckpts/WaveGen/model_1300000.safetensors \
--model_cfg ckpts/WaveGen/WaveGen_SF_Emilia_24k_50Hz_IBDrop02.yaml \
--vocab_file ckpts/WaveGen/vocab.txt \
--cfg_schedule scheduled_k3 \
--cfg_strength 1.25 \
--ref_audio reference.wav \
--ref_text "The transcript of the reference recording." \
--gen_text "The text to synthesize."
```
Both examples use EMA weights, **Prompt Anchor** (`Prompt_xt_Anchor`), **Scheduled K3** (base CFG 1.25), 50 steps, Mel temperature 0, and seed 0. The supplied models use inference bottleneck dropout 0.2. The pre-trained module uses constant CFG 3 when no schedule is specified, so its example selects Scheduled K3 explicitly. CUDA inference uses BF16 autocast with FP32 model weights; CPU inference uses FP32.
### Load the fine-tuned model from Hugging Face
The subfolder paths below select the 1400k model from [LeeSangHoon/WaveGen](https://huggingface.co/LeeSangHoon/WaveGen). The loader resolves `main` to one commit before downloading the three assets; use a full commit SHA instead of `main` to pin a specific release.
```python
import soundfile as sf
import torch
from wavegen_v_spec.infer.utils_infer import WaveGen
torch.set_num_threads(1)
model = WaveGen.from_pretrained(
"LeeSangHoon/WaveGen",
revision="main",
checkpoint_filename="WaveGen_VSpec/model_1400000.safetensors",
config_filename="WaveGen_VSpec/WaveGen_VSpec.yaml",
vocab_filename="WaveGen_VSpec/vocab.txt",
use_ema=True,
)
sample_rate, audio = model.synthesize(
"reference.wav",
"The transcript of the reference recording.",
"The text to synthesize.",
)
sf.write("generated.wav", audio, sample_rate, subtype="PCM_16")
```
For the pre-trained model, download the `WaveGen/` folder and use the pre-trained CLI above. See the [source repository](https://github.com/sh-lee-prml/WaveGen) for the local Python API, Gradio demo, and batch evaluation.
The source checkout defaults to the V-Spec 1400k Safetensors checkpoint. The base
and V-Spec loaders support existing local `.pt` checkpoints through an explicit
checkpoint-path override.
### WaveGen-Flash (4 Steps)
```bash
python -m wavegen_flash.infer.infer_cli \
--checkpoint ckpts/WaveGen_Flash/model.safetensors \
--model-config ckpts/WaveGen_Flash/WaveGen_Flash.yaml \
--vocab-file ckpts/WaveGen_Flash/vocab.txt \
--ref-audio reference.wav \
--ref-text "The transcript of the reference recording." \
--gen-text "The text to synthesize." \
--output generated_flash.wav
```
Or load the same bundle directly from this repository:
```python
import soundfile as sf
import torch
from wavegen_flash import WaveGenFlash
torch.set_num_threads(1)
model = WaveGenFlash.from_pretrained("LeeSangHoon/WaveGen", revision="main")
sample_rate, audio = model.synthesize(
"reference.wav",
"The transcript of the reference recording.",
"The text to synthesize.",
)
sf.write("generated_flash.wav", audio, sample_rate, subtype="PCM_16")
```
The Flash loader accepts Safetensors only and requires no additional base weights.
Its YAML fixes four Euler calls at `[0, 0.1, 0.25, 0.5]`, ending at 1.0, with
CFG 0 and IB dropout 0. The grid already includes time shift 3; it is not shifted
again during inference. The final Mel output head is skipped by default;
`--compute-final-mel` or `compute_final_mel=True` restores its computation.
The Mel input and shared network remain active on the final call.
The YAML sets `mel_temperature: 0.0` by default, while waveform temperature
remains 1. Mel temperature scales only the initial Gaussian Mel noise;
the clean reference Mel and later predicted Mel states are not scaled.
Both noise draws are preserved even at temperature 0. Use
`--mel-temperature 1` or `model.synthesize(..., mel_temperature=1.0)`
to reproduce the previous initialization with the same weights.
Compared with temperature 1, temperature 0 improves WER/CER, SIM, and
UTMOS on both complete Seed-TTS test sets.
The default duration limit is 30 seconds for prompt plus generated speech.
Split longer text or explicitly increase `max_duration_seconds`; the sampler
does not automatically split text. See the
[Flash guide](https://github.com/sh-lee-prml/WaveGen/blob/main/src/wavegen_flash/README.md)
for the complete inference interface.
## Safeguards and limitations
- Use reference speech with the speaker's informed consent and permission to use the recording.
- Do not use generated speech for deceptive impersonation, fraud, harassment, or fabricated statements presented as authentic speech.
- Clearly identify shared or published outputs as AI-generated audio, and protect reference recordings and personal information.
- Reference quality and transcript accuracy affect the generated speech. Pronunciation errors, omissions, repetitions, and speaker-similarity failures may occur.
- For long passages, split the text into shorter segments; the single-utterance examples above do not automatically split long text.
These usage guidelines are not automatic safety filters. Public-facing services should add appropriate access controls, abuse reporting, and review of suspected misuse.
## License
The model checkpoints are released under [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/).
WaveGen source-code contributions are released under the [MIT License](https://github.com/sh-lee-prml/WaveGen/blob/main/LICENSE). Reused source components retain their original licenses and notices; see the [source-code third-party notices](https://github.com/sh-lee-prml/WaveGen/blob/main/docs/THIRD_PARTY_NOTICES.md).