Blame for README.md

---
license: cc-by-nc-4.0
pipeline_tag: text-to-speech
library_name: wavegen
language:
- en
- zh
datasets:
- amphion/Emilia-Dataset
tags:
- WaveGen
- TTS
- zero-shot
- voice-cloning
- audio
- multilingual
- pytorch
---

# WaveGen [NeurIPS 2026]

**WaveGen: Truly End-to-End Waveform Generation via Internal Spectral Trajectory Learning**  
Sang-Hoon Lee and Ha-Yeong Choi

Ajou University and KT 



[Paper](https://arxiv.org/abs/2611.12527) · [Source Code](https://github.com/sh-lee-prml/WaveGen) · [Live Demo](https://huggingface.co/spaces/LeeSangHoon/WaveGen) · [Demo Page](https://wavegen26.github.io/demo) · [Benchmark Demo](https://sh-lee-prml.github.io/wavegen-tts-demo/) · [Spectral Forcing Demo](https://wavegen26.github.io/demo/spectral_forcing_visualization/)

WaveGen generates speech directly from text and reference audio using Spectral Forcing. The models are trained on English and Mandarin Chinese speech from Emilia, using **24 kHz audio** and **50 Hz waveform patches** (480 samples per patch).

## Models

This repository contains the pre-trained, V-Spec, and four-step Flash checkpoints in separate folders.

| Model | Checkpoint | Folder | Inference steps |
|---|---|---|---|
| WaveGen (Pre-trained) | 1300k EMA | `WaveGen/` | 50 |
| **WaveGen (Fine-tuned with v-spec losses) — Recommended** | **1400k EMA** | **`WaveGen_VSpec/`** | **50** |
| WaveGen-Flash | 48k EMA | `WaveGen_Flash/` | **4 (fixed)** |

**WaveGen (Fine-tuned with v-spec losses)** is the **recommended model for inference** and the default in the source repository. All checkpoints contain **FP32 EMA weights only**, in Safetensors format. Optimizer state and other training state are excluded; these files are intended for inference, not for resuming a training run.

**WaveGen-Flash** is optimized for four-step speech generation. Its complete merged
weights load from one Safetensors file, together with its YAML and vocabulary.
The Flash release includes inference code and model weights; its training recipe
is not included.

## Seed-TTS evaluation

Results on the complete standard Seed-TTS test sets for the EMA weights released here.
The pre-trained and V-Spec models use **Prompt Anchor**, **Scheduled K3** (base CFG 1.25), **50 Euler steps**,
Mel temperature 0, inference bottleneck dropout 0.2, and seed 0, with FP32 model
weights and BF16 CUDA inference. The original test prompts are used.

Flash uses **four Euler steps**, the time grid **`[0, 0.1, 0.25, 0.5, 1.0]`**,
CFG 0, inference bottleneck dropout 0, waveform temperature 1, **initial Mel
temperature 0**, and seed 0. It uses the same original test prompts and
FP32-weight/BF16-inference precision. Prompt states interpolate from their
initial noise toward the reference at each step; the initial Mel noise is
scaled by the Mel temperature. The Flash results below use the new default
temperature 0. WER/CER and UTMOS are truncated to two decimal places,
and SIM to three decimal places (not rounded).

The complete test sets contain 1,088 English and 2,020 Chinese utterances.

<table>
  <colgroup span="1"></colgroup>
  <colgroup span="3"></colgroup>
  <colgroup span="3"></colgroup>
  <thead>
    <tr>
      <th rowspan="2" scope="col" align="left">Model</th>
      <th colspan="3" scope="colgroup" align="center">EN</th>
      <th colspan="3" scope="colgroup" align="center">ZH</th>
    </tr>
    <tr>
      <th scope="col" align="center">WER (%) ↓</th>
      <th scope="col" align="center">SIM ↑</th>
      <th scope="col" align="center">UTMOS ↑</th>
      <th scope="col" align="center">CER (%) ↓</th>
      <th scope="col" align="center">SIM ↑</th>
      <th scope="col" align="center">UTMOS ↑</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th scope="row" align="left"><strong>WaveGen</strong><br><sub>Pre-trained · 1300k EMA</sub></th>
      <td align="center">1.47</td>
      <td align="center">0.755</td>
      <td align="center">3.68</td>
      <td align="center">1.75</td>
      <td align="center">0.805</td>
      <td align="center">2.87</td>
    </tr>
    <tr>
      <th scope="row" align="left"><strong>WaveGen V-Spec</strong><br><sub>1400k EMA · Recommended</sub></th>
      <td align="center">1.39</td>
      <td align="center">0.743</td>
      <td align="center">3.59</td>
      <td align="center">1.65</td>
      <td align="center">0.783</td>
      <td align="center">2.79</td>
    </tr>
    <tr>
      <th scope="row" align="left"><strong>WaveGen-Flash</strong><br><sub>48k EMA · 4 steps</sub></th>
      <td align="center">1.49</td>
      <td align="center">0.727</td>
      <td align="center">3.53</td>
      <td align="center">1.59</td>
      <td align="center">0.768</td>
      <td align="center">2.74</td>
    </tr>
  </tbody>
</table>

**WaveGen achieves state-of-the-art performance among TTS models trained on the Emilia dataset.**

WER is reported for English and CER for Chinese. Error rates are percentages;
all metrics are macro-averaged over utterances. The Safetensors exports preserve
the FP32 EMA inference tensor values.

## Download

Download the checkpoint, YAML, and vocabulary from the desired model folder and place that folder under `ckpts/` in the WaveGen source repository:

```text
ckpts/
├── WaveGen/
│   ├── model_1300000.safetensors
│   ├── WaveGen_SF_Emilia_24k_50Hz_IBDrop02.yaml
│   └── vocab.txt
├── WaveGen_VSpec/
│   ├── model_1400000.safetensors
│   ├── WaveGen_VSpec.yaml
│   └── vocab.txt
└── WaveGen_Flash/
    ├── model.safetensors
    ├── WaveGen_Flash.yaml
    └── vocab.txt
```

Keep each checkpoint together with its matching configuration and vocabulary. This Hub repository uses `WaveGen/`, `WaveGen_VSpec/`, and `WaveGen_Flash/` at its root; `ckpts/` is their destination in your local source checkout.

## Usage

Follow the [installation instructions](https://github.com/sh-lee-prml/WaveGen#installation), then run from the WaveGen source repository root. Supply a reference recording, its exact transcript, and the text to synthesize. Automatic transcription is not required.

### Fine-tuned model (recommended default)

```bash
python -m wavegen_v_spec.infer.infer_cli \
  --ckpt_file ckpts/WaveGen_VSpec/model_1400000.safetensors \
  --model_cfg ckpts/WaveGen_VSpec/WaveGen_VSpec.yaml \
  --vocab_file ckpts/WaveGen_VSpec/vocab.txt \
  --ref_audio reference.wav \
  --ref_text "The transcript of the reference recording." \
  --gen_text "The text to synthesize."
```

### Pre-trained model

```bash
python -m wavegen.infer.infer_cli \
  --ckpt_file ckpts/WaveGen/model_1300000.safetensors \
  --model_cfg ckpts/WaveGen/WaveGen_SF_Emilia_24k_50Hz_IBDrop02.yaml \
  --vocab_file ckpts/WaveGen/vocab.txt \
  --cfg_schedule scheduled_k3 \
  --cfg_strength 1.25 \
  --ref_audio reference.wav \
  --ref_text "The transcript of the reference recording." \
  --gen_text "The text to synthesize."
```

Both examples use EMA weights, **Prompt Anchor** (`Prompt_xt_Anchor`), **Scheduled K3** (base CFG 1.25), 50 steps, Mel temperature 0, and seed 0. The supplied models use inference bottleneck dropout 0.2. The pre-trained module uses constant CFG 3 when no schedule is specified, so its example selects Scheduled K3 explicitly. CUDA inference uses BF16 autocast with FP32 model weights; CPU inference uses FP32.

### Load the fine-tuned model from Hugging Face

The subfolder paths below select the 1400k model from [LeeSangHoon/WaveGen](https://huggingface.co/LeeSangHoon/WaveGen). The loader resolves `main` to one commit before downloading the three assets; use a full commit SHA instead of `main` to pin a specific release.

```python
import soundfile as sf
import torch
from wavegen_v_spec.infer.utils_infer import WaveGen

torch.set_num_threads(1)
model = WaveGen.from_pretrained(
    "LeeSangHoon/WaveGen",
    revision="main",
    checkpoint_filename="WaveGen_VSpec/model_1400000.safetensors",
    config_filename="WaveGen_VSpec/WaveGen_VSpec.yaml",
    vocab_filename="WaveGen_VSpec/vocab.txt",
    use_ema=True,
)
sample_rate, audio = model.synthesize(
    "reference.wav",
    "The transcript of the reference recording.",
    "The text to synthesize.",
)
sf.write("generated.wav", audio, sample_rate, subtype="PCM_16")
```

For the pre-trained model, download the `WaveGen/` folder and use the pre-trained CLI above. See the [source repository](https://github.com/sh-lee-prml/WaveGen) for the local Python API, Gradio demo, and batch evaluation.

The source checkout defaults to the V-Spec 1400k Safetensors checkpoint. The base
and V-Spec loaders support existing local `.pt` checkpoints through an explicit
checkpoint-path override.

### WaveGen-Flash (4 Steps)

```bash
python -m wavegen_flash.infer.infer_cli \
  --checkpoint ckpts/WaveGen_Flash/model.safetensors \
  --model-config ckpts/WaveGen_Flash/WaveGen_Flash.yaml \
  --vocab-file ckpts/WaveGen_Flash/vocab.txt \
  --ref-audio reference.wav \
  --ref-text "The transcript of the reference recording." \
  --gen-text "The text to synthesize." \
  --output generated_flash.wav
```

Or load the same bundle directly from this repository:

```python
import soundfile as sf
import torch
from wavegen_flash import WaveGenFlash

torch.set_num_threads(1)
model = WaveGenFlash.from_pretrained("LeeSangHoon/WaveGen", revision="main")
sample_rate, audio = model.synthesize(
    "reference.wav",
    "The transcript of the reference recording.",
    "The text to synthesize.",
)
sf.write("generated_flash.wav", audio, sample_rate, subtype="PCM_16")
```

The Flash loader accepts Safetensors only and requires no additional base weights.
Its YAML fixes four Euler calls at `[0, 0.1, 0.25, 0.5]`, ending at 1.0, with
CFG 0 and IB dropout 0. The grid already includes time shift 3; it is not shifted
again during inference. The final Mel output head is skipped by default;
`--compute-final-mel` or `compute_final_mel=True` restores its computation.
The Mel input and shared network remain active on the final call.

The YAML sets `mel_temperature: 0.0` by default, while waveform temperature
remains 1. Mel temperature scales only the initial Gaussian Mel noise;
the clean reference Mel and later predicted Mel states are not scaled.
Both noise draws are preserved even at temperature 0. Use
`--mel-temperature 1` or `model.synthesize(..., mel_temperature=1.0)`
to reproduce the previous initialization with the same weights.
Compared with temperature 1, temperature 0 improves WER/CER, SIM, and
UTMOS on both complete Seed-TTS test sets.

The default duration limit is 30 seconds for prompt plus generated speech.
Split longer text or explicitly increase `max_duration_seconds`; the sampler
does not automatically split text. See the
[Flash guide](https://github.com/sh-lee-prml/WaveGen/blob/main/src/wavegen_flash/README.md)
for the complete inference interface.

## Safeguards and limitations

- Use reference speech with the speaker's informed consent and permission to use the recording.
- Do not use generated speech for deceptive impersonation, fraud, harassment, or fabricated statements presented as authentic speech.
- Clearly identify shared or published outputs as AI-generated audio, and protect reference recordings and personal information.
- Reference quality and transcript accuracy affect the generated speech. Pronunciation errors, omissions, repetitions, and speaker-similarity failures may occur.
- For long passages, split the text into shorter segments; the single-utterance examples above do not automatically split long text.

These usage guidelines are not automatic safety filters. Public-facing services should add appropriate access controls, abuse reporting, and review of suspected misuse.

## License

The model checkpoints are released under [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/).

WaveGen source-code contributions are released under the [MIT License](https://github.com/sh-lee-prml/WaveGen/blob/main/LICENSE). Reused source components retain their original licenses and notices; see the [source-code third-party notices](https://github.com/sh-lee-prml/WaveGen/blob/main/docs/THIRD_PARTY_NOTICES.md).