|
Download README.md from LeeSangHoon/WaveGen: direct link, hf CLI and curl.
- Browser
- Download file 12.2 kB
-
https://huggingface.co/LeeSangHoon/WaveGen/resolve/main/README.md
- Command line
-
hf download hf://LeeSangHoon/WaveGen/README.md
-
curl -L -o README.md https://huggingface.co/LeeSangHoon/WaveGen/resolve/main/README.md
12.2 kB
| license: cc-by-nc-4.0 | |
| pipeline_tag: text-to-speech | |
| library_name: wavegen | |
| language: | |
| - en | |
| - zh | |
| datasets: | |
| - amphion/Emilia-Dataset | |
| tags: | |
| - WaveGen | |
| - TTS | |
| - zero-shot | |
| - voice-cloning | |
| - audio | |
| - multilingual | |
| - pytorch | |
| # WaveGen [NeurIPS 2026] | |
| **WaveGen: Truly End-to-End Waveform Generation via Internal Spectral Trajectory Learning** | |
| Sang-Hoon Lee and Ha-Yeong Choi | |
| Ajou University and KT | |
| [Paper](https://arxiv.org/abs/2611.12527) · [Source Code](https://github.com/sh-lee-prml/WaveGen) · [Live Demo](https://huggingface.co/spaces/LeeSangHoon/WaveGen) · [Demo Page](https://wavegen26.github.io/demo) · [Benchmark Demo](https://sh-lee-prml.github.io/wavegen-tts-demo/) · [Spectral Forcing Demo](https://wavegen26.github.io/demo/spectral_forcing_visualization/) | |
| WaveGen generates speech directly from text and reference audio using Spectral Forcing. The models are trained on English and Mandarin Chinese speech from Emilia, using **24 kHz audio** and **50 Hz waveform patches** (480 samples per patch). | |
| ## Models | |
| This repository contains the pre-trained, V-Spec, and four-step Flash checkpoints in separate folders. | |
| | Model | Checkpoint | Folder | Inference steps | | |
| |---|---|---|---| | |
| | WaveGen (Pre-trained) | 1300k EMA | `WaveGen/` | 50 | | |
| | **WaveGen (Fine-tuned with v-spec losses) — Recommended** | **1400k EMA** | **`WaveGen_VSpec/`** | **50** | | |
| | WaveGen-Flash | 48k EMA | `WaveGen_Flash/` | **4 (fixed)** | | |
| **WaveGen (Fine-tuned with v-spec losses)** is the **recommended model for inference** and the default in the source repository. All checkpoints contain **FP32 EMA weights only**, in Safetensors format. Optimizer state and other training state are excluded; these files are intended for inference, not for resuming a training run. | |
| **WaveGen-Flash** is optimized for four-step speech generation. Its complete merged | |
| weights load from one Safetensors file, together with its YAML and vocabulary. | |
| The Flash release includes inference code and model weights; its training recipe | |
| is not included. | |
| ## Seed-TTS evaluation | |
| Results on the complete standard Seed-TTS test sets for the EMA weights released here. | |
| The pre-trained and V-Spec models use **Prompt Anchor**, **Scheduled K3** (base CFG 1.25), **50 Euler steps**, | |
| Mel temperature 0, inference bottleneck dropout 0.2, and seed 0, with FP32 model | |
| weights and BF16 CUDA inference. The original test prompts are used. | |
| Flash uses **four Euler steps**, the time grid **`[0, 0.1, 0.25, 0.5, 1.0]`**, | |
| CFG 0, inference bottleneck dropout 0, waveform temperature 1, **initial Mel | |
| temperature 0**, and seed 0. It uses the same original test prompts and | |
| FP32-weight/BF16-inference precision. Prompt states interpolate from their | |
| initial noise toward the reference at each step; the initial Mel noise is | |
| scaled by the Mel temperature. The Flash results below use the new default | |
| temperature 0. WER/CER and UTMOS are truncated to two decimal places, | |
| and SIM to three decimal places (not rounded). | |
| The complete test sets contain 1,088 English and 2,020 Chinese utterances. | |
| <table> | |
| <colgroup span="1"></colgroup> | |
| <colgroup span="3"></colgroup> | |
| <colgroup span="3"></colgroup> | |
| <thead> | |
| <tr> | |
| <th rowspan="2" scope="col" align="left">Model</th> | |
| <th colspan="3" scope="colgroup" align="center">EN</th> | |
| <th colspan="3" scope="colgroup" align="center">ZH</th> | |
| </tr> | |
| <tr> | |
| <th scope="col" align="center">WER (%) ↓</th> | |
| <th scope="col" align="center">SIM ↑</th> | |
| <th scope="col" align="center">UTMOS ↑</th> | |
| <th scope="col" align="center">CER (%) ↓</th> | |
| <th scope="col" align="center">SIM ↑</th> | |
| <th scope="col" align="center">UTMOS ↑</th> | |
| </tr> | |
| </thead> | |
| <tbody> | |
| <tr> | |
| <th scope="row" align="left"><strong>WaveGen</strong><br><sub>Pre-trained · 1300k EMA</sub></th> | |
| <td align="center">1.47</td> | |
| <td align="center">0.755</td> | |
| <td align="center">3.68</td> | |
| <td align="center">1.75</td> | |
| <td align="center">0.805</td> | |
| <td align="center">2.87</td> | |
| </tr> | |
| <tr> | |
| <th scope="row" align="left"><strong>WaveGen V-Spec</strong><br><sub>1400k EMA · Recommended</sub></th> | |
| <td align="center">1.39</td> | |
| <td align="center">0.743</td> | |
| <td align="center">3.59</td> | |
| <td align="center">1.65</td> | |
| <td align="center">0.783</td> | |
| <td align="center">2.79</td> | |
| </tr> | |
| <tr> | |
| <th scope="row" align="left"><strong>WaveGen-Flash</strong><br><sub>48k EMA · 4 steps</sub></th> | |
| <td align="center">1.49</td> | |
| <td align="center">0.727</td> | |
| <td align="center">3.53</td> | |
| <td align="center">1.59</td> | |
| <td align="center">0.768</td> | |
| <td align="center">2.74</td> | |
| </tr> | |
| </tbody> | |
| </table> | |
| **WaveGen achieves state-of-the-art performance among TTS models trained on the Emilia dataset.** | |
| WER is reported for English and CER for Chinese. Error rates are percentages; | |
| all metrics are macro-averaged over utterances. The Safetensors exports preserve | |
| the FP32 EMA inference tensor values. | |
| ## Download | |
| Download the checkpoint, YAML, and vocabulary from the desired model folder and place that folder under `ckpts/` in the WaveGen source repository: | |
| ```text | |
| ckpts/ | |
| ├── WaveGen/ | |
| │ ├── model_1300000.safetensors | |
| │ ├── WaveGen_SF_Emilia_24k_50Hz_IBDrop02.yaml | |
| │ └── vocab.txt | |
| ├── WaveGen_VSpec/ | |
| │ ├── model_1400000.safetensors | |
| │ ├── WaveGen_VSpec.yaml | |
| │ └── vocab.txt | |
| └── WaveGen_Flash/ | |
| ├── model.safetensors | |
| ├── WaveGen_Flash.yaml | |
| └── vocab.txt | |
| ``` | |
| Keep each checkpoint together with its matching configuration and vocabulary. This Hub repository uses `WaveGen/`, `WaveGen_VSpec/`, and `WaveGen_Flash/` at its root; `ckpts/` is their destination in your local source checkout. | |
| ## Usage | |
| Follow the [installation instructions](https://github.com/sh-lee-prml/WaveGen#installation), then run from the WaveGen source repository root. Supply a reference recording, its exact transcript, and the text to synthesize. Automatic transcription is not required. | |
| ### Fine-tuned model (recommended default) | |
| ```bash | |
| python -m wavegen_v_spec.infer.infer_cli \ | |
| --ckpt_file ckpts/WaveGen_VSpec/model_1400000.safetensors \ | |
| --model_cfg ckpts/WaveGen_VSpec/WaveGen_VSpec.yaml \ | |
| --vocab_file ckpts/WaveGen_VSpec/vocab.txt \ | |
| --ref_audio reference.wav \ | |
| --ref_text "The transcript of the reference recording." \ | |
| --gen_text "The text to synthesize." | |
| ``` | |
| ### Pre-trained model | |
| ```bash | |
| python -m wavegen.infer.infer_cli \ | |
| --ckpt_file ckpts/WaveGen/model_1300000.safetensors \ | |
| --model_cfg ckpts/WaveGen/WaveGen_SF_Emilia_24k_50Hz_IBDrop02.yaml \ | |
| --vocab_file ckpts/WaveGen/vocab.txt \ | |
| --cfg_schedule scheduled_k3 \ | |
| --cfg_strength 1.25 \ | |
| --ref_audio reference.wav \ | |
| --ref_text "The transcript of the reference recording." \ | |
| --gen_text "The text to synthesize." | |
| ``` | |
| Both examples use EMA weights, **Prompt Anchor** (`Prompt_xt_Anchor`), **Scheduled K3** (base CFG 1.25), 50 steps, Mel temperature 0, and seed 0. The supplied models use inference bottleneck dropout 0.2. The pre-trained module uses constant CFG 3 when no schedule is specified, so its example selects Scheduled K3 explicitly. CUDA inference uses BF16 autocast with FP32 model weights; CPU inference uses FP32. | |
| ### Load the fine-tuned model from Hugging Face | |
| The subfolder paths below select the 1400k model from [LeeSangHoon/WaveGen](https://huggingface.co/LeeSangHoon/WaveGen). The loader resolves `main` to one commit before downloading the three assets; use a full commit SHA instead of `main` to pin a specific release. | |
| ```python | |
| import soundfile as sf | |
| import torch | |
| from wavegen_v_spec.infer.utils_infer import WaveGen | |
| torch.set_num_threads(1) | |
| model = WaveGen.from_pretrained( | |
| "LeeSangHoon/WaveGen", | |
| revision="main", | |
| checkpoint_filename="WaveGen_VSpec/model_1400000.safetensors", | |
| config_filename="WaveGen_VSpec/WaveGen_VSpec.yaml", | |
| vocab_filename="WaveGen_VSpec/vocab.txt", | |
| use_ema=True, | |
| ) | |
| sample_rate, audio = model.synthesize( | |
| "reference.wav", | |
| "The transcript of the reference recording.", | |
| "The text to synthesize.", | |
| ) | |
| sf.write("generated.wav", audio, sample_rate, subtype="PCM_16") | |
| ``` | |
| For the pre-trained model, download the `WaveGen/` folder and use the pre-trained CLI above. See the [source repository](https://github.com/sh-lee-prml/WaveGen) for the local Python API, Gradio demo, and batch evaluation. | |
| The source checkout defaults to the V-Spec 1400k Safetensors checkpoint. The base | |
| and V-Spec loaders support existing local `.pt` checkpoints through an explicit | |
| checkpoint-path override. | |
| ### WaveGen-Flash (4 Steps) | |
| ```bash | |
| python -m wavegen_flash.infer.infer_cli \ | |
| --checkpoint ckpts/WaveGen_Flash/model.safetensors \ | |
| --model-config ckpts/WaveGen_Flash/WaveGen_Flash.yaml \ | |
| --vocab-file ckpts/WaveGen_Flash/vocab.txt \ | |
| --ref-audio reference.wav \ | |
| --ref-text "The transcript of the reference recording." \ | |
| --gen-text "The text to synthesize." \ | |
| --output generated_flash.wav | |
| ``` | |
| Or load the same bundle directly from this repository: | |
| ```python | |
| import soundfile as sf | |
| import torch | |
| from wavegen_flash import WaveGenFlash | |
| torch.set_num_threads(1) | |
| model = WaveGenFlash.from_pretrained("LeeSangHoon/WaveGen", revision="main") | |
| sample_rate, audio = model.synthesize( | |
| "reference.wav", | |
| "The transcript of the reference recording.", | |
| "The text to synthesize.", | |
| ) | |
| sf.write("generated_flash.wav", audio, sample_rate, subtype="PCM_16") | |
| ``` | |
| The Flash loader accepts Safetensors only and requires no additional base weights. | |
| Its YAML fixes four Euler calls at `[0, 0.1, 0.25, 0.5]`, ending at 1.0, with | |
| CFG 0 and IB dropout 0. The grid already includes time shift 3; it is not shifted | |
| again during inference. The final Mel output head is skipped by default; | |
| `--compute-final-mel` or `compute_final_mel=True` restores its computation. | |
| The Mel input and shared network remain active on the final call. | |
| The YAML sets `mel_temperature: 0.0` by default, while waveform temperature | |
| remains 1. Mel temperature scales only the initial Gaussian Mel noise; | |
| the clean reference Mel and later predicted Mel states are not scaled. | |
| Both noise draws are preserved even at temperature 0. Use | |
| `--mel-temperature 1` or `model.synthesize(..., mel_temperature=1.0)` | |
| to reproduce the previous initialization with the same weights. | |
| Compared with temperature 1, temperature 0 improves WER/CER, SIM, and | |
| UTMOS on both complete Seed-TTS test sets. | |
| The default duration limit is 30 seconds for prompt plus generated speech. | |
| Split longer text or explicitly increase `max_duration_seconds`; the sampler | |
| does not automatically split text. See the | |
| [Flash guide](https://github.com/sh-lee-prml/WaveGen/blob/main/src/wavegen_flash/README.md) | |
| for the complete inference interface. | |
| ## Safeguards and limitations | |
| - Use reference speech with the speaker's informed consent and permission to use the recording. | |
| - Do not use generated speech for deceptive impersonation, fraud, harassment, or fabricated statements presented as authentic speech. | |
| - Clearly identify shared or published outputs as AI-generated audio, and protect reference recordings and personal information. | |
| - Reference quality and transcript accuracy affect the generated speech. Pronunciation errors, omissions, repetitions, and speaker-similarity failures may occur. | |
| - For long passages, split the text into shorter segments; the single-utterance examples above do not automatically split long text. | |
| These usage guidelines are not automatic safety filters. Public-facing services should add appropriate access controls, abuse reporting, and review of suspected misuse. | |
| ## License | |
| The model checkpoints are released under [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/). | |
| WaveGen source-code contributions are released under the [MIT License](https://github.com/sh-lee-prml/WaveGen/blob/main/LICENSE). Reused source components retain their original licenses and notices; see the [source-code third-party notices](https://github.com/sh-lee-prml/WaveGen/blob/main/docs/THIRD_PARTY_NOTICES.md). | |