Open-XTTS-Trainer
Reproducible 3-Component Training Suite for XTTS v2
A modular framework bringing together training and fine-tuning workflows for all three XTTS v2 components: Discrete VAE (DVAE), Autoregressive GPT Backbone (with Vocabulary Extension for new scripts), and the Latent HiFi-GAN Vocoder.
Quick Links
- π Interactive Demo: Open in Google Colab (Run End-to-End Verification)
- π€ Hugging Face Hub: amitbhakt/open-xtts-trainer
Scope & Contribution
The XTTS v2 neural architecture consists of three distinct components:
- Stage 1 β Discrete VAE (DVAE): Compresses raw mel-spectrograms into discrete audio tokens.
- Stage 2 β Autoregressive GPT Backbone: Generates speech tokens conditioned on text and speaker latents (enhanced here with dynamic vocabulary extension for unsupported scripts).
- Stage 3 β Latent HiFi-GAN Vocoder: Synthesizes audio waveforms conditioned on GPT acoustic latents and speaker embeddings.
The Problem Addressed
When Coqui AI shut down, the community was left without official training recipes for the Discrete VAE and the latent-conditioned HiFi-GAN vocoder. Additionally, expanding XTTS v2 to non-Latin scripts (e.g. Arabic, Indic, Thai, Dhivehi) frequently caused embedding out-of-bounds crashes, while repository abandonment led to torchcodec dependency issues on modern Python builds.
The contribution of this repository is unifying, repairing, and open-sourcing the separate component-training workflows into one accessible, reproducible, dependency-rot-free suite.
Component-Level Verification vs. End-to-End Inference:
The benchmarks presented below demonstrate component-level training convergence (gradient flow, stable adversarial discriminator equilibrium, and acoustic reconstruction descent) on real human speech. Full text-to-speech generation integrates these trained components into the standard XTTS inference pipeline.
Empirical Verification & Component Convergence (Real Human Speech)
All components have been empirically verified on an NVIDIA GPU (T4 / CUDA 12) using genuine human speech (LJSpeech benchmark corpus + JFK Inaugural Address). The loss trajectories confirm rapid harmonic adaptation, discriminator balance, and deep acoustic convergence:
| Training Component & Objective | Initial Loss | Final Loss | Convergence Behavior on Human Voice |
|---|---|---|---|
| Stage 3: Latent HiFi-GAN (Generator) | 198.66 | 68.25 | Fast descent matching natural speech formants |
| Stage 3: Latent HiFi-GAN (Mel Loss) | 190.83 | 63.49 | 67% acoustic error reduction on human speech |
| Stage 3: Discriminators (MPD + MSD) | 8.02 | 3.62 | Stable adversarial equilibrium (no mode collapse) |
| Stage 1: Discrete VAE (Mel Recon) | 8.13 | 2.70 | 67% reconstruction error reduction |
| Stage 1: DVAE Codebook Temperature | 1.000 | 0.999 | Controlled Gumbel-Softmax codebook annealing |
Component Workflows
βββββββββββββββββββββββββββββββββββ
β Stage 1: Discrete VAE ββββΊ Mel-Spectrogram Compression & Audio Tokenizer
βββββββββββββββββββββββββββββββββββ
β
βΌ
βββββββββββββββββββββββββββββββββββ
β Stage 2: GPT Alignment ββββΊ Text & Vocab Extension βββΊ Predicts DVAE Codes
βββββββββββββββββββββββββββββββββββ
β
βΌ
βββββββββββββββββββββββββββββββββββ
β Stage 3: Latent HiFi-GAN ββββΊ Converts GPT Latents + Speaker Embeddings βββΊ Waveform
βββββββββββββββββββββββββββββββββββ
| Goal | Required Workflows |
|---|---|
| Voice adaptation / Speaker cloning | Run Stage 2 (GPT) + Stage 3 (HiFi-GAN) |
| New script / language (e.g. Arabic, Hindi, Dhivehi) | Run Stage 2 (with Vocab Extender) + Stage 3 (HiFi-GAN) |
| Custom audio codec / Sample rate change (e.g. 48kHz) | Run Stage 1 (DVAE from scratch) + Stage 2 + Stage 3 |
Installation
git clone https://github.com/amitbhakt/open-xtts-trainer.git
cd open-xtts-trainer
pip install -e .
Quickstart
1. Preparing Speech Data
Download public-domain benchmark human speech (LJSpeech sentences + JFK address resampled to 24kHz mono) or slice a 5-minute continuous speech dataset:
# 8-sample micro benchmark
python -m xtts_trainer.speech_data --num_samples 8 --output_dir sample_audio
# Scaled 5-minute speech dataset (100 clean clips)
python -m xtts_trainer.speech_data --minutes 5 --output_dir sample_audio
2. Stage 2: Vocabulary Extender & GPT Backbone Fine-Tuning
Extend Tokenizer for a New Language Script:
python xtts_trainer/stage2_gpt/vocab_extender.py \ --target_text /path/to/target_corpus.txt \ --base_vocab xtts_base/vocab.json \ --output_vocab xtts_base/vocab_extended.jsonFine-tune the Autoregressive GPT Backbone:
python xtts_trainer/stage2_gpt/train.py \ --audio_dir sample_audio \ --vocab_path xtts_base/vocab_extended.json \ --output_dir checkpoints/gpt \ --epochs 10
3. Stage 3: Latent HiFi-GAN Vocoder
Extract GPT Latents from your audio dataset:
python xtts_trainer/stage3_hifigan/extract_latents.py \ --audio_dir sample_audio \ --output_dir extracted_latents \ --device cudaTrain the HiFi-GAN Vocoder:
python xtts_trainer/stage3_hifigan/train.py \ --audio_dir sample_audio \ --latents_dir extracted_latents \ --output_dir checkpoints/hifigan \ --batch_size 8 \ --epochs 25 \ --device cudaSynthesize Waveforms with Trained Vocoder:
python -m xtts_trainer.stage3_hifigan.inference \ --checkpoint checkpoints/hifigan/hifigan_epoch_25.pt \ --latent extracted_latents/speech_001.pt \ --output synthesized_voice.wav
4. Stage 1: Discrete VAE (DVAE) Training
python xtts_trainer/stage1_dvae/train.py \
--audio_dir sample_audio \
--output_dir checkpoints/dvae \
--continue_path xtts_base/dvae.pth \
--batch_size 8 \
--epochs 25 \
--device cuda
5-Point End-to-End Pipeline Verification Proof
To establish reproducible execution proof across all three trained components, run the instrumented integration proof suite:
python scripts/verify_pipeline_proof.py \
--dvae_checkpoint checkpoints/dvae/dvae.pth \
--gpt_checkpoint checkpoints/gpt/model.pth \
--hifigan_checkpoint checkpoints/hifigan/hifigan.pth \
--reference_audio sample_audio/speech_003.wav \
--output_dir proof_outputs
What It Verifies:
| Proof Gate | Execution Assertion | What It Establishes |
|---|---|---|
| Gate 1: Stage 1 (DVAE) | Encodes audio $\to$ discrete codebook IDs $\in [0, 1023]$ $\to$ decodes mel | Verifies acoustic tokenizer and reconstruction fidelity |
| Gate 2: Stage 2 (GPT) | Loads compatible trained parameters & extracts reference latents | Confirms conditioning flow with zero random-latent fallbacks |
| Gate 3: Stage 3 (HiFi-GAN) | Loads trained decoder with strict=True ($100%$ parameter match) |
Guarantees audio is decoded by your checkpoint (zero bypass) |
| Gate 4: Multi-Sentence Suite | Synthesizes sibilant (s1_sibilant), fluent (s2_fluent), and benchmark (s3_benchmark) sentences |
Evaluates phonetic stability across continuous text |
| Gate 5: Audio Quality Proof | Asserts finite waveform, healthy amplitude ($\text{RMS} \ge 0.02$), non-clipping peak ($\le 1.05$), and measures $>4\text{ kHz}$ sibilant energy | Eliminates vocoder amplitude collapse and validates phase health |
Execution outputs and verification metrics are automatically recorded to proof_outputs/pipeline_proof_manifest.json.
Citation
If you use this training suite in your academic work or project, please cite:
@software{bhakt2026openxtts,
author = {Bhakt, Amit},
title = {open-xtts-trainer: Reproducible 3-Component Training Suite for XTTS v2},
year = {2026},
doi = {10.5281/zenodo.23573888},
url = {https://zenodo.org/records/23573888}
}
License
This project is licensed under the Mozilla Public License 2.0 (MPL-2.0).
