x-vits / README.md
fdemelo's picture
Model card: style_strength, guidance and temperature options
e742eec verified
|
Raw History Blame Contribute Delete
17.1 kB
---
license: cc-by-nc-4.0
language:
- en
pipeline_tag: text-to-speech
library_name: onnx
tags:
- text-to-speech
- tts
- piper
- vits
- onnx
- prosody
- expressive-speech
- emotional-tts
- flow-matching
datasets:
- ylacombe/expresso
---
# x-vits · Expresso ex02 (English, expressive)
An expressive **[Piper](https://github.com/rhasspy/piper)/VITS** voice that can speak in **six styles** and
**stress the words you mark**, or the words it thinks matter. Made with
**[x-vits](https://github.com/femelo/x-vits)**, a recipe that adds prosody modelling (intonation, rhythm,
emphasis, vocal effort) to small, fast VITS voices. It's about 25M parameters in all, exported to ONNX, and runs on a CPU.
**Version 0.2.** Needs the x-vits 0.2 runtime (`pip install` from the GitHub repo, below).
## What's different from a plain Piper voice
- 🎭 **Styles:** `default`, `happy`, `sad`, `confused`, `enunciated`, `laughing`.
- 💥 **Emphasis:** write `*word*` to stress a word. Without marks, a small model predicts which words to stress.
- 🌊 **Sampled intonation:** a flow-matching model picks a plausible pitch, voicing, energy and vocal-effort contour per phoneme, instead of averaging every sentence into the same melody.
- 📣 **Vocal effort:** how bright and pressed the voice is, learned per style: happy and laughing are brighter, sad is darker.
- 🎚️ **Style-aware timing:** sad slows down, confused hesitates, enunciated takes its pauses.
## What's new in 0.2
- **Vocal effort** as a prosody feature, measured on voiced sounds and bridged smoothly across consonants and pauses.
- **A wider pitch range** in training: the speaker's high notes (happy, laughing) were clipped before.
- **Voicing follows the phonemes**, which removes occasional blips (a voiced *k*, a vowel that dropped its voice).
- **Louder emphasised words**, peaking with the pitch.
The 0.1 voice is still playable below, next to 0.2, for comparison. Its files are in this repo's history.
## 🎧 Samples
**About the comparison.** "Piper" below is not a flat, neutral voice. It's the same speaker, fine-tuned
with plain Piper/VITS on the *same* expressive recordings in all six styles, so it already has lively
prosody: VITS samples its own timing and intonation, and this one learned an average of the six
speaking manners. What it can't do is choose: there is no way to ask it for "sad" or to stress a
particular word. x-vits adds that control. All synthesised samples use the same random seed and
none were cherry-picked.
### One sentence, six styles
Plain Piper has a single way to read *"You broke the tie score!"*. x-vits reads it in whichever style
you ask for. The speaker's own recordings of this held-out sentence are on the right.
<table>
<tr><th colspan="4">Piper (any style): <audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/same_piper.mp3"></audio></th></tr>
<tr><th>style</th><th>x-vits 0.2</th><th>x-vits 0.1</th><th>recording</th></tr>
<tr><td><b>default</b></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/same_default_xvits.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/v1/same_default_xvits.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/same_default_recording.mp3"></audio></td></tr>
<tr><td><b>happy</b></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/same_happy_xvits.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/v1/same_happy_xvits.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/same_happy_recording.mp3"></audio></td></tr>
<tr><td><b>sad</b></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/same_sad_xvits.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/v1/same_sad_xvits.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/same_sad_recording.mp3"></audio></td></tr>
<tr><td><b>confused</b></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/same_confused_xvits.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/v1/same_confused_xvits.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/same_confused_recording.mp3"></audio></td></tr>
<tr><td><b>enunciated</b></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/same_enunciated_xvits.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/v1/same_enunciated_xvits.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/same_enunciated_recording.mp3"></audio></td></tr>
<tr><td><b>laughing</b></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/same_laughing_xvits.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/v1/same_laughing_xvits.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/same_laughing_recording.mp3"></audio></td></tr>
</table>
Measured on these clips, 0.2 follows the speaker more closely than 0.1 in several styles. Happy has
the speaker's wide pitch range (15.0 semitones; recording 14.8, 0.1 12.5), sad is as narrow as the
recording (5.1 semitones; 0.1 6.7), confused slows down like the speaker (2.30 s; recording 2.53,
0.1 1.94), and laughing moves much more (7.3 semitones; 0.1 4.2, recording 12.6). The voice still
can't produce actual laughter.
### Different sentences, each in its recorded style
Held-out sentences, never seen in training, each with the recording of that sentence in that style.
<table>
<tr><th>style</th><th>text</th><th>Piper</th><th>x-vits 0.2</th><th>x-vits 0.1</th><th>recording</th></tr>
<tr><td><b>default</b></td><td>I'm more of an all terrain vehicle type.</td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/default_piper.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/default_xvits.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/v1/default_xvits.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/default_recording.mp3"></audio></td></tr>
<tr><td><b>happy</b></td><td>We didn't get the weather right, guys.</td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/happy_piper.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/happy_xvits.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/v1/happy_xvits.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/happy_recording.mp3"></audio></td></tr>
<tr><td><b>sad</b></td><td>No, in Salt Lake City, today, it won't snow.</td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/sad_piper.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/sad_xvits.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/v1/sad_xvits.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/sad_recording.mp3"></audio></td></tr>
<tr><td><b>confused</b></td><td>Is young adult fiction very popular?</td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/confused_piper.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/confused_xvits.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/v1/confused_xvits.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/confused_recording.mp3"></audio></td></tr>
<tr><td><b>enunciated</b></td><td>One PM to nine PM, it'll be cloudy, and in the low fifties.</td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/enunciated_piper.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/enunciated_xvits.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/v1/enunciated_xvits.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/enunciated_recording.mp3"></audio></td></tr>
<tr><td><b>laughing</b></td><td>Their moment had come, or so they said.</td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/laughing_piper.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/laughing_xvits.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/v1/laughing_xvits.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/laughing_recording.mp3"></audio></td></tr>
</table>
### Emphasis, by moving the `*mark*` (style `default`)
Piper has no equivalent: it stresses whatever its training suggests.
<table>
<tr><th>text</th><th>x-vits 0.2</th><th>x-vits 0.1</th></tr>
<tr><td>I never said she stole my money.</td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/emphasis_0.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/v1/emphasis_0.mp3"></audio></td></tr>
<tr><td><b>*I*</b> never said she stole my money.</td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/emphasis_1.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/v1/emphasis_1.mp3"></audio></td></tr>
<tr><td>I never said <b>*she*</b> stole my money.</td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/emphasis_2.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/v1/emphasis_2.mp3"></audio></td></tr>
<tr><td>I never said she stole my <b>*money*</b>.</td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/emphasis_3.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/v1/emphasis_3.mp3"></audio></td></tr>
</table>
## 🚀 Usage
```bash
sudo apt install libespeak-ng1 # phonemizer backend (macOS: brew install espeak-ng)
pip install "x-vits @ git+https://github.com/femelo/x-vits"
```
```python
import soundfile as sf
from xvits.runtime import XVits
voice = XVits.from_pretrained("fdemelo/x-vits")
audio = voice.synthesize("We didn't get the weather right, guys.", style="happy")
sf.write("happy.wav", audio, voice.sample_rate)
# explicit emphasis, a different style, a fixed seed and a faster CPU setting
audio = voice.synthesize("I never said *she* stole my money.", style="confused", seed=7, steps=4)
# a stronger style and a bigger pitch peak on the stressed word
audio = voice.synthesize("I *really* didn't expect that.", style="happy", style_strength=1.5, emphasis_scale=2.0)
sf.write("confused.wav", audio, voice.sample_rate)
```
Or from the command line:
```bash
xvits-speak --model fdemelo/x-vits --text "Is it *really* you?" --style sad --out sad.wav
```
| option | values | |
|---|---|---|
| `style` | `default` `happy` `sad` `confused` `enunciated` `laughing` | none gives a neutral mix |
| `emphasis` | `auto` · `marked` · `none` | `auto`: marks if present, otherwise predicted |
| `emphasis_scale` | default 1.5 | multiplies the emphasis pitch peak, lengthening and loudness; 1.0 = average emphasis in the recordings |
| `style_strength` | default 1.0 | style exaggeration: 0 none, 2 double (happy: +2.2, +3.5, +4.8 semitones above the speaker's average at 0, 1, 2) |
| `guidance` | `"timing,pitch,energy,effort"` | per-channel exaggeration; default `"1.0,2.0,4.0,1.0"` |
| `temperature` | `"timing,pitch,energy"` | variety; default `"0.8,0.7,1.0"`, higher is livelier |
| `length_scale` | default 1.08 | larger is slower |
| `steps` | default 16 | 4 is about 2.5× faster on CPU and changes pitch by about 0.1 semitone |
| `seed` | int | reproducible output |
Speed with onnxruntime on 4 CPU threads: real-time factor about 0.6 (16 steps) or 0.25 (4 steps).
## 📦 Files
| file | role | size |
|---|---|---|
| `encoder.onnx` | phonemes → text states | 26 MB |
| `predictor.onnx` | one step of the flow-matching prosody predictor (style-conditioned) | 31 MB |
| `durations.onnx` | style-aware stochastic duration predictor | 2.5 MB |
| `acoustic.onnx` | prosody-conditioned prior, length regulation, flow, HiFi-GAN | 36 MB |
| `prominence.onnx` | predicts which words to stress | 4.8 MB |
| `dp_cond.npz`, `emphasis_template.json` | style vectors for the duration predictor; the emphasis shape | small |
| `runtime.json`, `config.json` | runtime settings; Piper phoneme map (eSpeak `en`) | |
These graphs are driven by `xvits.runtime`, so they aren't drop-in files for Piper's own runtime.
The export was verified against PyTorch end to end (maximum difference about 3e-5 at several sentence lengths).
## 🧪 Training
- **Base:** Piper's LJSpeech `medium` checkpoint (LJSpeech is public domain).
- **Data:** [Expresso](https://speechbot.github.io/expresso/) speaker **ex02**, read speech in six styles,
2.2 h after leaving out whisper, singing, long-form and `<laugh>` rows. 245 utterances held out.
- **Recipe** ([x-vits](https://github.com/femelo/x-vits) 0.2, `xvits-train`):
1. fine-tune the base checkpoint on ex02;
2. extract per-phoneme duration, pitch, energy and vocal-effort targets;
3. fine-tune with a zero-initialised prosody input to the prior, including frame-level interpolation;
4. train an OT-CFM prosody predictor with a style input;
5. train a style-conditioned duration predictor;
6. train a word-prominence student on acoustic labels;
7. fit an emphasis template (pitch, duration, loudness) anchored on the stressed vowel, from Expresso's `*emphasis*` marks.
**Held-out results** (129 sentences, two seeds; pitch and timing match are correlations with the recording of the same sentence):
| | pitch match | timing match | WER (Whisper) | UTMOS |
|---|---|---|---|---|
| Piper, fine-tuned | 0.37 | 0.30 | 1.7 % | 4.08 |
| x-vits 0.1 | 0.37 | 0.39 | 2.1 % | 4.01 |
| **x-vits 0.2** | **0.40** | 0.37 | 3.6 % | 3.86 |
| the recordings | | | | 4.12 |
UTMOS and Whisper both prefer calm read speech: UTMOS gives the speaker's own laughing takes 2.49, and
0.2 loses most of its score in happy and laughing, the styles that became more expressive. Listen
above and decide for yourself.
The prominence student finds the recording's main stress 42 % of the time, against 34 % for "last word" and 15 % for a random word.
## ⚠️ License and limitations
- **CC BY-NC 4.0.** Expresso is licensed for non-commercial use only, and so is this voice. The x-vits code is MIT.
- English, one female speaker, read speech. Styles are as recorded in Expresso; there are no other emotions.
- Laughing is rendered through intonation, timing and brightness; the voice doesn't produce actual laughter.
- Questions sometimes end with a smaller rise than a natural reading.
- Don't use this voice to impersonate real people or to mislead listeners.
## Citation
If you use this voice, please cite Expresso:
```bibtex
@inproceedings{nguyen2023expresso,
title = {EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesis},
author = {Nguyen, Tu Anh and Hsu, Wei-Ning and D'Avirro, Antony and Shi, Bowen and Gat, Itai and
Fazel-Zarani, Maryam and Remez, Tal and Copet, Jade and Synnaeve, Gabriel and Hassid, Michael and
Kreuk, Felix and Adi, Yossi and Dupoux, Emmanuel},
booktitle = {Interspeech},
year = {2023}
}
```