Instructions to use fdemelo/x-vits with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Piper
How to use fdemelo/x-vits with Piper:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
|
Download README.md from fdemelo/x-vits: direct link, hf CLI and curl.
- Browser
- Download file 17.1 kB
-
https://huggingface.co/fdemelo/x-vits/resolve/main/README.md
- Command line
-
hf download hf://fdemelo/x-vits/README.md
-
curl -L -o README.md https://huggingface.co/fdemelo/x-vits/resolve/main/README.md
17.1 kB
| license: cc-by-nc-4.0 | |
| language: | |
| - en | |
| pipeline_tag: text-to-speech | |
| library_name: onnx | |
| tags: | |
| - text-to-speech | |
| - tts | |
| - piper | |
| - vits | |
| - onnx | |
| - prosody | |
| - expressive-speech | |
| - emotional-tts | |
| - flow-matching | |
| datasets: | |
| - ylacombe/expresso | |
| # x-vits · Expresso ex02 (English, expressive) | |
| An expressive **[Piper](https://github.com/rhasspy/piper)/VITS** voice that can speak in **six styles** and | |
| **stress the words you mark**, or the words it thinks matter. Made with | |
| **[x-vits](https://github.com/femelo/x-vits)**, a recipe that adds prosody modelling (intonation, rhythm, | |
| emphasis, vocal effort) to small, fast VITS voices. It's about 25M parameters in all, exported to ONNX, and runs on a CPU. | |
| **Version 0.2.** Needs the x-vits 0.2 runtime (`pip install` from the GitHub repo, below). | |
| ## What's different from a plain Piper voice | |
| - 🎭 **Styles:** `default`, `happy`, `sad`, `confused`, `enunciated`, `laughing`. | |
| - 💥 **Emphasis:** write `*word*` to stress a word. Without marks, a small model predicts which words to stress. | |
| - 🌊 **Sampled intonation:** a flow-matching model picks a plausible pitch, voicing, energy and vocal-effort contour per phoneme, instead of averaging every sentence into the same melody. | |
| - 📣 **Vocal effort:** how bright and pressed the voice is, learned per style: happy and laughing are brighter, sad is darker. | |
| - 🎚️ **Style-aware timing:** sad slows down, confused hesitates, enunciated takes its pauses. | |
| ## What's new in 0.2 | |
| - **Vocal effort** as a prosody feature, measured on voiced sounds and bridged smoothly across consonants and pauses. | |
| - **A wider pitch range** in training: the speaker's high notes (happy, laughing) were clipped before. | |
| - **Voicing follows the phonemes**, which removes occasional blips (a voiced *k*, a vowel that dropped its voice). | |
| - **Louder emphasised words**, peaking with the pitch. | |
| The 0.1 voice is still playable below, next to 0.2, for comparison. Its files are in this repo's history. | |
| ## 🎧 Samples | |
| **About the comparison.** "Piper" below is not a flat, neutral voice. It's the same speaker, fine-tuned | |
| with plain Piper/VITS on the *same* expressive recordings in all six styles, so it already has lively | |
| prosody: VITS samples its own timing and intonation, and this one learned an average of the six | |
| speaking manners. What it can't do is choose: there is no way to ask it for "sad" or to stress a | |
| particular word. x-vits adds that control. All synthesised samples use the same random seed and | |
| none were cherry-picked. | |
| ### One sentence, six styles | |
| Plain Piper has a single way to read *"You broke the tie score!"*. x-vits reads it in whichever style | |
| you ask for. The speaker's own recordings of this held-out sentence are on the right. | |
| <table> | |
| <tr><th colspan="4">Piper (any style): <audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/same_piper.mp3"></audio></th></tr> | |
| <tr><th>style</th><th>x-vits 0.2</th><th>x-vits 0.1</th><th>recording</th></tr> | |
| <tr><td><b>default</b></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/same_default_xvits.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/v1/same_default_xvits.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/same_default_recording.mp3"></audio></td></tr> | |
| <tr><td><b>happy</b></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/same_happy_xvits.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/v1/same_happy_xvits.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/same_happy_recording.mp3"></audio></td></tr> | |
| <tr><td><b>sad</b></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/same_sad_xvits.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/v1/same_sad_xvits.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/same_sad_recording.mp3"></audio></td></tr> | |
| <tr><td><b>confused</b></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/same_confused_xvits.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/v1/same_confused_xvits.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/same_confused_recording.mp3"></audio></td></tr> | |
| <tr><td><b>enunciated</b></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/same_enunciated_xvits.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/v1/same_enunciated_xvits.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/same_enunciated_recording.mp3"></audio></td></tr> | |
| <tr><td><b>laughing</b></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/same_laughing_xvits.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/v1/same_laughing_xvits.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/same_laughing_recording.mp3"></audio></td></tr> | |
| </table> | |
| Measured on these clips, 0.2 follows the speaker more closely than 0.1 in several styles. Happy has | |
| the speaker's wide pitch range (15.0 semitones; recording 14.8, 0.1 12.5), sad is as narrow as the | |
| recording (5.1 semitones; 0.1 6.7), confused slows down like the speaker (2.30 s; recording 2.53, | |
| 0.1 1.94), and laughing moves much more (7.3 semitones; 0.1 4.2, recording 12.6). The voice still | |
| can't produce actual laughter. | |
| ### Different sentences, each in its recorded style | |
| Held-out sentences, never seen in training, each with the recording of that sentence in that style. | |
| <table> | |
| <tr><th>style</th><th>text</th><th>Piper</th><th>x-vits 0.2</th><th>x-vits 0.1</th><th>recording</th></tr> | |
| <tr><td><b>default</b></td><td>I'm more of an all terrain vehicle type.</td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/default_piper.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/default_xvits.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/v1/default_xvits.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/default_recording.mp3"></audio></td></tr> | |
| <tr><td><b>happy</b></td><td>We didn't get the weather right, guys.</td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/happy_piper.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/happy_xvits.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/v1/happy_xvits.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/happy_recording.mp3"></audio></td></tr> | |
| <tr><td><b>sad</b></td><td>No, in Salt Lake City, today, it won't snow.</td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/sad_piper.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/sad_xvits.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/v1/sad_xvits.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/sad_recording.mp3"></audio></td></tr> | |
| <tr><td><b>confused</b></td><td>Is young adult fiction very popular?</td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/confused_piper.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/confused_xvits.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/v1/confused_xvits.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/confused_recording.mp3"></audio></td></tr> | |
| <tr><td><b>enunciated</b></td><td>One PM to nine PM, it'll be cloudy, and in the low fifties.</td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/enunciated_piper.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/enunciated_xvits.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/v1/enunciated_xvits.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/enunciated_recording.mp3"></audio></td></tr> | |
| <tr><td><b>laughing</b></td><td>Their moment had come, or so they said.</td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/laughing_piper.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/laughing_xvits.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/v1/laughing_xvits.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/laughing_recording.mp3"></audio></td></tr> | |
| </table> | |
| ### Emphasis, by moving the `*mark*` (style `default`) | |
| Piper has no equivalent: it stresses whatever its training suggests. | |
| <table> | |
| <tr><th>text</th><th>x-vits 0.2</th><th>x-vits 0.1</th></tr> | |
| <tr><td>I never said she stole my money.</td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/emphasis_0.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/v1/emphasis_0.mp3"></audio></td></tr> | |
| <tr><td><b>*I*</b> never said she stole my money.</td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/emphasis_1.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/v1/emphasis_1.mp3"></audio></td></tr> | |
| <tr><td>I never said <b>*she*</b> stole my money.</td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/emphasis_2.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/v1/emphasis_2.mp3"></audio></td></tr> | |
| <tr><td>I never said she stole my <b>*money*</b>.</td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/emphasis_3.mp3"></audio></td><td><audio controls preload="none" src="https://huggingface.co/fdemelo/x-vits/resolve/main/samples/v1/emphasis_3.mp3"></audio></td></tr> | |
| </table> | |
| ## 🚀 Usage | |
| ```bash | |
| sudo apt install libespeak-ng1 # phonemizer backend (macOS: brew install espeak-ng) | |
| pip install "x-vits @ git+https://github.com/femelo/x-vits" | |
| ``` | |
| ```python | |
| import soundfile as sf | |
| from xvits.runtime import XVits | |
| voice = XVits.from_pretrained("fdemelo/x-vits") | |
| audio = voice.synthesize("We didn't get the weather right, guys.", style="happy") | |
| sf.write("happy.wav", audio, voice.sample_rate) | |
| # explicit emphasis, a different style, a fixed seed and a faster CPU setting | |
| audio = voice.synthesize("I never said *she* stole my money.", style="confused", seed=7, steps=4) | |
| # a stronger style and a bigger pitch peak on the stressed word | |
| audio = voice.synthesize("I *really* didn't expect that.", style="happy", style_strength=1.5, emphasis_scale=2.0) | |
| sf.write("confused.wav", audio, voice.sample_rate) | |
| ``` | |
| Or from the command line: | |
| ```bash | |
| xvits-speak --model fdemelo/x-vits --text "Is it *really* you?" --style sad --out sad.wav | |
| ``` | |
| | option | values | | | |
| |---|---|---| | |
| | `style` | `default` `happy` `sad` `confused` `enunciated` `laughing` | none gives a neutral mix | | |
| | `emphasis` | `auto` · `marked` · `none` | `auto`: marks if present, otherwise predicted | | |
| | `emphasis_scale` | default 1.5 | multiplies the emphasis pitch peak, lengthening and loudness; 1.0 = average emphasis in the recordings | | |
| | `style_strength` | default 1.0 | style exaggeration: 0 none, 2 double (happy: +2.2, +3.5, +4.8 semitones above the speaker's average at 0, 1, 2) | | |
| | `guidance` | `"timing,pitch,energy,effort"` | per-channel exaggeration; default `"1.0,2.0,4.0,1.0"` | | |
| | `temperature` | `"timing,pitch,energy"` | variety; default `"0.8,0.7,1.0"`, higher is livelier | | |
| | `length_scale` | default 1.08 | larger is slower | | |
| | `steps` | default 16 | 4 is about 2.5× faster on CPU and changes pitch by about 0.1 semitone | | |
| | `seed` | int | reproducible output | | |
| Speed with onnxruntime on 4 CPU threads: real-time factor about 0.6 (16 steps) or 0.25 (4 steps). | |
| ## 📦 Files | |
| | file | role | size | | |
| |---|---|---| | |
| | `encoder.onnx` | phonemes → text states | 26 MB | | |
| | `predictor.onnx` | one step of the flow-matching prosody predictor (style-conditioned) | 31 MB | | |
| | `durations.onnx` | style-aware stochastic duration predictor | 2.5 MB | | |
| | `acoustic.onnx` | prosody-conditioned prior, length regulation, flow, HiFi-GAN | 36 MB | | |
| | `prominence.onnx` | predicts which words to stress | 4.8 MB | | |
| | `dp_cond.npz`, `emphasis_template.json` | style vectors for the duration predictor; the emphasis shape | small | | |
| | `runtime.json`, `config.json` | runtime settings; Piper phoneme map (eSpeak `en`) | | | |
| These graphs are driven by `xvits.runtime`, so they aren't drop-in files for Piper's own runtime. | |
| The export was verified against PyTorch end to end (maximum difference about 3e-5 at several sentence lengths). | |
| ## 🧪 Training | |
| - **Base:** Piper's LJSpeech `medium` checkpoint (LJSpeech is public domain). | |
| - **Data:** [Expresso](https://speechbot.github.io/expresso/) speaker **ex02**, read speech in six styles, | |
| 2.2 h after leaving out whisper, singing, long-form and `<laugh>` rows. 245 utterances held out. | |
| - **Recipe** ([x-vits](https://github.com/femelo/x-vits) 0.2, `xvits-train`): | |
| 1. fine-tune the base checkpoint on ex02; | |
| 2. extract per-phoneme duration, pitch, energy and vocal-effort targets; | |
| 3. fine-tune with a zero-initialised prosody input to the prior, including frame-level interpolation; | |
| 4. train an OT-CFM prosody predictor with a style input; | |
| 5. train a style-conditioned duration predictor; | |
| 6. train a word-prominence student on acoustic labels; | |
| 7. fit an emphasis template (pitch, duration, loudness) anchored on the stressed vowel, from Expresso's `*emphasis*` marks. | |
| **Held-out results** (129 sentences, two seeds; pitch and timing match are correlations with the recording of the same sentence): | |
| | | pitch match | timing match | WER (Whisper) | UTMOS | | |
| |---|---|---|---|---| | |
| | Piper, fine-tuned | 0.37 | 0.30 | 1.7 % | 4.08 | | |
| | x-vits 0.1 | 0.37 | 0.39 | 2.1 % | 4.01 | | |
| | **x-vits 0.2** | **0.40** | 0.37 | 3.6 % | 3.86 | | |
| | the recordings | | | | 4.12 | | |
| UTMOS and Whisper both prefer calm read speech: UTMOS gives the speaker's own laughing takes 2.49, and | |
| 0.2 loses most of its score in happy and laughing, the styles that became more expressive. Listen | |
| above and decide for yourself. | |
| The prominence student finds the recording's main stress 42 % of the time, against 34 % for "last word" and 15 % for a random word. | |
| ## ⚠️ License and limitations | |
| - **CC BY-NC 4.0.** Expresso is licensed for non-commercial use only, and so is this voice. The x-vits code is MIT. | |
| - English, one female speaker, read speech. Styles are as recorded in Expresso; there are no other emotions. | |
| - Laughing is rendered through intonation, timing and brightness; the voice doesn't produce actual laughter. | |
| - Questions sometimes end with a smaller rise than a natural reading. | |
| - Don't use this voice to impersonate real people or to mislead listeners. | |
| ## Citation | |
| If you use this voice, please cite Expresso: | |
| ```bibtex | |
| @inproceedings{nguyen2023expresso, | |
| title = {EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesis}, | |
| author = {Nguyen, Tu Anh and Hsu, Wei-Ning and D'Avirro, Antony and Shi, Bowen and Gat, Itai and | |
| Fazel-Zarani, Maryam and Remez, Tal and Copet, Jade and Synnaeve, Gabriel and Hassid, Michael and | |
| Kreuk, Felix and Adi, Yossi and Dupoux, Emmanuel}, | |
| booktitle = {Interspeech}, | |
| year = {2023} | |
| } | |
| ``` | |