H/ASP speaker encoder
A speaker encoder that maps speech to 512-dimensional speaker embeddings
(d-vectors), for use with coqui-tts.
It's the encoder used in the YourTTS paper
and by the YourTTS models in coqui-tts
(tts_models/multilingual/multi-dataset/your_tts and
tts_models/multilingual/cml-tts/your_tts) for zero-shot voice cloning. It's
also commonly used to measure speaker similarity (SECS) between synthesized and
real speech.
This is a copy of
Edresson/Speaker_Encoder_H_ASP
with the model configuration (config.json) added, so that it can be loaded
directly with coqui-tts. The checkpoint is unchanged.
Usage
Install coqui-tts:
pip install coqui-tts
coqui-tts accepts the repository ID tts-hub/speaker_encoder_hasp wherever a
speaker encoder is expected. The model and its config are downloaded with
huggingface_hub and stored in the Hugging Face cache, so no config path is
needed.
Compute an embedding from an audio file:
from TTS.tts.utils.speakers import SpeakerManager
manager = SpeakerManager(encoder_model_path="tts-hub/speaker_encoder_hasp")
embedding = manager.compute_embedding_from_clip("speaker.wav") # list of 512 floats
Compute embeddings for all speakers of a dataset, e.g. to train a model with
d-vectors. This encoder is the default, so --model_path can also be left out:
python TTS/bin/compute_embeddings.py --model_path tts-hub/speaker_encoder_hasp \
--config_dataset_path dataset_config.json --output_path speakers.pth
In a VITS/YourTTS config, e.g. for zero-shot voice cloning or the speaker
consistency loss, set model_args.speaker_encoder_model_path to
tts-hub/speaker_encoder_hasp and leave speaker_encoder_config_path empty.
Model details
- Architecture: ResNet with attentive statistics pooling (H/ASP, from Heo et al., 2020), 512-dimensional output.
- Input: 16 kHz audio, converted to 64-band log mel spectrograms inside the model. Audio at other sampling rates is resampled when loaded.
- Loss: softmax plus angular prototypical loss.
- Training data: VoxCeleb 2 (dev set, 5,994 speakers).
License
MIT, following the original repository and the
voxceleb_trainer code it is
based on. See LICENSE.
Citation
@inproceedings{casanova2022yourtts,
title = {{YourTTS}: Towards Zero-Shot Multi-Speaker {TTS} and Zero-Shot Voice Conversion for Everyone},
author = {Casanova, Edresson and Weber, Julian and Shulby, Christopher D. and Candido Junior, Arnaldo and G{\"o}lge, Eren and Ponti, Moacir A.},
booktitle = {Proceedings of the 39th International Conference on Machine Learning},
pages = {2709--2720},
year = {2022},
}
@article{heo2020clova,
title = {Clova Baseline System for the {VoxCeleb} Speaker Recognition Challenge 2020},
author = {Heo, Hee Soo and Lee, Bong-Jin and Huh, Jaesung and Chung, Joon Son},
journal = {arXiv preprint arXiv:2009.14153},
year = {2020},
}
- Downloads last month
- 39
Model tree for tts-hub/speaker_encoder_hasp
Base model
Edresson/Speaker_Encoder_H_ASP