PyTorch
coqui

H/ASP speaker encoder

A speaker encoder that maps speech to 512-dimensional speaker embeddings (d-vectors), for use with coqui-tts. It's the encoder used in the YourTTS paper and by the YourTTS models in coqui-tts (tts_models/multilingual/multi-dataset/your_tts and tts_models/multilingual/cml-tts/your_tts) for zero-shot voice cloning. It's also commonly used to measure speaker similarity (SECS) between synthesized and real speech.

This is a copy of Edresson/Speaker_Encoder_H_ASP with the model configuration (config.json) added, so that it can be loaded directly with coqui-tts. The checkpoint is unchanged.

Usage

Install coqui-tts:

pip install coqui-tts

coqui-tts accepts the repository ID tts-hub/speaker_encoder_hasp wherever a speaker encoder is expected. The model and its config are downloaded with huggingface_hub and stored in the Hugging Face cache, so no config path is needed.

Compute an embedding from an audio file:

from TTS.tts.utils.speakers import SpeakerManager

manager = SpeakerManager(encoder_model_path="tts-hub/speaker_encoder_hasp")
embedding = manager.compute_embedding_from_clip("speaker.wav")  # list of 512 floats

Compute embeddings for all speakers of a dataset, e.g. to train a model with d-vectors. This encoder is the default, so --model_path can also be left out:

python TTS/bin/compute_embeddings.py --model_path tts-hub/speaker_encoder_hasp \
    --config_dataset_path dataset_config.json --output_path speakers.pth

In a VITS/YourTTS config, e.g. for zero-shot voice cloning or the speaker consistency loss, set model_args.speaker_encoder_model_path to tts-hub/speaker_encoder_hasp and leave speaker_encoder_config_path empty.

Model details

  • Architecture: ResNet with attentive statistics pooling (H/ASP, from Heo et al., 2020), 512-dimensional output.
  • Input: 16 kHz audio, converted to 64-band log mel spectrograms inside the model. Audio at other sampling rates is resampled when loaded.
  • Loss: softmax plus angular prototypical loss.
  • Training data: VoxCeleb 2 (dev set, 5,994 speakers).

License

MIT, following the original repository and the voxceleb_trainer code it is based on. See LICENSE.

Citation

@inproceedings{casanova2022yourtts,
  title = {{YourTTS}: Towards Zero-Shot Multi-Speaker {TTS} and Zero-Shot Voice Conversion for Everyone},
  author = {Casanova, Edresson and Weber, Julian and Shulby, Christopher D. and Candido Junior, Arnaldo and G{\"o}lge, Eren and Ponti, Moacir A.},
  booktitle = {Proceedings of the 39th International Conference on Machine Learning},
  pages = {2709--2720},
  year = {2022},
}

@article{heo2020clova,
  title = {Clova Baseline System for the {VoxCeleb} Speaker Recognition Challenge 2020},
  author = {Heo, Hee Soo and Lee, Bong-Jin and Huh, Jaesung and Chung, Joon Son},
  journal = {arXiv preprint arXiv:2009.14153},
  year = {2020},
}
Downloads last month
39
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for tts-hub/speaker_encoder_hasp

Finetuned
(1)
this model

Papers for tts-hub/speaker_encoder_hasp