PyTorch
coqui
eginhard commited on
Commit
e53312d
·
1 Parent(s): 4fa767c

docs: add readme

Browse files
Files changed (1) hide show
  1. README.md +88 -5
README.md CHANGED
@@ -1,9 +1,92 @@
1
  ---
2
- license: other
3
- license_name: mit
4
- license_link: LICENSE
 
 
 
5
  ---
6
 
7
- This is the checkpoint of the H/ASP speaker encoder that is compatible with Coqui TTS. This speaker encoder was used on YourTTS and XTTS papers.
8
 
9
- Please refer to the original repository: https://github.com/clovaai/voxceleb_trainer/
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ datasets:
3
+ - VoxCeleb2
4
+ base_model:
5
+ - Edresson/Speaker_Encoder_H_ASP
6
+ license: mit
7
+ library_name: coqui
8
  ---
9
 
10
+ # H/ASP speaker encoder
11
 
12
+ A speaker encoder that maps speech to 512-dimensional speaker embeddings
13
+ (d-vectors), for use with [coqui-tts](https://github.com/idiap/coqui-ai-TTS).
14
+ It's the encoder used in the [YourTTS paper](https://arxiv.org/abs/2112.02418)
15
+ and by the YourTTS models in coqui-tts
16
+ (`tts_models/multilingual/multi-dataset/your_tts` and
17
+ `tts_models/multilingual/cml-tts/your_tts`) for zero-shot voice cloning. It's
18
+ also commonly used to measure speaker similarity (SECS) between synthesized and
19
+ real speech.
20
+
21
+ This is a copy of
22
+ [Edresson/Speaker_Encoder_H_ASP](https://huggingface.co/Edresson/Speaker_Encoder_H_ASP)
23
+ with the model configuration (`config.json`) added, so that it can be loaded
24
+ directly with coqui-tts. The checkpoint is unchanged.
25
+
26
+ ## Usage
27
+
28
+ Install [coqui-tts](https://github.com/idiap/coqui-ai-TTS):
29
+
30
+ ```bash
31
+ pip install coqui-tts
32
+ ```
33
+
34
+ coqui-tts accepts the repository ID `tts-hub/speaker_encoder_hasp` wherever a
35
+ speaker encoder is expected. The model and its config are downloaded with
36
+ `huggingface_hub` and stored in the Hugging Face cache, so no config path is
37
+ needed.
38
+
39
+ Compute an embedding from an audio file:
40
+
41
+ ```python
42
+ from TTS.tts.utils.speakers import SpeakerManager
43
+
44
+ manager = SpeakerManager(encoder_model_path="tts-hub/speaker_encoder_hasp")
45
+ embedding = manager.compute_embedding_from_clip("speaker.wav") # list of 512 floats
46
+ ```
47
+
48
+ Compute embeddings for all speakers of a dataset, e.g. to train a model with
49
+ d-vectors. This encoder is the default, so `--model_path` can also be left out:
50
+
51
+ ```bash
52
+ python TTS/bin/compute_embeddings.py --model_path tts-hub/speaker_encoder_hasp \
53
+ --config_dataset_path dataset_config.json --output_path speakers.pth
54
+ ```
55
+
56
+ In a VITS/YourTTS config, e.g. for zero-shot voice cloning or the speaker
57
+ consistency loss, set `model_args.speaker_encoder_model_path` to
58
+ `tts-hub/speaker_encoder_hasp` and leave `speaker_encoder_config_path` empty.
59
+
60
+ ## Model details
61
+
62
+ - **Architecture:** ResNet with attentive statistics pooling (H/ASP, from
63
+ [Heo et al., 2020](https://arxiv.org/abs/2009.14153)), 512-dimensional output.
64
+ - **Input:** 16 kHz audio, converted to 64-band log mel spectrograms inside the
65
+ model. Audio at other sampling rates is resampled when loaded.
66
+ - **Loss:** softmax plus angular prototypical loss.
67
+ - **Training data:** VoxCeleb 2 (dev set, 5,994 speakers).
68
+
69
+ ## License
70
+
71
+ MIT, following the original repository and the
72
+ [voxceleb_trainer](https://github.com/clovaai/voxceleb_trainer) code it is
73
+ based on. See `LICENSE`.
74
+
75
+ ## Citation
76
+
77
+ ```bibtex
78
+ @inproceedings{casanova2022yourtts,
79
+ title = {{YourTTS}: Towards Zero-Shot Multi-Speaker {TTS} and Zero-Shot Voice Conversion for Everyone},
80
+ author = {Casanova, Edresson and Weber, Julian and Shulby, Christopher D. and Candido Junior, Arnaldo and G{\"o}lge, Eren and Ponti, Moacir A.},
81
+ booktitle = {Proceedings of the 39th International Conference on Machine Learning},
82
+ pages = {2709--2720},
83
+ year = {2022},
84
+ }
85
+
86
+ @article{heo2020clova,
87
+ title = {Clova Baseline System for the {VoxCeleb} Speaker Recognition Challenge 2020},
88
+ author = {Heo, Hee Soo and Lee, Bong-Jin and Huh, Jaesung and Chung, Joon Son},
89
+ journal = {arXiv preprint arXiv:2009.14153},
90
+ year = {2020},
91
+ }
92
+ ```