docs: add readme
Browse files
README.md
CHANGED
|
@@ -1,9 +1,92 @@
|
|
| 1 |
---
|
| 2 |
-
|
| 3 |
-
|
| 4 |
-
|
|
|
|
|
|
|
|
|
|
| 5 |
---
|
| 6 |
|
| 7 |
-
|
| 8 |
|
| 9 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
+
datasets:
|
| 3 |
+
- VoxCeleb2
|
| 4 |
+
base_model:
|
| 5 |
+
- Edresson/Speaker_Encoder_H_ASP
|
| 6 |
+
license: mit
|
| 7 |
+
library_name: coqui
|
| 8 |
---
|
| 9 |
|
| 10 |
+
# H/ASP speaker encoder
|
| 11 |
|
| 12 |
+
A speaker encoder that maps speech to 512-dimensional speaker embeddings
|
| 13 |
+
(d-vectors), for use with [coqui-tts](https://github.com/idiap/coqui-ai-TTS).
|
| 14 |
+
It's the encoder used in the [YourTTS paper](https://arxiv.org/abs/2112.02418)
|
| 15 |
+
and by the YourTTS models in coqui-tts
|
| 16 |
+
(`tts_models/multilingual/multi-dataset/your_tts` and
|
| 17 |
+
`tts_models/multilingual/cml-tts/your_tts`) for zero-shot voice cloning. It's
|
| 18 |
+
also commonly used to measure speaker similarity (SECS) between synthesized and
|
| 19 |
+
real speech.
|
| 20 |
+
|
| 21 |
+
This is a copy of
|
| 22 |
+
[Edresson/Speaker_Encoder_H_ASP](https://huggingface.co/Edresson/Speaker_Encoder_H_ASP)
|
| 23 |
+
with the model configuration (`config.json`) added, so that it can be loaded
|
| 24 |
+
directly with coqui-tts. The checkpoint is unchanged.
|
| 25 |
+
|
| 26 |
+
## Usage
|
| 27 |
+
|
| 28 |
+
Install [coqui-tts](https://github.com/idiap/coqui-ai-TTS):
|
| 29 |
+
|
| 30 |
+
```bash
|
| 31 |
+
pip install coqui-tts
|
| 32 |
+
```
|
| 33 |
+
|
| 34 |
+
coqui-tts accepts the repository ID `tts-hub/speaker_encoder_hasp` wherever a
|
| 35 |
+
speaker encoder is expected. The model and its config are downloaded with
|
| 36 |
+
`huggingface_hub` and stored in the Hugging Face cache, so no config path is
|
| 37 |
+
needed.
|
| 38 |
+
|
| 39 |
+
Compute an embedding from an audio file:
|
| 40 |
+
|
| 41 |
+
```python
|
| 42 |
+
from TTS.tts.utils.speakers import SpeakerManager
|
| 43 |
+
|
| 44 |
+
manager = SpeakerManager(encoder_model_path="tts-hub/speaker_encoder_hasp")
|
| 45 |
+
embedding = manager.compute_embedding_from_clip("speaker.wav") # list of 512 floats
|
| 46 |
+
```
|
| 47 |
+
|
| 48 |
+
Compute embeddings for all speakers of a dataset, e.g. to train a model with
|
| 49 |
+
d-vectors. This encoder is the default, so `--model_path` can also be left out:
|
| 50 |
+
|
| 51 |
+
```bash
|
| 52 |
+
python TTS/bin/compute_embeddings.py --model_path tts-hub/speaker_encoder_hasp \
|
| 53 |
+
--config_dataset_path dataset_config.json --output_path speakers.pth
|
| 54 |
+
```
|
| 55 |
+
|
| 56 |
+
In a VITS/YourTTS config, e.g. for zero-shot voice cloning or the speaker
|
| 57 |
+
consistency loss, set `model_args.speaker_encoder_model_path` to
|
| 58 |
+
`tts-hub/speaker_encoder_hasp` and leave `speaker_encoder_config_path` empty.
|
| 59 |
+
|
| 60 |
+
## Model details
|
| 61 |
+
|
| 62 |
+
- **Architecture:** ResNet with attentive statistics pooling (H/ASP, from
|
| 63 |
+
[Heo et al., 2020](https://arxiv.org/abs/2009.14153)), 512-dimensional output.
|
| 64 |
+
- **Input:** 16 kHz audio, converted to 64-band log mel spectrograms inside the
|
| 65 |
+
model. Audio at other sampling rates is resampled when loaded.
|
| 66 |
+
- **Loss:** softmax plus angular prototypical loss.
|
| 67 |
+
- **Training data:** VoxCeleb 2 (dev set, 5,994 speakers).
|
| 68 |
+
|
| 69 |
+
## License
|
| 70 |
+
|
| 71 |
+
MIT, following the original repository and the
|
| 72 |
+
[voxceleb_trainer](https://github.com/clovaai/voxceleb_trainer) code it is
|
| 73 |
+
based on. See `LICENSE`.
|
| 74 |
+
|
| 75 |
+
## Citation
|
| 76 |
+
|
| 77 |
+
```bibtex
|
| 78 |
+
@inproceedings{casanova2022yourtts,
|
| 79 |
+
title = {{YourTTS}: Towards Zero-Shot Multi-Speaker {TTS} and Zero-Shot Voice Conversion for Everyone},
|
| 80 |
+
author = {Casanova, Edresson and Weber, Julian and Shulby, Christopher D. and Candido Junior, Arnaldo and G{\"o}lge, Eren and Ponti, Moacir A.},
|
| 81 |
+
booktitle = {Proceedings of the 39th International Conference on Machine Learning},
|
| 82 |
+
pages = {2709--2720},
|
| 83 |
+
year = {2022},
|
| 84 |
+
}
|
| 85 |
+
|
| 86 |
+
@article{heo2020clova,
|
| 87 |
+
title = {Clova Baseline System for the {VoxCeleb} Speaker Recognition Challenge 2020},
|
| 88 |
+
author = {Heo, Hee Soo and Lee, Bong-Jin and Huh, Jaesung and Chung, Joon Son},
|
| 89 |
+
journal = {arXiv preprint arXiv:2009.14153},
|
| 90 |
+
year = {2020},
|
| 91 |
+
}
|
| 92 |
+
```
|