|
Download README.md from TedZhangHao/DirectS2ST: direct link, hf CLI and curl.
- Browser
- Download file 1.85 kB
-
https://huggingface.co/TedZhangHao/DirectS2ST/resolve/main/README.md
- Command line
-
hf download hf://TedZhangHao/DirectS2ST/README.md
-
curl -L -o README.md https://huggingface.co/TedZhangHao/DirectS2ST/resolve/main/README.md
1.85 kB
| tags: | |
| - speech-to-speech-translation | |
| - expressive-speech-translation | |
| - mimi | |
| language: | |
| - en | |
| - de | |
| - es | |
| # DirectS2ST | |
| Checkpoints of DirectS2ST, the end-to-end expressive speech-to-speech translation | |
| model from *Holistic Parallel Supervision for Expressive Speech-to-Speech | |
| Translation*, trained on VC-Dub (professional dubbing + multilingual voice | |
| conversion). | |
| | Folder | Direction | | |
| | --- | --- | | |
| | `en-de/` | English → German | | |
| | `en-es/` | English → Spanish | | |
| Each folder holds `pytorch_model.bin`, `model_config.json` and the | |
| SentencePiece-8k text tokenizer (`text_tokenizer/`). | |
| Given English source speech, the model predicts all 16 Mimi codec streams of | |
| the target speech directly: a frozen w2v-BERT 2.0 encoder, an AR temporal | |
| decoder (source text, target text and codec stream c0 with one shared head), and | |
| an NAR depth decoder (c1-c15) conditioned on a CAMPPlus speaker prefix and a | |
| source codec prompt. | |
| ## Usage | |
| Code, setup and training: <https://github.com/hzted/truly-e2e-expressive-s2st-demo/tree/main/DirectS2ST> | |
| ``` | |
| hf download TedZhangHao/DirectS2ST --include "en-es/*" --local-dir checkpoints | |
| python inference/infer.py \ | |
| --checkpoint-dir checkpoints/en-es \ | |
| --campplus-model-root /path/to/seed-vc \ | |
| --campplus-checkpoint-path /path/to/campplus_cn_en_common.pt \ | |
| --source-wav input_en.wav --output-wav output_es.wav | |
| ``` | |
| Demo: <https://hzted.github.io/truly-e2e-expressive-s2st-demo/> | |
| ## Third-party components | |
| The checkpoints contain the frozen weights of | |
| [w2v-BERT 2.0](https://huggingface.co/facebook/w2v-bert-2.0) and | |
| [Mimi](https://huggingface.co/kyutai/mimi); their licenses apply to those | |
| parts. The CAMPPlus speaker encoder | |
| ([iic/speech_campplus_sv_zh_en_16k-common_advanced](https://modelscope.cn/models/iic/speech_campplus_sv_zh_en_16k-common_advanced)) | |
| is not included and is loaded at run time. | |