--- tags: - speech-to-speech-translation - expressive-speech-translation - mimi language: - en - de - es --- # DirectS2ST Checkpoints of DirectS2ST, the end-to-end expressive speech-to-speech translation model from *Holistic Parallel Supervision for Expressive Speech-to-Speech Translation*, trained on VC-Dub (professional dubbing + multilingual voice conversion). | Folder | Direction | | --- | --- | | `en-de/` | English → German | | `en-es/` | English → Spanish | Each folder holds `pytorch_model.bin`, `model_config.json` and the SentencePiece-8k text tokenizer (`text_tokenizer/`). Given English source speech, the model predicts all 16 Mimi codec streams of the target speech directly: a frozen w2v-BERT 2.0 encoder, an AR temporal decoder (source text, target text and codec stream c0 with one shared head), and an NAR depth decoder (c1-c15) conditioned on a CAMPPlus speaker prefix and a source codec prompt. ## Usage Code, setup and training: ``` hf download TedZhangHao/DirectS2ST --include "en-es/*" --local-dir checkpoints python inference/infer.py \ --checkpoint-dir checkpoints/en-es \ --campplus-model-root /path/to/seed-vc \ --campplus-checkpoint-path /path/to/campplus_cn_en_common.pt \ --source-wav input_en.wav --output-wav output_es.wav ``` Demo: ## Third-party components The checkpoints contain the frozen weights of [w2v-BERT 2.0](https://huggingface.co/facebook/w2v-bert-2.0) and [Mimi](https://huggingface.co/kyutai/mimi); their licenses apply to those parts. The CAMPPlus speaker encoder ([iic/speech_campplus_sv_zh_en_16k-common_advanced](https://modelscope.cn/models/iic/speech_campplus_sv_zh_en_16k-common_advanced)) is not included and is loaded at run time.