Download README.md from alphappp/PromptTTSD: direct link, hf CLI and curl.
- Browser
- Download file 8.13 kB
-
https://huggingface.co/alphappp/PromptTTSD/resolve/main/README.md
- Command line
-
hf download hf://alphappp/PromptTTSD/README.md
-
curl -L -o README.md https://huggingface.co/alphappp/PromptTTSD/resolve/main/README.md
pipeline_tag: text-to-speech
base_model: Soul-AILab/SoulX-Podcast-1.7B
base_model_relation: finetune
tags:
- multi-speaker
- speech-synthesis
- style-control
PromptTTSD
We post-trained SoulX-Podcast on the PromptDialog dataset to build PromptTTSD. With PromptTTSD, you can use open-ended natural-language descriptions to control how each speaker delivers a line in a multi-speaker dialogue.
PromptTTSD generates single-speaker speech and multi-speaker dialogue locally. It can return a complete waveform or stream audio one chunk at a time.
The model uses Hugging Face Transformers, PyTorch, a flow model, and HiFT. This project builds on the SoulX-Podcast model and inference code and includes components from CosyVoice. See the source files for copyright notices. Model weights are subject to their respective licenses.
Install
Clone and install
You need Git, uv, Python 3.11 or later, and a CUDA-capable GPU with an NVIDIA driver compatible with PyTorch. Run the remaining commands from the cloned repository.
git clone https://github.com/bigai-ai/PromptTTSD.git
cd PromptTTSD
uv sync --locked
You can also install the package with pip install -e . in an activated Python environment. With that setup, run the commands below without the uv run prefix.
Model download
Download the post-trained checkpoint from Hugging Face into the directory used by the examples. If the repository requires authentication, sign in with uv run hf auth login using an account with access.
uv run hf download alphappp/PromptTTSD --local-dir pretrained_models/PromptTTSD
To use a different directory, pass it to --model-path or PromptTTSD.from_pretrained(). Keep the original filenames, including soulxpodcast_config.json, alongside the Transformers shards, configuration, tokenizer files, flow.pt, hift.pt, and campplus.onnx. On the first run, s3tokenizer downloads speech_tokenizer_v2_25hz and caches it locally.
Basic Usage
Choose a GPU with CUDA_VISIBLE_DEVICES. By default, the model uses the first visible GPU. The dialogue example uses two reference speakers and a separate style description for each turn.
# Generate a complete single-speaker recording
uv run python inference.py --model-path pretrained_models/PromptTTSD \
--input-jsonl examples/single.jsonl --output-dir outputs/single
# Stream multi-speaker dialogue to a WAV file
uv run python inference.py --model-path pretrained_models/PromptTTSD \
--input-jsonl examples/dialogue.jsonl --output-dir outputs/dialogue --stream
Both input formats support --stream. By default, the CLI generates the complete waveform. It processes JSONL records sequentially in one process and writes one 24 kHz mono PCM16 WAV file per record, with 200 ms of silence between turns. In streaming mode, it flushes the WAV file after each chunk.
The examples above write outputs/single/samples/single.wav and outputs/dialogue/samples/dialogue.wav.
The CLI writes generated audio to samples/<utt>.wav and copies reference recordings to refer/ (for example, refer/<utt>__prompt0.wav). As each sample finishes, it writes a record to <input_stem>_inference.jsonl. Each record preserves the input fields, updates prompt_audio to point to the copied references, and sets output_audio to the generated WAV. Both paths are relative to the output directory. The prompt_audio field keeps its original string or list format.
Use one JSON object per input line.
uttsets the output name. If omitted, the CLI generates a name from the line number. Names must be unique within the input file.prompt_audioandprompt_textsupply the reference audio paths and their transcripts. Use a string for each field or two lists of equal length, with up to eight speakers. Relative audio paths are resolved from the directory containing the JSONL file.target_textaccepts a string or a list of strings, one per turn. Usespeakerto identify the speaker for each turn with a zero-based index. You can omit it for single-speaker input.dialogue_textcan replace bothtarget_textandspeaker. For example,"[S1]你好。[S2]你好呀。"contains two turns. Speaker tags in this format start at 1.contextsupplies optional instructions for individual turns. Provide one entry per target turn. Each instruction is wrapped in<|start_prompt|>…<|end_prompt|>before it enters the model. These instructions do not change how much dialogue history the model retains.
Match the reference transcript to the recording. Write numbers and similar expressions as they should be spoken.
Python API
from contextlib import closing
from prompt_ttsd import PromptTTSD
model = PromptTTSD.from_pretrained("pretrained_models/PromptTTSD")
inputs = dict(
prompt_audio="ref_audio/tongtong.wav",
prompt_text="在一座美丽的森林里,住着许多动物。",
target_text="你好,欢迎体验语音生成。",
)
result = model.infer(**inputs, stream=False)
# result["audio"] is a 1-D float32 tensor on the CPU. result["sample_rate"] is 24000.
with closing(model.infer(**inputs, stream=True)) as chunks:
for chunk in chunks:
audio = chunk["audio"]
# Play or write the audio here.
# If you exit early, closing() stops background token generation.
The Python API resolves relative paths from the current working directory. Multi-speaker input uses the same format as the JSONL examples. Make calls sequentially on a given model instance. Each call starts a fresh dialogue context.
Each streaming chunk contains audio, sample_rate, turn_index, and finalize. Turn indices start at 0. A chunk with finalize=True marks the end of a turn and may contain empty audio. Silence between turns belongs to the following turn and has finalize=False.
As each new block of mel features arrives, streaming HiFi-GAN resynthesizes the waveform from all features accumulated for the current turn. It emits only the audio it has yet to send, holding back the last 160 ms until more features arrive or the turn ends. The streaming path uses HiFT's default excitation without short-window source caching or crossfading. Model initialization disables cuDNN TF32 to prevent numerical differences in fundamental frequency (F0) across prefix lengths from accumulating as phase drift.
Dialogue history and context limits
Streaming and non-streaming inference use the same fixed history policy. Before adding each turn's target text, the model checks the existing context. It trims the history and rebuilds the KV cache when either of the following thresholds is exceeded.
- More than 10 turns, including reference turns.
- More than 6,192 tokens, counting both text and speech tokens.
When trimming, the model keeps up to two initial reference turns, both text and speech for the two most recent turns, and text only for the two turns immediately before those. It drops older dialogue history. If there is only one reference turn, it keeps that turn. Below or at both thresholds, it retains the full history.
The API has no option to change these thresholds or force it to retain all history. The 6,192-token threshold governs history trimming between turns. The model's context capacity still limits the length of an individual turn.
Acknowledgments
- We borrowed a lot of code from CosyVoice.
- We borrowed a lot of code from SoulX-Podcast.
Disclaimer
The content provided above is for academic purposes only and is intended to demonstrate technical capabilities. Some examples are sourced from the internet. If any content infringes on your rights, please contact us to request its removal.