PromptTTSD / README.md
alphappp's picture
Add task and base model metadata to the PromptTTSD model card
64ba7c2 verified
|
Raw History Blame Contribute Delete
8.13 kB
---
pipeline_tag: text-to-speech
base_model: Soul-AILab/SoulX-Podcast-1.7B
base_model_relation: finetune
tags:
- multi-speaker
- speech-synthesis
- style-control
---
# PromptTTSD
[![Demo: PromptDialog](https://img.shields.io/badge/Demo-PromptDialog-FFD21E)](https://huggingface.co/spaces/alphappp/PromptDialog-demo)
[![Code: PromptTTSD](https://img.shields.io/badge/Code-PromptTTSD-181717)](https://github.com/bigai-ai/PromptTTSD)
[![Dataset: PromptDialog-v2](https://img.shields.io/badge/Dataset-PromptDialog--v2-367C49)](https://huggingface.co/datasets/alphappp/PromptDialog-v2)
We post-trained SoulX-Podcast on the PromptDialog dataset to build PromptTTSD. With PromptTTSD, you can use open-ended natural-language descriptions to control how each speaker delivers a line in a multi-speaker dialogue.
PromptTTSD generates single-speaker speech and multi-speaker dialogue locally. It can return a complete waveform or stream audio one chunk at a time.
The model uses Hugging Face Transformers, PyTorch, a flow model, and HiFT. This project builds on the SoulX-Podcast model and inference code and includes components from CosyVoice. See the source files for copyright notices. Model weights are subject to their respective licenses.
## Install
### Clone and install
You need Git, uv, Python 3.11 or later, and a CUDA-capable GPU with an NVIDIA driver compatible with PyTorch. Run the remaining commands from the cloned repository.
```bash
git clone https://github.com/bigai-ai/PromptTTSD.git
cd PromptTTSD
uv sync --locked
```
You can also install the package with `pip install -e .` in an activated Python environment. With that setup, run the commands below without the `uv run` prefix.
### Model download
Download the post-trained checkpoint from Hugging Face into the directory used by the examples. If the repository requires authentication, sign in with `uv run hf auth login` using an account with access.
```bash
uv run hf download alphappp/PromptTTSD --local-dir pretrained_models/PromptTTSD
```
To use a different directory, pass it to `--model-path` or `PromptTTSD.from_pretrained()`. Keep the original filenames, including `soulxpodcast_config.json`, alongside the Transformers shards, configuration, tokenizer files, `flow.pt`, `hift.pt`, and `campplus.onnx`. On the first run, s3tokenizer downloads `speech_tokenizer_v2_25hz` and caches it locally.
## Basic Usage
Choose a GPU with `CUDA_VISIBLE_DEVICES`. By default, the model uses the first visible GPU. The dialogue example uses two reference speakers and a separate style description for each turn.
```bash
# Generate a complete single-speaker recording
uv run python inference.py --model-path pretrained_models/PromptTTSD \
--input-jsonl examples/single.jsonl --output-dir outputs/single
# Stream multi-speaker dialogue to a WAV file
uv run python inference.py --model-path pretrained_models/PromptTTSD \
--input-jsonl examples/dialogue.jsonl --output-dir outputs/dialogue --stream
```
Both input formats support `--stream`. By default, the CLI generates the complete waveform. It processes JSONL records sequentially in one process and writes one 24 kHz mono PCM16 WAV file per record, with 200 ms of silence between turns. In streaming mode, it flushes the WAV file after each chunk.
The examples above write `outputs/single/samples/single.wav` and `outputs/dialogue/samples/dialogue.wav`.
The CLI writes generated audio to `samples/<utt>.wav` and copies reference recordings to `refer/` (for example, `refer/<utt>__prompt0.wav`). As each sample finishes, it writes a record to `<input_stem>_inference.jsonl`. Each record preserves the input fields, updates `prompt_audio` to point to the copied references, and sets `output_audio` to the generated WAV. Both paths are relative to the output directory. The `prompt_audio` field keeps its original string or list format.
Use one JSON object per input line.
- `utt` sets the output name. If omitted, the CLI generates a name from the line number. Names must be unique within the input file.
- `prompt_audio` and `prompt_text` supply the reference audio paths and their transcripts. Use a string for each field or two lists of equal length, with up to eight speakers. Relative audio paths are resolved from the directory containing the JSONL file.
- `target_text` accepts a string or a list of strings, one per turn. Use `speaker` to identify the speaker for each turn with a zero-based index. You can omit it for single-speaker input.
- `dialogue_text` can replace both `target_text` and `speaker`. For example, `"[S1]你好。[S2]你好呀。"` contains two turns. Speaker tags in this format start at 1.
- `context` supplies optional instructions for individual turns. Provide one entry per target turn. Each instruction is wrapped in `<|start_prompt|>…<|end_prompt|>` before it enters the model. These instructions do not change how much dialogue history the model retains.
Match the reference transcript to the recording. Write numbers and similar expressions as they should be spoken.
## Python API
```python
from contextlib import closing
from prompt_ttsd import PromptTTSD
model = PromptTTSD.from_pretrained("pretrained_models/PromptTTSD")
inputs = dict(
prompt_audio="ref_audio/tongtong.wav",
prompt_text="在一座美丽的森林里,住着许多动物。",
target_text="你好,欢迎体验语音生成。",
)
result = model.infer(**inputs, stream=False)
# result["audio"] is a 1-D float32 tensor on the CPU. result["sample_rate"] is 24000.
with closing(model.infer(**inputs, stream=True)) as chunks:
for chunk in chunks:
audio = chunk["audio"]
# Play or write the audio here.
# If you exit early, closing() stops background token generation.
```
The Python API resolves relative paths from the current working directory. Multi-speaker input uses the same format as the JSONL examples. Make calls sequentially on a given model instance. Each call starts a fresh dialogue context.
Each streaming chunk contains `audio`, `sample_rate`, `turn_index`, and `finalize`. Turn indices start at 0. A chunk with `finalize=True` marks the end of a turn and may contain empty audio. Silence between turns belongs to the following turn and has `finalize=False`.
As each new block of mel features arrives, streaming HiFi-GAN resynthesizes the waveform from all features accumulated for the current turn. It emits only the audio it has yet to send, holding back the last 160 ms until more features arrive or the turn ends. The streaming path uses HiFT's default excitation without short-window source caching or crossfading. Model initialization disables cuDNN TF32 to prevent numerical differences in fundamental frequency (F0) across prefix lengths from accumulating as phase drift.
## Dialogue history and context limits
Streaming and non-streaming inference use the same fixed history policy. Before adding each turn's target text, the model checks the existing context. It trims the history and rebuilds the KV cache when either of the following thresholds is exceeded.
- More than **10 turns**, including reference turns.
- More than **6,192 tokens**, counting both text and speech tokens.
When trimming, the model keeps up to **two initial reference turns**, **both text and speech for the two most recent turns**, and **text only for the two turns immediately before those**. It drops older dialogue history. If there is only one reference turn, it keeps that turn. Below or at both thresholds, it retains the full history.
The API has no option to change these thresholds or force it to retain all history. The 6,192-token threshold governs history trimming between turns. The model's context capacity still limits the length of an individual turn.
## Acknowledgments
- We borrowed a lot of code from CosyVoice.
- We borrowed a lot of code from SoulX-Podcast.
## Disclaimer
The content provided above is for academic purposes only and is intended to demonstrate technical capabilities. Some examples are sourced from the internet. If any content infringes on your rights, please contact us to request its removal.