DviTTS

Zero-shot text-to-speech for English and six Indian languages: Hindi, Marathi, Bengali, Punjabi, Gujarati and Odia. Give it 3–10 s of someone's speech and its transcript and it reads new text in that voice, or pick one of 18 named voices.

Base model: stage 1 starts from Fun-CosyVoice3-0.5B-2512 (FunAudioLLM / Alibaba, Apache 2.0), a 0.5B language model that already knows how to produce speech. We retrained it to emit IndiCodec ids and to speak six Indian languages.

The DviTTS family:

Model Size Runs on Voices Get it
DviTTS (this repo) 0.5B stage 1 + 130M flow + IndiCodec decoder CUDA GPU (A10G: 0.37x real time) clone from 3–10 s, or 18 presets pip install (below)
DviTTS-Nano 110M stage 1 + 1-step flow, ONNX phone or laptop CPU, no PyTorch 18 presets Android app + code · demo

More samples from both models: the demo page.

Samples

Prompts are speakers the model never saw in training, from LibriTTS-R test-clean and FLEURS. Each output is one take (seed 0), not picked from several. Made with the released package (pip install) on a laptop RTX 3050. Every output carries the AudioSeal watermark.

LanguagePrompt (unseen speaker)DviTTS
English
prompt: LibriTTS-R

It might rain this evening, so please take an umbrella when you leave the house.
Hindi
prompt: FLEURS

आज शाम बारिश हो सकती है, इसलिए घर से निकलते समय छाता साथ ले जाइए।
Marathi
prompt: FLEURS

आज संध्याकाळी पाऊस पडण्याची शक्यता आहे, त्यामुळे घरातून निघताना छत्री सोबत घ्या.
Bengali
prompt: FLEURS

আজ সন্ধ্যায় বৃষ্টি হতে পারে, তাই বাড়ি থেকে বেরোনোর সময় ছাতা সঙ্গে নিন।
Punjabi
prompt: FLEURS

ਅੱਜ ਸ਼ਾਮ ਨੂੰ ਮੀਂਹ ਪੈ ਸਕਦਾ ਹੈ, ਇਸ ਲਈ ਘਰੋਂ ਨਿਕਲਦੇ ਸਮੇਂ ਛਤਰੀ ਨਾਲ ਲੈ ਜਾਓ।
Gujarati
prompt: FLEURS

આજે સાંજે વરસાદ પડી શકે છે, તેથી ઘરેથી નીકળતી વખતે છત્રી સાથે લઈ જજો.
Odia
prompt: FLEURS

ଆଜି ସନ୍ଧ୍ୟାରେ ବର୍ଷା ହୋଇପାରେ, ତେଣୁ ଘରୁ ବାହାରିବା ବେଳେ ଛତା ସାଙ୍ଗରେ ନିଅନ୍ତୁ।
Hindi prompt, English text
The meeting starts at 10:30 on Monday, so please be on time.

Install

python -m venv dvitts-env && source dvitts-env/bin/activate      # a fresh environment: the package pins its torch range
pip install "git+https://github.com/codeprakhar25/dvitts@v1.0.0"

Python 3.10+ and a CUDA GPU; installs PyTorch 2.6–2.8 (RTX 50-series needs 2.7+; not yet tested there). Weights download from this repo on first use; the codec comes from IndiCodec.

Windows: PyPI's Windows torch is CPU-only, so install the CUDA build first, then the package: pip install torch==2.8.0 torchaudio==2.8.0 --index-url https://download.pytorch.org/whl/cu128.

Use

dvitts "नमस्ते, आप कैसे हैं?" --voice ananya -o hello.wav
dvitts "Please call me back after the meeting." --prompt me.wav --prompt-text "exact words spoken in me.wav" -o out.wav
dvitts --list-voices
import soundfile as sf
from dvitts import DviTTS

tts = DviTTS.from_pretrained("prakharkhatri/dvitts")
y, sr = tts.say("ananya", "कल सुबह हम स्टेशन पर मिलेंगे।")              # a preset voice
sf.write("hello.wav", y, sr)

w, wsr = sf.read("me.wav", dtype="float32")                            # clone: 3-10 s of clean speech
v = tts.voice(w, wsr, "exact words spoken in me.wav")
y, sr = tts.say(v, "The meeting starts at 10:30, so please be on time.")

y, sr = tts.say("sneha", "आज संध्याकाळी पाऊस पडण्याची शक्यता आहे.", lang="mr")   # Devanagari reads as Hindi unless lang="mr"
for chunk, sr, info in tts.say_stream(v, "First sentence. Second sentence. Third sentence."):
    play(chunk, sr)                                                     # sentence by sentence: first audio early

Named voices (labels, not the speakers' names):

Language Female Male
English emma, grace james, oliver
Hindi ananya, kavya arjun, rohan
Marathi sneha aditya
Bengali ishita arnab
Punjabi jasleen gurpreet
Gujarati hetal kunal
Odia subhashree debasis

Languages

FLEURS, 100 sentences per language, unseen speakers as prompts (prompts cut to 10 s), CER in % by IndicConformer on our audio and on the real recording of the same sentence.

Language CER DviTTS CER real recording
Hindi 2.3 3.8
Marathi 2.6 2.8
Bengali 2.7 3.8
Punjabi 5.2 6.2
Gujarati 3.8 3.6
Odia 3.1 4.0

Tamil, Telugu, Kannada and Malayalam are to be added.

Benchmarks (unseen speakers only)

Whisper large-v3 for English, IndicConformer for Hindi; prompt transcripts come from ASR of the prompt (a user supplies only audio); seed per item; watermark on; AWS A10G.

Set n CER % WER %
Seed-TTS test-en 1,088 0.82 2.04
CMU Arctic 350 0.36 1.45
LibriTTS-R test 200 0.86 3.07
IndicVoices Hindi 300 3.53 8.92
FLEURS Hindi 100 4.52 9.01

FLEURS Hindi here is a different 100-sentence sample and prompt pairing from the language table above. Dead air of 1.5 s or more: 3 of 2,038 clips (all IndicVoices Hindi); clips that never ended: 0.

LibriSpeech-PC test-clean (F5-TTS protocol)

The F5-TTS cross-sentence list (1,127 pairs, 4–10 s targets), the protocol used by F5-TTS, ZipVoice and OmniVoice. Same audio scored both ways; "real" is the real target recording through the same ASR.

WER % (HuBERT-large) WER % (Whisper large-v3)
DviTTS 2.05 2.12
Real recording 1.89 2.37

MiniMax multilingual test set, Hindi (voice cloning)

MiniMaxAI/TTS-Multilingual-Test-Set Hindi: 100 lines, 2 Common Voice prompt speakers, Whisper large-v3 (language="hi"). How the text is normalised changes the number a lot, so three scorings of the same audio are given.

Scoring WER % CER %
Whole words 16.82 5.90
Whisper's stock text normaliser (removes vowel signs, as most papers' scripts do) 8.67 —
IndicConformer instead of Whisper 3.10 1.18

Published Whisper-large-v3 numbers on this set: OmniVoice 4.33, ElevenLabs Multilingual v2 5.83, MiniMax-Speech 6.96. Papers rarely say which Hindi normaliser they used, so compare with care. 9 of the 100 lines contain digits, which the front end reads as words and Whisper often writes back as digits.

Speed and memory

A10G, bf16 stage 1 on a CUDA graph, 16-step flow, trim and watermark on:

Real-time factor (Seed-TTS) 0.37
First audio, one sentence 1.54 s English, 2.73 s Hindi (say_stream starts after the first sentence)
GPU memory, speaking 2.2 GB allocated, ~3.2 GB in nvidia-smi
GPU memory, cloning a new voice peak 4.0 GB for ~0.6 s (the prompt encoder visits the GPU, then goes back to CPU RAM)

A 6 GB GPU runs presets; cloning is tight. CPU (device="cpu") works but is slow: use DviTTS-Nano on CPUs.

Defaults

Step Default Change it Why
Text normalisation on normalize=False digits are read as digits otherwise
Long text split at sentence ends (over 500 English / 400 Indian characters, or ~25 s of speech) say_long(..., gap_s=) one long call drops its opening and loops
Prompt over 10 s cut at the last pause before 10 s pass 10 s or less the model sees 10 s of prompt
Prompt under 3 s warning — 2–3 s prompts raise English word error by 6–8 points
Edge trim on DviTTS(..., trim=False) removes silence at the clip edges

Watermark

Every output carries an inaudible AudioSeal mark. Check a clip with dvitts --detect clip.wav. If you deploy this model, label its output as AI-generated where your local rules require it.

Limitations

  • No emotion or style control.
  • Five of the six Indian languages learned from fewer speakers; voices in those languages may clone less well.
  • Noisy prompts can give a take full of loud noise (rare). Use clean speech.
  • English number reading uses simple rules unless pip install "dvitts[en-norm]" (NeMo text processing, Linux).

Intended use and ethical considerations

For research and for building speech products with voices you have permission to use. Not for impersonation, fraud, or cloning anyone's voice without their consent.

The preset voices are speakers from Rasa and IndicVoices-R (AI4Bharat) and LibriTTS-R, all CC BY 4.0, credited in Acknowledgements. Display names (Ananya, Emma, …) are labels, not the speakers' names.

Do not use these voices to imitate the real speakers, or to make anyone believe they are hearing them. Clone only voices you have permission to use. Audio from this model carries an AudioSeal watermark by default.

Citation

If you use DviTTS, please cite:

@misc{khatri2026dvitts,
  title        = {DviTTS: Text-to-Speech for English and Six Indian Languages},
  author       = {Khatri, Prakhar},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/prakharkhatri/dvitts}}
}

DviTTS builds on CosyVoice 3 (its base model) and AudioSeal (its watermark); please cite them too:

@article{du2025cosyvoice,
  title   = {CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-training},
  author  = {Du, Zhihao and Gao, Changfeng and Wang, Yuxuan and Yu, Fan and Zhao, Tianyu and Wang, Hao and Lv, Xiang and Wang, Hui and Shi, Xian and An, Keyu and others},
  journal = {arXiv preprint arXiv:2505.17589},
  year    = {2025}
}

@article{sanroman2024proactive,
  title   = {Proactive Detection of Voice Cloning with Localized Watermarking},
  author  = {San Roman, Robin and Fernandez, Pierre and Elsahar, Hady and D{\'e}fossez, Alexandre and Furon, Teddy and Tran, Tuan},
  journal = {ICML},
  year    = {2024}
}

Acknowledgements

  • Base model: Fun-CosyVoice3-0.5B-2512 (FunAudioLLM, Alibaba; Apache 2.0) and its Qwen2 text tokenizer.
  • Watermark: AudioSeal (Meta; MIT). Silence trim: Silero VAD (MIT).
  • Text front end: num2words (LGPL-2.1, used unmodified) and indic-numtowords (AI4Bharat; MIT); optional NeMo text processing (NVIDIA; Apache 2.0) for English.
  • Training data (CC BY 4.0): Emilia-YODAS English subset (Amphion), LibriTTS and LibriTTS-R, VCTK, and from AI4Bharat Rasa, IndicVoices-R and IndicVoices (training split).
  • Preset voices: speakers from Rasa, IndicVoices-R and LibriTTS-R (CC BY 4.0). They belong to those speakers and must not be used to impersonate them.
  • Samples on this card: prompts from FLEURS (Google; CC BY 4.0) and LibriTTS-R test-clean, speakers not seen in training.

Licence

Apache 2.0 (LICENSE).

Downloads last month
30
Safetensors
Model size
0.5B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for prakharkhatri/dvitts

Finetuned
(22)
this model
Quantizations
1 model

Collection including prakharkhatri/dvitts

Paper for prakharkhatri/dvitts