Fealtyy/dhvani-tts

Dhvani TTS

Dhvani TTS reads Hindi, English and the code-mixed Hinglish people actually type, in any script, in one consistent voice built for voice agents.

Dhvani TTS is a 337M-parameter flow-matching text-to-speech model with a single voice, "Dhvani". You give it text the way users write it: Devanagari, Roman-script Hindi, English, or all three in one sentence. It speaks that text without your having to transliterate anything first. It is designed for collections, payment-reminder and customer-service agents, including those on the telephone network.

Model details

Developed by	Fealtyy
Model type	Non-autoregressive text-to-speech: Diffusion Transformer with conditional flow matching and per-character language conditioning
Parameters	337M
Architecture	F5-TTS Diffusion Transformer (dim 1024, depth 22, 16 heads), 100-bin mel at 24 kHz, Vocos vocoder; see config.json
Weights	weights/dhvani_tts.safetensors (fp32, ~1.35 GB)
Languages	Hindi (hi), English (en) and Hinglish code-mixing; Devanagari and Latin-script input
Voice	One speaker, "Dhvani", defined by the reference clip shipped with the model
Output	24 kHz waveform with phone-band content (≤ 4 kHz); optional 48 kHz bandwidth extension
Fine-tuned from	ai4bharat/IndicF5
License	Proprietary. See License

Highlights

• Text as typed. Roman Hindi ("aapka payment kal due hai"), English and Devanagari can appear in the same sentence. A contextual language tagger labels each Latin-script word as Hindi or English, and the model is conditioned on those labels.
• Accurate on everyday Hinglish. On a held-out test set, the character error rate on Roman-script Hindi is 0.019, against 0.104 for Veena (maya-research), the strongest public baseline we evaluated; over all 150 test sentences it is 0.0436 against 0.0555.
• English words in Latin script. Common English words typed in Latin script are pronounced far more accurately: on the validation set, pure-English error fell from 0.032 (v1.1) to 0.004 (v1.2), and "balance" is correct in 24 of 24 held-out sentences (10 of 24 in v1.0).
• Safe with numbers. Amounts, dates, times, phone and account numbers, acronyms such as KYC, EMI and CIBIL, and Indian names are rewritten into speakable words before synthesis.
• Built for the phone line. The voice is band-limited to the telephone range, so it passes through an 8 kHz G.711 channel essentially unchanged. A 48 kHz mode is available for playback on speakers.

Audio samples

Each sample was generated from the input exactly as typed below, in 48 kHz (HD) mode.

File	Input text (as typed)	Audio
samples/roman_hinglish.wav	Namaste Rohit ji, main Sunrise Finance se baat kar rahi hoon. Aapka Rs. 2,400 ka payment 10 November tak due hai.
samples/devanagari_hinglish.wav	आपकी EMI ₹12,450 की है, जो 15/11/2026 को due है। क्या आप आज payment कर सकते हैं?
samples/roman_hindi.wav	Kya aap mujhe bata sakte hain ki aap payment kab karenge?
samples/english.wav	Your account ending 0960 has an overdue amount of Rs. 6,487. Please pay by 5:30 pm today.

Quick start

Install PyTorch for your CUDA version first, then the rest:

pip install torch==2.9.0 torchaudio==2.9.0
pip install -r requirements.txt

Command line

python speak.py "Namaste Rohit ji, aapka Rs. 2400 ka payment 10 November tak due hai." out.wav
python speak.py "Namaste Rohit ji, aapka Rs. 2400 ka payment 10 November tak due hai." out.wav --hd   # 48 kHz

Python

from dhvani_tts import DhvaniTTS

tts = DhvaniTTS()     # reads production.json and loads weights/dhvani_tts.safetensors once

wav, sr = tts.speak("Aapki EMI kal due hai, kripya aaj hi payment kar dijiye.")
# wav is a float32 NumPy array; sr is 24000 (phone-band content)

wav, sr = tts.speak("Aapki EMI kal due hai.", hd=True)   # sr is 48000

# See exactly what the model will read, after number, name and script handling
print(tts.model_text("Rs. 2,400 ka payment 15/11/2026 tak due hai"))

Web demo

TTS_SHARE=0 python app.py      # http://127.0.0.1:7866

The demo offers two engines: "Dhvani (default)", and "Dhvani + convert to Devanagari (most intelligible, measured)", which writes the whole input in Devanagari, English included.

Run everything from the repository root, because the modules import each other by name. The first run downloads the public language tagger, vocoder and bandwidth extension model.

How it works

text ─▶ normalisation ─▶ language tagging ─▶ flow-matching DiT ─▶ vocoder ─▶ post-processing ─▶ audio

1. Normalisation
   • Numbers, amounts, dates and times become Devanagari words (Rs. 2,400 → दो हज़ार चार सौ रुपये).
   • Phone and account numbers are read digit by digit.
   • Acronyms become letter names (KYC → के वाई सी).
   • Indian names are written in Devanagari (Rohit → रोहित).
   • Roman-script Hindi words are written in Devanagari (khate → खाते), while English words keep their spelling. This step changes the script only, never the words, and needs IndicXlit (see Optional components).
2. Language tagging. A MuRIL-based contextual tagger labels every remaining Latin-script word as Hindi or English. The labels become per-character language ids for the model.
3. Synthesis. A Diffusion Transformer generates a mel spectrogram by conditional flow matching: 32 sampling steps, classifier-free guidance 3.0. It is conditioned on the voice reference (refs/dhvani_reference.wav and its exact transcript). Long inputs are synthesised sentence by sentence.
4. Vocoding. Vocos converts the mel spectrogram to a 24 kHz waveform.
5. Post-processing. Leading and trailing silence is trimmed and over-long pauses are shortened. With hd=True, LavaSR adds content above 4 kHz and outputs 48 kHz; the original audio below 4 kHz is kept unchanged.

Uses

Direct use

• Voice agents that speak Hindi, English and Hinglish: collections, payment reminders, customer service and notifications.
• Prompts and announcements for telephony systems (8 kHz).

Downstream use

• The speech stage behind an LLM-based agent that produces Hinglish text in any script.

Out-of-scope use

• Other voices. The model speaks one voice. Using other reference clips to clone other speakers is not supported and has not been evaluated.
• Impersonation and deception. Do not use the voice to pose as a real person. Tell callers they are speaking with an automated system wherever law or policy requires it.
• Other languages. Only Hindi and English were evaluated.
• Expressive or emotional speech. The model speaks in one neutral style.
• Long-form narration, such as audiobooks, has not been evaluated.

Evaluation

Protocol

• Test set. 150 held-out Hinglish sentences, 25 in each of six categories: pure Hindi, pure English, Roman-script Hindi, Devanagari Hinglish, Roman Hinglish and banking text. None of these sentences appears in the training text.
• Metric.
  • Every system's audio is passed through an 8 kHz phone channel, then transcribed with Whisper large-v3.
  • The transcript is scored against a Devanagari reference by convention-tolerant character error rate (CER). This metric ignores spelling variants that sound the same.
  • Differences between systems are paired bootstrap 95 % confidence intervals over sentences.
• English. Svarah, 100 sentences of Indian-English speech, scored by word error rate (WER) with Whisper-small.
• Naturalness. UTMOS, a learned estimate of mean opinion score.

Results: held-out test (CER, lower is better)

Category	Dhvani TTS	Veena (maya-research)	Difference (95 % CI)
Roman-script Hindi	0.019	0.104	−0.086 (−0.118 to −0.053)
Pure Hindi	0.019	0.039	−0.020 (−0.043 to +0.003)
Roman Hinglish	0.041	0.039	+0.002 (−0.019 to +0.024)
Devanagari Hinglish	0.042	0.026	+0.017 (−0.004 to +0.040)
Banking text	0.101	0.084	+0.017 (−0.030 to +0.067)
Pure English	0.040	0.041	−0.001 (−0.053 to +0.051)
All	0.0436	0.0555	−0.012 (−0.027 to +0.004)

Both systems were scored with the identical protocol on the same 150 sentences; differences are paired over sentences. Overall, Dhvani TTS has 21 % fewer errors, a difference that is not statistically significant. It is significantly better on Roman-script Hindi; no category is significantly worse. Vaakya, evaluated the same way, scores 0.082.

Results: effect of Roman-Hindi handling (CER, validation set of 150 sentences, v1.0)

Input handling	All	Roman-script Hindi
Roman Hindi read as typed	0.0439	0.104
Roman Hindi written in Devanagari (default)	0.0282	0.026

The difference is −0.0157, with a 95 % confidence interval of −0.023 to −0.009. With the default handling, version 1.1 scores 0.0264 and version 1.2 scores 0.0157 on the same validation set (difference −0.011, 95 % CI −0.020 to −0.003).

English and naturalness

• Svarah (Indian English): WER 0.139 (v1.0). Human recordings score 0.080 with the same recogniser.
• UTMOS: 3.70 for the native phone-band output; 3.79 with the 48 kHz extension (v1.0).

Versions

Version	Date	Changes
1.2	2026-10-26	More accurate common English words in Latin script (validation pure English 0.032 → 0.004; validation overall 0.0264 → 0.0157, significant). Held-out test CER 0.0455 → 0.0436.
1.1	2026-10-23	More accurate English words in Latin script ("balance": 24/24 held-out sentences, from 10/24). Held-out test CER 0.0557 → 0.0455.
1.0	2026-10-22	First release.

Limitations

• English pronunciation carries an Indian accent, and some words can still be rendered Hindi-style. Very short sentences (four or five words) that mix a Hindi/English homograph such as "main" are the least reliable input; banking-style sentences of that kind score below Veena.
• Bandwidth. Native output carries no content above 4 kHz, so it sounds muffled on speakers. Use hd=True for speaker playback. On a telephone call it makes no audible difference.
• Names. 237 common Indian names are built in (indian_names.json). Other names need IndicXlit. A name at the start of a sentence is treated as a name only when an honorific follows, such as ji, sir or madam.
• One voice, one style. There is no emotion or style control.
• Metric limits. ASR-based scores cannot judge accent or naturalness, and the recogniser occasionally paraphrases instead of transcribing. Listen to the output before relying on these numbers for a new domain.

Bias, risks and recommendations

• Disclosure. The voice is synthetic but natural-sounding, and listeners may not realise they are hearing a machine. Disclose automated calls as your jurisdiction requires.
• Accuracy in financial calls. Payment and collections calls have real consequences for the people who receive them. Check amounts and dates with tts.model_text(...) before deployment.
• Dialect coverage. Coverage is centred on standard Hindi; regional pronunciations may be rendered less naturally.

Optional components

IndicXlit (recommended). IndicXlit writes Roman-script Hindi words and unlisted Indian names in Devanagari. It needs Python 3.10:

python3.10 -m venv xlit_venv310
xlit_venv310/bin/pip install ai4bharat-transliteration

Alternatively, set HINDI_XLIT_PYTHON to an existing Python 3.10 interpreter that has it installed. Without IndicXlit nothing breaks, but Roman-script Hindi is read as typed. That is less accurate: validation CER is 0.044 instead of 0.028.

faster-whisper. Enables tts.speak(text, takes=N), which synthesises N takes per sentence and keeps the one the recogniser transcribes best.

Repository contents

Path	Description
dhvani_tts.py	DhvaniTTS: the entry point and text pipeline
speak.py	Command-line synthesis
app.py	Gradio web demo
config.json	Model configuration: architecture, mel settings, vocoder, language ids, sampling
production.json	Weights path and sampling settings used by dhvani_tts.py
weights/dhvani_tts.safetensors	Model weights (safetensors, fp32, 337M parameters, ~1.35 GB)
samples/	Four 48 kHz audio samples (see Audio samples)
refs/dhvani_reference.wav, refs/dhvani_reference.json	Voice reference and its exact transcript
models/IndicF5/checkpoints/vocab.txt	Character vocabulary
indian_names.json	Built-in Indian name list (237 names)
hindi_frontend.py, hindi_translit.py	Text normalisation and transliteration
contextual_lid.py, language_tags.py	Hindi/English language tagging
engines_hi.py, f5_language.py, language_conditioning.py	Model construction and language-conditioned inference
bwe.py	48 kHz bandwidth extension
tsm.py, text_metrics.py, xlit_worker.py	Time-scale modification, CER for take selection, IndicXlit worker
harshall_lib_normalize.py	Third-party Hinglish normaliser, included verbatim (see NOTICE)
LICENSE	Licence for the Dhvani TTS weights, voice and reference
NOTICE	Third-party attributions and licences

Training

Dhvani TTS was fine-tuned from ai4bharat/IndicF5 on a proprietary corpus of Hindi, Hinglish and English speech. The training data and training recipe are not disclosed.

License

The Dhvani TTS weights, the Dhvani voice and the reference clip are proprietary. All rights are reserved by Fealtyy; see LICENSE. Third-party components are distributed under their own licences; see NOTICE.

Citation

@misc{fealtyy2026dhvanitts,
  title        = {Dhvani TTS: Hinglish Text-to-Speech for Voice Agents},
  author       = {{Fealtyy}},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/}},
  note         = {Fine-tuned from ai4bharat/IndicF5}
}

@article{chen2024f5tts,
  title   = {F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching},
  author  = {Chen, Yushen and Niu, Zhikang and Ma, Ziyang and Deng, Keqi and Wang, Chunhui and Zhao, Jian and Yu, Kai and Chen, Xie},
  journal = {arXiv preprint arXiv:2410.06885},
  year    = {2024}
}

@misc{ai4bharat2025indicf5,
  title        = {IndicF5: High-Quality Text-to-Speech for Indian Languages},
  author       = {{AI4Bharat}},
  year         = {2025},
  howpublished = {\url{https://huggingface.co/ai4bharat/IndicF5}}
}

Contact

Fealtyy.