Volt-TTS / README.md
karthik-1599's picture
Add audio samples
3b73b19
|
Raw History Blame Contribute Delete
13.4 kB
---
license: apache-2.0
pipeline_tag: text-to-speech
tags: [text-to-speech, tts, indic, voice-cloning, multilingual]
language: [kn, hi, ta, te, ml, mr, bn, gu, pa, or, as, ne, ur, sa, ks, sd, kok, mai, mni, brx, doi, sat, bho, mag, hne]
---
# Volt-TTS
<video src="https://huggingface.co/openarcane/Volt-TTS/resolve/main/Volt-TTS-teaser.mp4" controls autoplay muted loop playsinline width="100%"></video>
Volt-TTS is a text-to-speech model for 25 Indian languages, finetuned from
[Qwen3-TTS-12Hz-1.7B-Base](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-Base) by the Qwen team. It speaks in
two built-in voices, or clones any voice from a short reference clip. This is a preview release.
## Highlights
- **25 Indian languages** in one model, chosen per line.
- **Two built-in voices**: Arjun (male) and Meera (female), both speaking every language.
- **Voice cloning** from a 5-15 s clip, with or without its transcript, in any language.
## Samples
Built-in voices (Arjun, Meera) with default settings.
| Voice | Language | Text | Audio |
|---|---|---|---|
| Arjun | Kannada | ಬೆಂಗಳೂರು ಕರ್ನಾಟಕದ ರಾಜಧಾನಿ. ಇಲ್ಲಿನ ಹವಾಮಾನ ವರ್ಷವಿಡೀ ಹಿತವಾಗಿರುತ್ತದೆ. ಉದ್ಯಾನಗಳು, ಕೆರೆಗಳು ಮತ್ತು ತಂತ್ರಜ್ಞಾನ ಕಂಪನಿಗಳಿಂದ ಈ ನಗರ ದೇಶದಾದ್ಯಂತ ಪ್ರಸಿದ್ಧವಾಗಿದೆ. | <audio controls src="https://huggingface.co/openarcane/Volt-TTS/resolve/main/samples/arjun_kannada.wav"></audio> |
| Meera | Hindi | भारत में हर कुछ किलोमीटर पर भाषा और खाना बदल जाता है। यही विविधता इस देश को खास बनाती है, और हर यात्रा में कुछ नया सीखने को मिलता है। | <audio controls src="https://huggingface.co/openarcane/Volt-TTS/resolve/main/samples/meera_hindi.wav"></audio> |
| Meera | Tamil | சென்னை மெரினா கடற்கரை உலகின் நீளமான கடற்கரைகளில் ஒன்று. மாலை நேரத்தில் குடும்பங்கள் இங்கே வந்து காற்று வாங்கி, சுண்டல் சாப்பிட்டு மகிழ்கிறார்கள். | <audio controls src="https://huggingface.co/openarcane/Volt-TTS/resolve/main/samples/meera_tamil.wav"></audio> |
| Arjun | Telugu | హైదరాబాద్ బిర్యానీ రుచికి ప్రపంచమంతా అభిమానులు ఉన్నారు. చార్మినార్ దగ్గర సాయంత్రం నడుస్తూ, వేడి వేడి ఇరానీ చాయ్ తాగడం ఒక మంచి అనుభవం. | <audio controls src="https://huggingface.co/openarcane/Volt-TTS/resolve/main/samples/arjun_telugu.wav"></audio> |
| Meera | Bengali | কলকাতার দুর্গাপূজা শুধু একটি উৎসব নয়, এটি শহরের প্রাণ। প্রতিটি পাড়ায় সুন্দর প্যান্ডেল সাজানো হয়, আর রাতভর মানুষ ঠাকুর দেখতে বের হয়। | <audio controls src="https://huggingface.co/openarcane/Volt-TTS/resolve/main/samples/meera_bengali.wav"></audio> |
| Arjun | Malayalam | കേരളത്തിലെ കായലുകളിലൂടെയുള്ള ഹൗസ്ബോട്ട് യാത്ര വളരെ മനോഹരമാണ്. പച്ചപ്പ് നിറഞ്ഞ തീരങ്ങളും ശാന്തമായ വെള്ളവും മനസ്സിന് വലിയ സമാധാനം നൽകുന്നു. | <audio controls src="https://huggingface.co/openarcane/Volt-TTS/resolve/main/samples/arjun_malayalam.wav"></audio> |
| Meera | Marathi | पुण्यातला पाऊस काही वेगळाच असतो. डोंगरावरून येणारे ढग, गरम चहा आणि भजी, यामुळे पावसाळा हा सगळ्यांचा आवडता ऋतू आहे. | <audio controls src="https://huggingface.co/openarcane/Volt-TTS/resolve/main/samples/meera_marathi.wav"></audio> |
## Model details
| | |
|---|---|
| Parameters | 2.02B (the 1.93B base plus 93M for the new Indic token embeddings) |
| Base model | [Qwen3-TTS-12Hz-1.7B-Base](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-Base) |
| Languages | 25 Indian languages |
| Speech codec | 12.5 Hz (from the base model) |
| Precision | bf16 |
| License | Apache-2.0 |
## Languages
Pass the value in the `language=` column to `generate_voice_clone`.
| Language | `language=` | Language | `language=` |
|---|---|---|---|
| Assamese | `"assamese"` | Malayalam | `"malayalam"` |
| Bengali | `"bengali"` | Manipuri | `"manipuri"` |
| Bhojpuri | `"bhojpuri"` | Marathi | `"marathi"` |
| Bodo | `"bodo"` | Nepali | `"nepali"` |
| Chhattisgarhi | `"chhattisgarhi"` | Odia | `"odia"` |
| Dogri | `"dogri"` | Punjabi | `"punjabi"` |
| Gujarati | `"gujarati"` | Sanskrit | `"sanskrit"` |
| Hindi | `"hindi"` | Santali | `"santali"` |
| Kannada | `"kannada"` | Sindhi | `"sindhi"` |
| Kashmiri | `"kashmiri"` | Tamil | `"tamil"` |
| Konkani | `"konkani"` | Telugu | `"telugu"` |
| Magahi | `"magahi"` | Urdu | `"urdu"` |
| Maithili | `"maithili"` | | |
## Quickstart
Needs a CUDA GPU with about 8 GB of memory. Warnings about SoX and the tokenizer class can be ignored.
```bash
pip install -U qwen-tts soundfile
```
**Built-in voices**
```python
import soundfile as sf
import torch
from huggingface_hub import hf_hub_download
from safetensors import safe_open
from qwen_tts import Qwen3TTSModel
from qwen_tts.inference.qwen3_tts_model import VoiceClonePromptItem
tts = Qwen3TTSModel.from_pretrained("openarcane/Volt-TTS", device_map="cuda:0", dtype=torch.bfloat16)
def speaker(name):
"""A built-in voice: "arjun" (male) or "meera" (female)."""
path = hf_hub_download("openarcane/Volt-TTS", f"speakers/{name}.safetensors")
with safe_open(path, "pt") as f:
return [VoiceClonePromptItem(ref_code=f.get_tensor("ref_code"), ref_spk_embedding=f.get_tensor("ref_spk_embedding"),
x_vector_only_mode=False, icl_mode=True, ref_text=f.metadata()["ref_text"])]
wavs, sr = tts.generate_voice_clone(
text=("ಬೆಂಗಳೂರು ಕರ್ನಾಟಕದ ರಾಜಧಾನಿ. ಇಲ್ಲಿನ ಹವಾಮಾನ ವರ್ಷವಿಡೀ ಹಿತವಾಗಿರುತ್ತದೆ. "
"ಉದ್ಯಾನಗಳು, ಕೆರೆಗಳು ಮತ್ತು ತಂತ್ರಜ್ಞಾನ ಕಂಪನಿಗಳಿಂದ ಈ ನಗರ ದೇಶದಾದ್ಯಂತ ಪ್ರಸಿದ್ಧವಾಗಿದೆ."),
language="kannada", voice_clone_prompt=speaker("arjun"))
sf.write("arjun_kannada.wav", wavs[0], sr)
wavs, sr = tts.generate_voice_clone(
text="आज मौसम बहुत अच्छा है। सुबह हल्की बारिश हुई थी, इसलिए हवा में ठंडक है। चलो, शाम को पार्क में टहलने चलते हैं।",
language="hindi", voice_clone_prompt=speaker("meera"))
sf.write("meera_hindi.wav", wavs[0], sr)
```
**Voice cloning.** Give a 5-15 s clip of the voice and what is said in it. The clip can be in any language.
```python
wavs, sr = tts.generate_voice_clone(
text=("ఈ రోజు వాతావరణం చాలా బాగుంది. ఉదయం కొంచెం వర్షం పడింది, అందుకే గాలి చల్లగా ఉంది. "
"సాయంత్రం పార్కుకు నడకకు వెళ్దాం."),
language="telugu",
ref_audio="my_voice.wav", # path, URL or base64
ref_text="Transcript of my_voice.wav.", # no transcript? drop it and pass x_vector_only_mode=True
)
sf.write("my_voice_telugu.wav", wavs[0], sr)
```
**Batch.** Pass lists to speak several lines at once, each in its own language:
```python
lines = [("இன்று வானிலை நன்றாக உள்ளது. காலையில் லேசான மழை பெய்ததால் காற்று குளிர்ச்சியாக இருக்கிறது. "
"மாலையில் கடற்கரைக்குச் செல்லலாம்.", "tamil"),
("আজ আবহাওয়া খুব সুন্দর। সকালে হালকা বৃষ্টি হয়েছিল, তাই বাতাস ঠান্ডা। চলো, সন্ধ্যায় পার্কে হাঁটতে যাই।", "bengali"),
("आज हवामान खूप छान आहे. सकाळी हलका पाऊस पडला, त्यामुळे हवेत गारवा आहे. चला, संध्याकाळी बागेत फिरायला जाऊ.", "marathi"),
("Volt speaks twenty five Indian languages and English. Give it a short clip of any voice, "
"and it reads your text in that voice, in the language you choose.", "english")]
wavs, sr = tts.generate_voice_clone(text=[t for t, _ in lines], language=[l for _, l in lines],
voice_clone_prompt=speaker("meera") * len(lines))
for (_, lang), w in zip(lines, wavs):
sf.write(f"meera_{lang}.wav", w, sr)
```
The default sampling settings (`temperature=0.9, top_k=50, repetition_penalty=1.05`) come from
`generation_config.json`. Pass any of them to `generate_voice_clone` to change them, e.g. `temperature=0.7`.
Call `torch.manual_seed(...)` first for a repeatable result.
## Benchmark (FLEURS, 8 languages)
We benchmarked on **FLEURS** (Google), using its real **audio and text**. Each language uses test items from
the FLEURS test split. For each item, the voice reference is a real FLEURS recording with its transcript, and the
model reads a different FLEURS test sentence in that voice. None of this data was used in training.
**Note:** these are early, small-scale numbers from one ASR system, and they leave out English loanwords, so
they are lower than a strict score. Counting every word, CER is higher, for example Hindi
7.5 %, Malayalam 2.8 % and Tamil 4.0 %. Treat them as a rough guide, not a full evaluation.
| Language | CER | WER |
|---|---|---|
| Hindi | 1.0 % | 4.1 % |
| Malayalam | 1.7 % | 13.5 % |
| Kannada | 2.0 % | 9.3 % |
| Telugu | 2.3 % | 10.0 % |
| Odia | 2.7 % | 12.4 % |
| Tamil | 2.8 % | 15.2 % |
| Punjabi | 3.1 % | 8.4 % |
| Assamese | 5.6 % | 21.8 % |
| **Average** | **2.7 %** | **11.8 %** |
- **CER / WER:** the generated audio was transcribed with ElevenLabs Scribe v3, with the language forced. Each
transcript was compared with the target sentence after Unicode normalisation. CER is computed per language,
then averaged.
- **English loanwords:** Scribe often writes English loanwords in Latin script even when they are spoken
correctly, which the metric counts as errors. These regions are excluded from both the reference and the
transcript. Raw CER with them included is higher, for example Hindi 7.5 %, Malayalam 2.8 % and Tamil 4.0 %.
- **Speaker similarity:** the cloned voice scores above 0.94 cosine similarity to the reference speaker
(Malayalam 0.942, Tamil 0.943; WavLM-base-plus-sv embeddings). Two halves of the same real recording score
about 0.96, the practical ceiling.
- Sampling settings are the defaults above (temperature 0.9, top-k 50), so every run differs slightly.
## Training
- **Method**: LoRA finetuning (rank 32, alpha 64, 40.5M trainable parameters) on the attention, MLP, text
projection and output layers; the new token embeddings were trained in full.
- **Data**: Rasa (22 languages), SYSPIN (9 languages) and LibriTTS-R (English): 1.18M clips, about 2,240 hours,
sampled so smaller languages are not drowned out.
- **Tokenizer**: extended with 45,622 Indic pieces (vocab 197,298) and 25 new language ids.
- **Voice cloning**: about half the clips were trained with a reference clip and its transcript, the rest with the
speaker embedding only.
- **Validation**: held-out set of 2,291 clips, loss 2.31 (base model before training: 4.22).
## Limitations
- Preview release. Quality varies by language and speaker; low-resource languages are less tested.
- No text normaliser yet: numbers, symbols and maths may be read wrongly.
- Cross-lingual cloning (for example an English voice speaking Kannada) works but was not trained explicitly.
## License and credits
Apache-2.0. Volt-TTS is a finetune of [Qwen3-TTS-12Hz-1.7B-Base](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-Base)
by the Qwen team. It uses the `qwen-tts` package, which is why the config says `qwen3_tts`.
Training data: Rasa (AI4Bharat), SYSPIN (IISc / ARTPARK) and LibriTTS-R. Please follow their licenses when using
this model. The built-in voices Arjun and Meera each come from one SYSPIN clip.