OmniVoice-Urdu-3h

Next-Generation Fine-Tuned Model for High-Fidelity Urdu Zero-Shot Text-To-Speech, Voice Cloning & Voice Design

---

🎯 Model Overview

This checkpoint is a specialized Urdu adaptation of the foundational OmniVoice (k2-fsa/OmniVoice) framework. While the original architecture supports 600+ languages through flow-matching diffusion transformers, this fine-tuned release integrates ~3 hours of clean, diverse Urdu speech data to dramatically enhance phonetic accuracy, native prosody, and contextual phrasing.

Property	Description / Value
🧠 Base Model	k2-fsa/OmniVoice (Diffusion Transformer DLM)
🎧 Training Corpus	~3 Hours High-Quality Clean Multi-Speaker Urdu Speech
🌐 Language Support	Urdu (Primary), English (Preserved from Base)
⚙️ Core Architecture	Diffusion Transformer (DiT) with Flow Matching Formulation
🔊 Sample Rate & Format	24 kHz Waveform Output
⚡ Real-Time Factor (RTF)	~0.025 (Up to 40× faster than real-time on NVIDIA H100 GPU)
📝 License	Apache-2.0

---

🌟 Capabilities

🎤 1. Zero-Shot Voice Cloning

Instantly clone any target Urdu speaker using a 3–10 second clean reference clip — no additional fine-tuning or adaptation steps required.

audio = model.generate(
    text="آج موسم کتنا خوبصورت ہے,",
    ref_audio="speaker_ref.wav",
    # ref_text is automatically transcribed via Whisper if omitted
)

🎨 2. Voice Design

Synthesize natural Urdu speech with custom synthetic persona prompts — specify age, gender, pitch, and accent using plain text instructions.

audio = model.generate(
    text="یہ ایک تجربہ ہے آواز ڈیزائن کا,",
    instruct="female, young, high pitch",  # or "male, deep voice, pakistani accent"
)

🧠 3. Auto Voice Synthesis

Quickly synthesize natural, well-balanced Urdu speech without specifying any reference prompt or instructions.

audio = model.generate(text="میں آپ کی مدد کروں گا,")

🔧 Advanced Generation & Paralinguistic Controls

Control Feature	Parameter Syntax	Code Example
Non-verbal Cues	Inline tags inside text	[laughter], [sigh], [surprise-ah]
Speech Pacing	speed multiplier float	speed=1.2 (Faster) / speed=0.85 (Slower)
Exact Duration	duration target in seconds	duration=5.0
Quality vs Speed	num_step diffusion steps	num_step=16 (Fast) / 32 (High Fidelity)

---

🚀 Quick Start

Installation

pip install omnivoice

# Optional GPU Acceleration via FlashInfer (Recommended for Production)
pip install flashinfer-python==0.6.15.post1 "flashinfer-jit-cache==0.6.15.post1+cu128" \
    --extra-index-url https://flashinfer.ai/whl/cu128/

Python API Usage

import soundfile as sf
import torch
from omnivoice import OmniVoice

# Load fine-tuned Urdu model
model = OmniVoice.from_pretrained(
    "mlwithabdullah/omnivoice-urdu-3h",
    device_map="cuda:0",  # Options: "cuda:0", "mps", "xpu", or "cpu"
    dtype=torch.float16,  # fp16 recommended for CUDA
    load_asr=True,  # Enables Whisper auto-transcription for reference audio
)

# 1. Zero-Shot Voice Cloning
audio = model.generate(
    text="پاکستان زندہ آباد، میرا وطن میرا جان,",
    ref_audio="urdu_speaker.wav",
)
sf.write("cloned_urdu.wav", audio[0], 24000)

# 2. Voice Design (Prompted Synthesis)
audio = model.generate(
    text="اس ماڈل نے بہت کام کیا ہے,",
    instruct="male, deep voice, pakistani accent",
)
sf.write("designed_urdu.wav", audio[0], 24000)

💻 Command Line Interface (CLI) Commands

# Voice Cloning via CLI
omnivoice-infer \
    --model mlwithabdullah/omnivoice-urdu-3h \
    --text "اردو میں بولنا اب آسان ہے," \
    --ref_audio urdu_ref.wav \
    --output cloned.wav

# Voice Design via CLI
omnivoice-infer \
    --model mlwithabdullah/omnivoice-urdu-3h \
    --text "آواز ڈیزائن کا تجربہ," \
    --instruct "female, soft, pakistani accent" \
    --output designed.wav

🌐 Local Interactive Web Interface

omnivoice-demo --share

---

📊 Fine-tuning Details

Hyperparameter / Setting	Value
Training Dataset	~3 Hours Multi-speaker Clean Urdu Audio Speech
Manifest / Data Format	JSONL Manifests + WebDataset Tar Shards
Total Optimizer Steps	5,000 steps (Fine-tuned from base checkpoint)
Learning Rate	5e-5
Batch Size & Hardware	8 samples / GPU (2× NVIDIA H100 SXM on Modal Cloud)
Mixed Precision	fp16 (Ampere/Hopper bf16 compatible)
Attention Optimization	PyTorch SDPA / flex_attention

# Key training configuration settings:
init_from_checkpoint: "k2-fsa/OmniVoice"
steps: 5000
learning_rate: 5e-5

---

📈 Evaluation

A head-to-head benchmark evaluation was conducted comparing k2-fsa/OmniVoice (Base Model) against mlwithabdullah/omnivoice-urdu-3h (Fine-tuned) using a 5-sentence test suite spanning formal and casual Urdu speech registers.

Metric	Base Model (k2-fsa/OmniVoice)	Fine-Tuned (OmniVoice-Urdu-3h)	Relative Improvement
Mean WER (Word Error Rate) ⬇	22.32%	4.30%	5.2× Reduction
Mean CER (Character Error Rate) ⬇	18.34%	1.62%	11.3× Reduction
Speaker Similarity (SECS) ⬆	0.7998	0.7555	Minimal Trade-off

💡 Key Takeaways & Insights

• Pronunciation & Intelligibility: Word Error Rate dropped dramatically from 22.3% down to 4.3%. While the base model occasionally struggled with corrupted scripts or Hindi script confusion on Urdu inputs, the fine-tuned checkpoint outputs consistent native Urdu speech.
• Speaker Similarity: A slight drop in cosine speaker similarity (0.80 → 0.76 SECS) is standard when adapting zero-shot models to target prosodic dynamics. 0.75+ SECS maintains strong voice-cloning fidelity.

📄 Detailed Sentence-Level Benchmark Breakdown

#	Urdu Sentence Fragment	Base WER	Base CER	Base SECS	FT WER	FT CER	FT SECS
1	السلام علیکم، آپ کی کال ہماری کسٹمر سپورٹ...	4.55%	2.17%	0.7979	4.55%	1.09%	0.7679
2	اوئے بھائی، ایک بات بتاؤں؟ لیکن دیکھو...	3.70%	0.83%	0.7933	3.70%	0.83%	0.7273
3	اگر آپ اپنا اکاؤنٹ بیلنس چیک کرنا چاہتے ہیں...	100%*	87.96%*	0.7962	4.55%	0.93%	0.7613
4	ارے یار، کل محلے میں کیا ہوا تمہیں پتا ہے؟...	0.00%	0.00%	0.7892	8.70%	5.26%	0.7122
5	آپ کی درخواست کامیابی کے ساتھ موصول ہو گئی...	3.33%	0.75%	0.8222	0.00%	0.00%	0.8089

*Sentence 3 resulted in output corruption on the base model, leading to ASR script mismatch (Devanagari script return).

---

🔊 Audio Samples

Below is the direct audio playback comparison across all 5 benchmark sentences comparing the Base Model vs this Fine-Tuned Checkpoint.

#	Urdu Benchmark Sentence	Base Model (k2-fsa/OmniVoice)	Fine-Tuned (OmniVoice-Urdu-3h)
1	السلام علیکم، آپ کی کال ہماری کسٹمر سپورٹ ٹیم تک پہنچ گئی ہے...
2	اوئے بھائی، ایک بات بتاؤں؟ لیکن دیکھو...
3	اگر آپ اپنا اکاؤنٹ بیلنس چیک کرنا چاہتے ہیں...
4	ارے یار، کل محلے میں کیا ہوا تمہیں پتا ہے؟...
5	آپ کی درخواست کامیابی کے ساتھ موصول ہو گئی ہے...

---

💡 Best Practices for Urdu Inference

1. Reference Selection: Provide 3–10 seconds of clear, background-noise-free Urdu audio for best voice cloning accurate timbre.
2. Text Normalization: Pass normalize_text=True when generating speech containing digits, dates, or complex punctuation.
3. Hardware Acceleration: Install FlashInfer to achieve up to 2.6× speedups on modern NVIDIA GPUs.

---

📚 Citation & Acknowledgments

@article{zhu2026omnivoice,
  title={OmniVoice: Towards Omnilingual Zero-Shot Text-to-Speech with Diffusion Language Models},
  author={Zhu, Han and Ye, Lingxuan and Kang, Wei and Yao, Zengwei and Guo, Liyong and Kuang, Fangjun and Han, Zhifeng and Zhuang, Weiji and Lin, Long and Povey, Daniel},
  journal={arXiv preprint arXiv:2604.00688},
  year={2026}
}

Fine-tuned on Urdu speech data by Abdullah (mlwithabdullah/omnivoice-urdu-3h).

---

⚖️ License & Ethics

This model is licensed under Apache-2.0.

⚠️ Ethical Notice: Unconsented voice cloning, deepfake generation, or illegal impersonation is strictly prohibited. Users are fully responsible for ensuring compliance with applicable regional laws.

🤗 Base Model • 💻 GitHub Repository • 🎮 Interactive Space • 📄 Paper