| --- |
| license: cc-by-nc-4.0 |
| language: |
| - kk |
| tags: |
| - kazakh |
| - text-to-speech |
| - voice-design |
| - non-verbal-vocalization |
| - prompt-controllable-tts |
| - lora |
| base_model: k2-fsa/OmniVoice |
| datasets: |
| - issai/KazParaD |
| --- |
| |
| # KazGenericTTS: Prompt-Controllable Kazakh TTS with Non-Verbal Vocalizations |
|
|
| [](https://is2ai.github.io/kazparad/) |
| [](https://issai-kazparad-gradio-demo.hf.space/) |
| [](https://huggingface.co/datasets/issai/KazParaD/) |
|
|
| KazGenericTTS is a Kazakh text-to-speech model with attribute control (gender, age, pitch, emotion via a comma-separated instruction) and non-verbal vocalization (NVV) synthesis via 18 inline tags (`[laugh]`, `[sigh]`, etc.). It is a fine-tune of the OmniVoice foundation model on the KazParaD corpus. |
|
|
| ## Base model and architecture |
|
|
| Backbone: OmniVoice — a Qwen3-0.6B language model coupled to the higgs-audio-v2 neural audio codec (24 kHz). |
|
|
| ## Training data |
|
|
| Fine-tuned jointly on the demographic and NVV subsets of [issai/KazParaD](https://huggingface.co/datasets/issai/KazParaD). Age was conditioned from the source free-text speaker descriptions (5-way: child / teenager / young adult / middle-aged / elderly), which differ from the dataset's 4-way `age_category` column (child / young / adult / senior). |
|
|
| ## Adaptation regimes and hyperparameters |
|
|
| 5,000 steps, AdamW (weight decay 0.01, grad-clip 1.0), cosine schedule, 5% warmup; bf16 mixed precision, FlexAttention, token-packed batching (8,192 tokens × 4 gradient-accumulation steps × 1 GPU = 32,768 tokens/step), seed 42, hardware 1 NVIDIA H100. Step budget (5,000) is the OmniVoice recipe default; validation control metrics plateau well before it. |
|
|
| | | Full fine-tuning (KazGenericTTS) | LoRA variant | |
| |--|--|--| |
| | Trainable params | 612,577,280 (100%) | 26,886,144 (≈4.4% of full FT) | |
| | Learning rate | 1e-5 | 1e-4 | |
| | Fully-trained modules | all | audio_embeddings, audio_heads | |
| | LoRA rank `r` | — | 16 | |
| | LoRA `alpha` | — | 32 | |
| | LoRA dropout | — | 0.05 | |
| | LoRA bias | — | none | |
| | LoRA target modules | — | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj | |
| |
| The primary released adapted model is the full fine-tune. |
| |
| ## Prompt format |
| |
| Attributes are a comma-separated instruction from a controlled vocabulary; NVV tags go inline in the text. The model consumes: |
| |
| ``` |
| <|lang_start|>Kazakh<|lang_end|><|instruct_start|>{instruct}<|instruct_end|><|text_start|>{text}<|text_end|> |
| ``` |
| |
| Real examples: |
| |
| ``` |
| instruct: "female, young adult, high pitch, happy" |
| text: "Мұны естіп [laugh] қатты күлдім." |
| ``` |
| |
| Controlled vocabulary: gender {male, female}; age {child, teenager, young adult, middle-aged, elderly}; pitch {very low, low, moderate, high, very high} pitch; emotion {angry, disgusted, fearful, happy, neutral, sad, surprised}. NVV tags (18): `[laugh] [chuckle] [sigh] [gasp] [groan] [cough] [sniffle] [yawn] [whispers] [singing] [stutters] [burps] [hiccups] [chewing] [smacks lips] [swallows] [tsking] [long pause]`. |
| |
| > Note: the model's age vocabulary is the 5-way scheme above (child / teenager / young adult / middle-aged / elderly), because it was trained on regex-parsed instructions. This differs from the published dataset's 4-way `age_category` (child / young / adult / senior) — prompt the model with the 5-way tokens. |
|
|
| ## Inference |
|
|
| ```python |
| import torch, soundfile as sf |
| from omnivoice import OmniVoice |
| |
| model = OmniVoice.from_pretrained("issai/KazGenericTTS", device_map="cuda", dtype=torch.float16) |
| audio = model.generate( |
| text="Мұны естіп [laugh] қатты күлдім.", |
| language="Kazakh", |
| instruct="female, young adult, high pitch, happy", |
| )[0] |
| sf.write("out.wav", audio, model.sampling_rate) |
| ``` |
| ## Evaluation |
|
|
| Headline results (full ablations and the NVV moderator study are in the paper). |
|
|
| | | CER↓ | Gender↑ | Pitch ρ (F/M)↑ | NVV AUC↑ | |
| |--|--:|--:|--:|--:| |
| | Base (OmniVoice, zero-shot) | 4.1 | 0.88 | 0.92 | 0.55 | |
| | **KazGenericTTS** | 2.4 | 1.00 | 0.82 / 0.94 | 0.86 | |
|
|
| - Naturalness MOS 3.04 ± 0.37 (30 native listeners; real-recording topline 4.51 ± 0.21). |
| - Fine-tuning's headline gain is NVV synthesis (AUC 0.55 → 0.86); intelligibility, gender and pitch control are on par with or above the base. |
| - A LoRA variant (4.4% of parameters) matches full fine-tuning on all axes except mid-range male pitch. |
|
|
| ## License |
|
|
| Released under CC BY-NC 4.0. The model is fine-tuned from `k2-fsa/OmniVoice` and trained on the KazParaD corpus (which includes a synthetic NVV subset and web-sourced audio). |
|
|
| ## Citation |
|
|
| Paper is under review. A citation will be added here once it is published. Until then, please link to this repository. |
