Update README.md

c1b0bf9 verified 23 days ago

4.87 kB

license: cc-by-nc-4.0
language:
  - kk
tags:
  - kazakh
  - text-to-speech
  - voice-design
  - non-verbal-vocalization
  - prompt-controllable-tts
  - lora
base_model: k2-fsa/OmniVoice
datasets:
  - issai/KazParaD

KazGenericTTS: Prompt-Controllable Kazakh TTS with Non-Verbal Vocalizations

KazGenericTTS is a Kazakh text-to-speech model with attribute control (gender, age, pitch, emotion via a comma-separated instruction) and non-verbal vocalization (NVV) synthesis via 18 inline tags ([laugh], [sigh], etc.). It is a fine-tune of the OmniVoice foundation model on the KazParaD corpus.

Base model and architecture

Backbone: OmniVoice — a Qwen3-0.6B language model coupled to the higgs-audio-v2 neural audio codec (24 kHz).

Training data

Fine-tuned jointly on the demographic and NVV subsets of issai/KazParaD. Age was conditioned from the source free-text speaker descriptions (5-way: child / teenager / young adult / middle-aged / elderly), which differ from the dataset's 4-way age_category column (child / young / adult / senior).

Adaptation regimes and hyperparameters

5,000 steps, AdamW (weight decay 0.01, grad-clip 1.0), cosine schedule, 5% warmup; bf16 mixed precision, FlexAttention, token-packed batching (8,192 tokens × 4 gradient-accumulation steps × 1 GPU = 32,768 tokens/step), seed 42, hardware 1 NVIDIA H100. Step budget (5,000) is the OmniVoice recipe default; validation control metrics plateau well before it.

	Full fine-tuning (KazGenericTTS)	LoRA variant
Trainable params	612,577,280 (100%)	26,886,144 (≈4.4% of full FT)
Learning rate	1e-5	1e-4
Fully-trained modules	all	audio_embeddings, audio_heads
LoRA rank `r`	—	16
LoRA `alpha`	—	32
LoRA dropout	—	0.05
LoRA bias	—	none
LoRA target modules	—	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj

The primary released adapted model is the full fine-tune.

Prompt format

Attributes are a comma-separated instruction from a controlled vocabulary; NVV tags go inline in the text. The model consumes:

<|lang_start|>Kazakh<|lang_end|><|instruct_start|>{instruct}<|instruct_end|><|text_start|>{text}<|text_end|>

Real examples:

instruct: "female, young adult, high pitch, happy"
text:     "Мұны естіп [laugh] қатты күлдім."

Controlled vocabulary: gender {male, female}; age {child, teenager, young adult, middle-aged, elderly}; pitch {very low, low, moderate, high, very high} pitch; emotion {angry, disgusted, fearful, happy, neutral, sad, surprised}. NVV tags (18): [laugh] [chuckle] [sigh] [gasp] [groan] [cough] [sniffle] [yawn] [whispers] [singing] [stutters] [burps] [hiccups] [chewing] [smacks lips] [swallows] [tsking] [long pause].

Note: the model's age vocabulary is the 5-way scheme above (child / teenager / young adult / middle-aged / elderly), because it was trained on regex-parsed instructions. This differs from the published dataset's 4-way age_category (child / young / adult / senior) — prompt the model with the 5-way tokens.

Inference

import torch, soundfile as sf
from omnivoice import OmniVoice

model = OmniVoice.from_pretrained("issai/KazGenericTTS", device_map="cuda", dtype=torch.float16)
audio = model.generate(
    text="Мұны естіп [laugh] қатты күлдім.",
    language="Kazakh",
    instruct="female, young adult, high pitch, happy",
)[0]
sf.write("out.wav", audio, model.sampling_rate)

Evaluation

Headline results (full ablations and the NVV moderator study are in the paper).

	CER↓	Gender↑	Pitch ρ (F/M)↑	NVV AUC↑
Base (OmniVoice, zero-shot)	4.1	0.88	0.92	0.55
KazGenericTTS	2.4	1.00	0.82 / 0.94	0.86

Naturalness MOS 3.04 ± 0.37 (30 native listeners; real-recording topline 4.51 ± 0.21).
Fine-tuning's headline gain is NVV synthesis (AUC 0.55 → 0.86); intelligibility, gender and pitch control are on par with or above the base.
A LoRA variant (4.4% of parameters) matches full fine-tuning on all axes except mid-range male pitch.

License

Released under CC BY-NC 4.0. The model is fine-tuned from k2-fsa/OmniVoice and trained on the KazParaD corpus (which includes a synthetic NVV subset and web-sourced audio).

Citation

Paper is under review. A citation will be added here once it is published. Until then, please link to this repository.