🎙️ Tui Voice Mini (Malayalam) — 4.5M Parameter Edge TTS

GitHub License: Apache 2.0 Parameters Size Audio

Tui Voice Mini Malayalam (tui-voice-mini-ml) is an ultra-compact, production-ready Malayalam Text-to-Speech (TTS) neural model. Fine-tuned from tiny-voca using high-fidelity studio recordings from the AI4Bharat Rasa Malayalam female corpus, it generates crisp, natural 24 kHz speech with just 4,537,291 parameters (~10 MB footprint).

Unlike conventional neural TTS pipelines (e.g., FastPitch + HiFi-GAN at ~110M params or Meta MMS-TTS VITS at ~36M params) requiring dedicated cloud GPUs and large memory buffers, Tui Voice Mini runs 100% locally on-device—in web browsers via WebAssembly, on iOS/Android, and on low-power ARM microcontrollers like Raspberry Pi and ROS robots.


⚡ Key Highlights

  • True On-Device Footprint: Only 4.5M parameters. Requires < 30 MB RAM at inference time.
  • Integrated Source-Filter Synthesizer: Direct harmonic oscillator and filtered noise excitation—requires zero external neural vocoder (no separate 100 MB HiFi-GAN or Vocos needed).
  • Native Runtime Prosody Control: Algebraic --pitch-shift and --pitch-range controls to manipulate voice timbre and pitch contour with zero latency.
  • Full Malayalam G2P Coverage: Explicit 76-slot budget mapping covering all 59 Malayalam phonemes, long/short vowel distinctions, conjunct clusters, and explicit Chillus (ൺ, ൻ, ർ, ൽ, ൾ, ൿ).
  • Production Lineage: Fully trained for 40,000 steps with multi-period and multi-resolution adversarial GAN discriminators.

📦 Released Model Weights & Formats

File Name Format Size Target Deployment
tui-voice-mini-ml.pt PyTorch FP32 18.2 MB Standard Python / PyTorch deployment
tui-voice-mini-ml-fp16.pt PyTorch FP16 9.1 MB High-speed GPU / low-memory Python inference
onnx/tui-voice-mini-ml.onnx ONNX (opset 17) 21.4 MB WebAssembly (Web Browser), Android, iOS, C++
onnx/tui-voice-mini-ml-int8.onnx Quantized INT8 ~5.4 MB Raspberry Pi, ROS Robots, Edge Devices
config.json JSON 2.1 KB Model hyperparameters and 76-token vocabulary
vocab_plan.json JSON 4.2 KB Upstream phoneme slot substitution record
checkpoints/tui-voice-mini-ml-step40k.pt PyTorch Full 87.2 MB Final step-40,000 training state (weights + optimizer)
checkpoints/tui-voice-mini-ml-step*.pt PyTorch Full ~87 MB Checkpoint snapshots at steps 5k, 8k, 10k, 15k, 20k, 25k, 30k, 35k

🎧 Audio Showcase & Real-World Demos

Listen to actual audio clips generated by Tui Voice Mini across 6 distinct linguistic domains:

Category Example Malayalam Input Text Audio Sample
Adheesh Sir Check-in അധീഷ് സാർ ഇന്നു സുഖം ആണോ, എന്തൊക്കെ ഉണ്ട് വിശേഷം? audio_samples/everyday_whatsapp_adheesh_sir.wav
Everyday Greeting ഹലോ, സുഖമാണോ? എവിടെയാണ് ഇപ്പോൾ? audio_samples/everyday_whatsapp_hello_sugamano.wav
Breaking Weather News പ്രധാന വാർത്തകൾ. സംസ്ഥാനത്ത് അടുത്ത അഞ്ച് ദിവസം ശക്തമായ മഴയ്ക്ക് സാധ്യതയുണ്ടെന്ന് കാലാവസ്ഥാ വകുപ്പ് അറിയിച്ചു. audio_samples/news_and_media_news_weather_warning.wav
Kochi Metro Station അടുത്ത സ്റ്റേഷൻ എം ജി റോഡ്. വാതിലുകൾ ഇടത്തു വശത്ത് തുറക്കുന്നതാണ്. audio_samples/public_announcements_metro_station.wav
Railway Platform ശ്രദ്ധിക്കുക, തിരുവനന്തപുരത്തു നിന്നും കോഴിക്കോട്ടേക്കുള്ള ജനശതാബ്ദി എക്സ്പ്രസ്സ് രണ്ടാം നമ്പർ പ്ലാറ്റ്‌ഫോമിൽ എത്തിച്ചേരുന്നു. audio_samples/public_announcements_railway_station.wav
Customer Support (IVR) ഞങ്ങളുടെ കസ്റ്റമർ കെയർ സർവീസിലേക്ക് സ്വാഗതം. മലയാളത്തിൽ തുടരുന്നതിനായി ഒന്നിൽ അമർത്തുക. audio_samples/customer_care_ivr_ivr_welcome.wav
On-Device Edge AI ഇന്റർനെറ്റ് ഇല്ലാതെ തന്നെ ഏത് മൊബൈൽ ഫോണിലും ഈ ശബ്ദ സാങ്കേതികവിദ്യ തത്സമയം പ്രവർത്തിക്കും. audio_samples/ai_tech_showcase_tech_edge_ai.wav

(All audio clips are available in the audio_samples/ directory).


💻 Quickstart & Python Usage

1. Installation

# Clone the official repository
git clone https://github.com/Cyrax321/tui-voice.git
cd tui-voice
pip install -r requirements.txt

# Install eSpeak NG for Malayalam phonemization
# Ubuntu/Debian: sudo apt-get install espeak-ng
# macOS: brew install espeak-ng

2. Synthesize Speech with PyTorch

import torch
import soundfile as sf
from tui_voice import phonemes as P
from tiny_voca.config import Config
from tiny_voca.model import TinyVoca
from huggingface_hub import hf_hub_download
import json

# 1. Download model & config from Hugging Face
model_path = hf_hub_download("Cyrax321/tui-voice-mini-ml", "tui-voice-mini-ml.pt")
config_path = hf_hub_download("Cyrax321/tui-voice-mini-ml", "config.json")
plan_path = hf_hub_download("Cyrax321/tui-voice-mini-ml", "vocab_plan.json")

# 2. Initialize model
cfg = Config.from_dict(json.load(open(config_path)))
model = TinyVoca(cfg)
state = torch.load(model_path, map_location="cpu", weights_only=False)
model.load_state_dict(state.get("model", state), strict=False)
model.eval()

# 3. Phonemize & Encode input text
text = "നമസ്കാരം, ഇത് തുയി വോയ്സ് മിനി മലയാളം മോഡൽ ആണ്."
plan = json.load(open(plan_path))
tokens = P.phonemize(text)
token_ids = P.encode(tokens, plan)

tok_tensor = torch.tensor([token_ids], dtype=torch.long)
stress = torch.zeros_like(tok_tensor)
lengths = torch.tensor([len(token_ids)], dtype=torch.long)
speaker = torch.tensor([0], dtype=torch.long)

# 4. Synthesize 24 kHz audio
with torch.inference_mode():
    audio, _, _ = model.synthesize(
        tok_tensor, stress, lengths, speaker,
        speed=1.0,
        pitch_shift=0.15,  # Locks into natural Malayalam female register
        pitch_range=0.85,  # Smooth, consistent melodic contour
        voiced_vowels=True
    )

sf.write("output.wav", audio.squeeze(0).numpy(), 24000)
print("Saved 24 kHz audio to output.wav!")

3. Lightweight ONNX Runtime Inference (Zero PyTorch Dependency)

import onnxruntime as ort
import numpy as np
import soundfile as sf
from huggingface_hub import hf_hub_download

# Download ONNX model (21.4 MB) or INT8 model (5.4 MB)
onnx_path = hf_hub_download("Cyrax321/tui-voice-mini-ml", "onnx/tui-voice-mini-ml.onnx")
session = ort.InferenceSession(onnx_path)

# Run direct graph evaluation
one = lambda v: np.array([v], dtype=np.float32)
audio, durations = session.run(None, {
    "tokens": np.array([[12, 45, 23, 67, 18]], dtype=np.int64),
    "stress": np.zeros((1, 5), dtype=np.int64),
    "voice": np.array([0], dtype=np.int64),
    "speed": one(1.0),
    "pitch_shift": one(0.15),
    "pitch_range": one(0.85),
})

sf.write("onnx_output.wav", audio[0], 24000)

🔬 Architecture Specifications

Module Component Name Parameters % of Model Function
Embeddings phoneme_embed, stress_embed, voice_embed 16,704 0.37% Maps 76 phonemes + stress + voice into 192-dim vector space
Text Encoder encoder 1,588,864 35.02% 4 Transformer blocks, 192 channels, 4 heads, RoPE attention
Duration Head duration_head, pause_head 123,906 2.73% Predicts frame durations and punctuation pauses
Prosody Predictor frame_transformer_a + f0_head 795,971 17.54% 2 Transformer blocks predicting continuous $F_0$ and voicing
Cepstral Decoder frame_transformer_b 1,985,984 43.77% 5 Transformer blocks predicting acoustic filter coefficients
Source-Filter Head synth_head 25,862 0.57% Differentiable harmonic pulse-train and noise filter
Total Model Complete Inference Pipeline 4,537,291 100.0% End-to-End waveform generation (24 kHz)

📈 Training Convergence Metrics (Step 40,000)

Tui Voice Mini was trained on an NVIDIA L4 GPU with high-RAM for 40,000 steps using AdamW optimizer and cosine learning rate decay:

  • Validation Pause Accuracy: 1.000 (100% exact punctuation pause placement)
  • Validation Duration Length Error: 0.097 (< 9.8% cadence error)
  • Validation Voicing Accuracy: 93.4% (crisp distinction between voiced vowels and unvoiced consonants)
  • Final Pitch Error ($F_0$ RMSE): 3.73 semitones (down from 8.61 at step 10)
  • Discriminator GAN Loss: 0.277 (stable adversarial equilibrium)
  • Teacher Forcing Fraction: 0.0 (fully autonomous, free-running speech synthesis)

📊 Benchmark Comparison vs. Other Malayalam TTS Models

Metric AI4Bharat Indic-TTS Meta MMS-TTS Piper Malayalam Tui Voice Mini (Ours)
Architecture FastPitch + HiFi-GAN VITS (CVAE) VITS-derived Source-Filter + GAN
Parameters ~110 Million ~36 Million ~15.7 Million 4.5 Million (8x – 25x Smaller!)
Disk Size ~350 MB – 450 MB ~145 MB ~60 MB 18.2 MB (FP32) / 5.4 MB (INT8)
RAM Footprint ~1.2 GB+ ~450 MB ~200 MB < 30 MB (Ultra-Lightweight)
CPU RTF (Latency) 0.80 – 1.20 0.30 – 0.45 0.25 – 0.35 0.08 – 0.12 (10x Real-Time)
Browser (WASM) Offline ❌ No ❌ No ⚠️ Heavy ✅ YES (Runs in browser)
Raspberry Pi / Robots ❌ No ⚠️ Slow ⚠️ Moderate ✅ YES (Optimized INT8)
Runtime Pitch Control ❌ No ❌ No ❌ No ✅ Full algebraic control

📜 Research Benchmarking Suite

The complete benchmarking suite and data splits are included in the repository:

  • benchmark/data/splits/dev_set.jsonl: 100 sentences for pipeline validation.
  • benchmark/data/splits/main_test_set.jsonl: 500 sentences balanced across Conversational (100), News/Formal (100), Long Compounds (75), Questions (75), Numbers/Currency (75), and Code-Switching (75).
  • benchmark/data/splits/stress_test_set.jsonl: 200 sentences testing rare conjuncts and tongue twisters.
  • Acoustic & Integrity Metrics: Grapheme CER (INT-01), WER (INT-02), SER (INT-03), Audio Integrity (AUD-01 to AUD-07), F0 RMSE (PRO-01), and Efficiency (EFF-01 to EFF-11).

To reproduce the benchmark:

python3 benchmark/scripts/prepare_dataset.py

✍️ Citation & Attribution

@software{tuivoice2026malayalam,
  author = {Cyrax321},
  title = {Tui Voice Mini: Ultra-Compact 4.5M Parameter Malayalam Text-to-Speech},
  year = {2026},
  publisher = {Hugging Face / GitHub},
  url = {https://huggingface.co/Cyrax321/tui-voice-mini-ml}
}
Downloads last month
-
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Dataset used to train Cyrax321/tui-voice-mini-ml