README.md

metadata

license: cc-by-nc-4.0
language:
  - ne
  - en
pipeline_tag: text-to-speech
library_name: pytorch
tags:
  - text-to-speech
  - tts
  - nepali
  - styletts2
  - kokoro
  - customer-service

Swar

Swar (स्वर, "voice") is a small, experimental text-to-speech model for Nepali and English, including sentences that mix the two, tuned toward a calm customer-service manner of speaking.

• 38.5 M parameters, 16 kHz output. The architecture is derived from Kokoro-82M (StyleTTS2 family), built from its layer classes: non-autoregressive, so a sentence is generated in one pass.
• Reads Devanagari and Latin letters directly: no phonemiser, no language switch. Numbers, currency ("Rs. 250", "रु ५००"), times and phone numbers are written out in words in the language of their sentence.
• Default voice: bijaya (voices/bijaya.pt). Any 3-10 s clip of one speaker gives another voice.
• Fast: about 15x real time on a desktop CPU with all cores, about 6x on a single thread, and about 30x on a consumer GPU (measured on a Ryzen 9 5900X and an RTX 3090).

Try it in the Space: Nepali Call Center Experimental - Powered by Swar.

Quick start

pip install torch torchaudio transformers safetensors huggingface_hub soundfile

import sys, soundfile as sf
from huggingface_hub import snapshot_download

folder = snapshot_download("elitexp/Swar")
sys.path.insert(0, folder)
from swar import Swar

tts = Swar.from_pretrained(folder)                       # GPU if available, else CPU
wav, sr = tts.synthesize("नमस्ते, ग्राहक सेवा केन्द्रमा यहाँलाई स्वागत छ।")   # voice "bijaya"
sf.write("hello.wav", wav, sr)

# another voice from a reference clip, a little slower, with clear pauses at punctuation
wav, sr = tts.synthesize("Your balance is Rs. 250. धन्यवाद।",
                         voice="my_reference.wav", speed=0.95, pause_floor_ms=300)

From the command line, inside the downloaded folder:

python -m swar.inference "नमस्ते, यहाँलाई स्वागत छ।" -o hello.wav

Options

argument	meaning
voice	a name from voices/ (default bijaya), a .pt of style vectors, or an audio clip (wav / flac / mp3 / ogg)
speed	0.5-2.0, scales the predicted durations (1.0 = as predicted)
pause_floor_ms	every comma / full stop / question mark inside a sentence lasts at least this long (0 = as predicted)
loanwords	in a mostly-Nepali sentence, common English words are said the Nepali way ("recharge" -> रिचार्ज, "balance" -> ब्यालेन्स; list in swar/loanwords.py). Default on
sentence_pause_ms	silence between two sentences (default 350); the start and end of each sentence are trimmed, so the first sound comes within about 0.1 s
return_info	also return timing and the exact letters the model read

What is inside

part	params	role
PL-BERT (ALBERT, letters) + projection	3.5 M	contextual letter features
prosody predictor	9.7 M	duration per letter, pitch and energy at 80 Hz
prosodic style encoder	3.5 M	manner of speaking from a reference clip
text encoder	3.2 M	letter features for the decoder
decoder (iSTFTNet with harmonic source)	15.1 M	waveform at 16 kHz
acoustic style encoder	3.5 M	timbre from a reference clip

A voice is two 128-d vectors: acoustic (timbre, read by the decoder) and prosodic (manner, read by the predictor). voices/bijaya.pt stores them as {"s_ac", "s_pr"}.

Files: swar.safetensors (inference weights only), config.json, swar/ (model, text front-end, mel, inference; Kokoro's layer code is vendored under swar/kokoro/ with its Apache-2.0 licence), voices/.

Training

Trained from scratch in five stages, after the StyleTTS2 / Kokoro recipe adapted to letters at 16 kHz:

1. PL-BERT on Nepali and English text (masked letters).
2. Letter aligner (CTC) on both languages, for durations.
3. Acoustic model: text encoder, decoder and both style encoders, adversarial training with multi-period and multi-resolution discriminators (300 k steps).
4. Prosody: duration, pitch and energy predictors and the prosodic style encoder (100 k steps).
5. Register: a short fine-tune (6 k steps) of the prosody half on about 8 hours of customer-service style Nepali recordings, with general data replayed so the rest of the model does not drift.

Data (acoustic and prosody stages): about 800 hours of Nepali speech from public corpora (including OpenSLR 43 and OpenSLR 143) and other licensed and in-house recordings, part of it synthetic, and about 570 hours of English speech synthesised with Kokoro-82M (af_heart).

Evaluation

• Copy-synthesis intelligibility (stage 3, durations and pitch taken from the real clip, recognised by the same ASR as the real recordings): Nepali character error rate through a G.711 telephone channel 0.199 against 0.220 for the real recordings (285 clips); English word error rate 0.097 against 0.098 (200 clips).
• Customer-service register (stage 5): median pitch stays within 15 Hz of the reference voice. The pause rate (about 0.13 pauses per second, about 0.20 with pause_floor_ms=300) is still below the 0.30-0.45 target band: pauses are only lengthened where the text has punctuation.
• Full text-to-speech intelligibility has not yet been measured with the same protocol.

Limitations

• Experimental. Rare words, names and some English spellings can be mispronounced; Nepali rhythm is less polished than English.
• 16 kHz, telephone-oriented; not a studio-quality voice.
• Voices from a reference clip copy that clip's recording conditions too; use clean audio.

Responsible use

Swar can speak in a voice taken from a short recording. Only use voices you have permission to use, label synthetic speech as synthetic where people could mistake it for a person, and do not use the model to impersonate anyone or to deceive.

Acknowledgements

Swar would not exist without Kokoro-82M by hexgrad. Its architecture is Swar's blueprint: the text encoder, the prosody predictor (duration, pitch and energy), the iSTFTNet decoder with its harmonic source and the split acoustic / prosodic style design all come from Kokoro, and Kokoro's own layer code is vendored in swar/kokoro/. Swar re-sizes that design for 16 kHz, reads letters instead of phonemes, adds trainable style encoders and is trained from scratch; Kokoro (voice af_heart) also generated Swar's English training speech. Kokoro in turn builds on StyleTTS2 (Li et al.), whose style encoder and PL-BERT recipe Swar also uses.

License

The Swar weights (swar.safetensors, voices/) and Swar's own code are released under CC BY-NC 4.0: free for research and other non-commercial use, with attribution. The vendored Kokoro code in swar/kokoro/ stays under its original Apache-2.0 licence (swar/kokoro/LICENSE); the StyleTTS2 style encoder code is MIT. Training data includes OpenSLR corpora under their own licences.