--- license: cc-by-nc-4.0 language: - ne - en pipeline_tag: text-to-speech library_name: pytorch tags: - text-to-speech - tts - nepali - styletts2 - kokoro - customer-service --- # Swar **Swar** (स्वर, "voice") is a small, experimental text-to-speech model for **Nepali and English**, including sentences that mix the two, tuned toward a calm customer-service manner of speaking. - **38.5 M parameters**, 16 kHz output. The architecture is derived from [Kokoro-82M](https://huggingface.co/hexgrad/Kokoro-82M) (StyleTTS2 family), built from its layer classes: non-autoregressive, so a sentence is generated in one pass. - Reads **Devanagari and Latin letters directly**: no phonemiser, no language switch. Numbers, currency ("Rs. 250", "रु ५००"), times and phone numbers are written out in words in the language of their sentence. - **Default voice: `bijaya`** (`voices/bijaya.pt`). Any 3-10 s clip of one speaker gives another voice. - Fast: about **15x real time on a desktop CPU** with all cores, about 6x on a single thread, and about 30x on a consumer GPU (measured on a Ryzen 9 5900X and an RTX 3090). Try it in the Space: [Nepali Call Center Experimental - Powered by Swar](https://huggingface.co/spaces/elitexp/Nepali-Swar-Call-Center-Experimental). ## Quick start ```bash pip install torch torchaudio transformers safetensors huggingface_hub soundfile ``` ```python import sys, soundfile as sf from huggingface_hub import snapshot_download folder = snapshot_download("elitexp/Swar") sys.path.insert(0, folder) from swar import Swar tts = Swar.from_pretrained(folder) # GPU if available, else CPU wav, sr = tts.synthesize("नमस्ते, ग्राहक सेवा केन्द्रमा यहाँलाई स्वागत छ।") # voice "bijaya" sf.write("hello.wav", wav, sr) # another voice from a reference clip, a little slower, with clear pauses at punctuation wav, sr = tts.synthesize("Your balance is Rs. 250. धन्यवाद।", voice="my_reference.wav", speed=0.95, pause_floor_ms=300) ``` From the command line, inside the downloaded folder: ```bash python -m swar.inference "नमस्ते, यहाँलाई स्वागत छ।" -o hello.wav ``` ### Options | argument | meaning | |---|---| | `voice` | a name from `voices/` (default `bijaya`), a `.pt` of style vectors, or an audio clip (wav / flac / mp3 / ogg) | | `speed` | 0.5-2.0, scales the predicted durations (1.0 = as predicted) | | `pause_floor_ms` | every comma / full stop / question mark *inside* a sentence lasts at least this long (0 = as predicted) | | `loanwords` | in a mostly-Nepali sentence, common English words are said the Nepali way ("recharge" -> रिचार्ज, "balance" -> ब्यालेन्स; list in `swar/loanwords.py`). Default on | | `sentence_pause_ms` | silence between two sentences (default 350); the start and end of each sentence are trimmed, so the first sound comes within about 0.1 s | | `return_info` | also return timing and the exact letters the model read | ## What is inside | part | params | role | |---|---|---| | PL-BERT (ALBERT, letters) + projection | 3.5 M | contextual letter features | | prosody predictor | 9.7 M | duration per letter, pitch and energy at 80 Hz | | prosodic style encoder | 3.5 M | manner of speaking from a reference clip | | text encoder | 3.2 M | letter features for the decoder | | decoder (iSTFTNet with harmonic source) | 15.1 M | waveform at 16 kHz | | acoustic style encoder | 3.5 M | timbre from a reference clip | A voice is two 128-d vectors: acoustic (timbre, read by the decoder) and prosodic (manner, read by the predictor). `voices/bijaya.pt` stores them as `{"s_ac", "s_pr"}`. Files: `swar.safetensors` (inference weights only), `config.json`, `swar/` (model, text front-end, mel, inference; Kokoro's layer code is vendored under `swar/kokoro/` with its Apache-2.0 licence), `voices/`. ## Training Trained from scratch in five stages, after the StyleTTS2 / Kokoro recipe adapted to letters at 16 kHz: 1. **PL-BERT** on Nepali and English text (masked letters). 2. **Letter aligner** (CTC) on both languages, for durations. 3. **Acoustic model**: text encoder, decoder and both style encoders, adversarial training with multi-period and multi-resolution discriminators (300 k steps). 4. **Prosody**: duration, pitch and energy predictors and the prosodic style encoder (100 k steps). 5. **Register**: a short fine-tune (6 k steps) of the prosody half on about 8 hours of customer-service style Nepali recordings, with general data replayed so the rest of the model does not drift. Data (acoustic and prosody stages): about 800 hours of Nepali speech from public corpora (including OpenSLR 43 and OpenSLR 143) and other licensed and in-house recordings, part of it synthetic, and about 570 hours of English speech synthesised with Kokoro-82M (`af_heart`). ## Evaluation - **Copy-synthesis intelligibility** (stage 3, durations and pitch taken from the real clip, recognised by the same ASR as the real recordings): Nepali character error rate through a G.711 telephone channel 0.199 against 0.220 for the real recordings (285 clips); English word error rate 0.097 against 0.098 (200 clips). - **Customer-service register** (stage 5): median pitch stays within 15 Hz of the reference voice. The pause rate (about 0.13 pauses per second, about 0.20 with `pause_floor_ms=300`) is still below the 0.30-0.45 target band: pauses are only lengthened where the text has punctuation. - Full text-to-speech intelligibility has not yet been measured with the same protocol. ## Limitations - Experimental. Rare words, names and some English spellings can be mispronounced; Nepali rhythm is less polished than English. - 16 kHz, telephone-oriented; not a studio-quality voice. - Voices from a reference clip copy that clip's recording conditions too; use clean audio. ## Responsible use Swar can speak in a voice taken from a short recording. Only use voices you have permission to use, label synthetic speech as synthetic where people could mistake it for a person, and do not use the model to impersonate anyone or to deceive. ## Acknowledgements Swar would not exist without **[Kokoro-82M](https://huggingface.co/hexgrad/Kokoro-82M)** by hexgrad. Its architecture is Swar's blueprint: the text encoder, the prosody predictor (duration, pitch and energy), the iSTFTNet decoder with its harmonic source and the split acoustic / prosodic style design all come from Kokoro, and Kokoro's own layer code is vendored in `swar/kokoro/`. Swar re-sizes that design for 16 kHz, reads letters instead of phonemes, adds trainable style encoders and is trained from scratch; Kokoro (voice `af_heart`) also generated Swar's English training speech. Kokoro in turn builds on [StyleTTS2](https://github.com/yl4579/StyleTTS2) (Li et al.), whose style encoder and PL-BERT recipe Swar also uses. ## License The Swar weights (`swar.safetensors`, `voices/`) and Swar's own code are released under **[CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/)**: free for research and other non-commercial use, with attribution. The vendored Kokoro code in `swar/kokoro/` stays under its original Apache-2.0 licence (`swar/kokoro/LICENSE`); the StyleTTS2 style encoder code is MIT. Training data includes [OpenSLR](https://www.openslr.org/) corpora under their own licences.