| --- |
| license: cc-by-nc-4.0 |
| language: |
| - ne |
| - en |
| pipeline_tag: text-to-speech |
| library_name: pytorch |
| tags: |
| - text-to-speech |
| - tts |
| - nepali |
| - styletts2 |
| - kokoro |
| - customer-service |
| --- |
| |
| # Swar |
|
|
| **Swar** (स्वर, "voice") is a small, experimental text-to-speech model for **Nepali and |
| English**, including sentences that mix the two, tuned toward a calm customer-service manner |
| of speaking. |
|
|
| - **38.5 M parameters**, 16 kHz output. The architecture is derived from |
| [Kokoro-82M](https://huggingface.co/hexgrad/Kokoro-82M) (StyleTTS2 family), built from its layer |
| classes: non-autoregressive, so a sentence is generated in one pass. |
| - Reads **Devanagari and Latin letters directly**: no phonemiser, no language switch. |
| Numbers, currency ("Rs. 250", "रु ५००"), times and phone numbers are written out in words in |
| the language of their sentence. |
| - **Default voice: `bijaya`** (`voices/bijaya.pt`). Any 3-10 s clip of one speaker gives |
| another voice. |
| - Fast: about **15x real time on a desktop CPU** with all cores, about 6x on a single thread, |
| and about 30x on a consumer GPU (measured on a Ryzen 9 5900X and an RTX 3090). |
|
|
| Try it in the Space: [Nepali Call Center Experimental - Powered by Swar](https://huggingface.co/spaces/elitexp/Nepali-Swar-Call-Center-Experimental). |
|
|
| ## Quick start |
|
|
| ```bash |
| pip install torch torchaudio transformers safetensors huggingface_hub soundfile |
| ``` |
|
|
| ```python |
| import sys, soundfile as sf |
| from huggingface_hub import snapshot_download |
| |
| folder = snapshot_download("elitexp/Swar") |
| sys.path.insert(0, folder) |
| from swar import Swar |
| |
| tts = Swar.from_pretrained(folder) # GPU if available, else CPU |
| wav, sr = tts.synthesize("नमस्ते, ग्राहक सेवा केन्द्रमा यहाँलाई स्वागत छ।") # voice "bijaya" |
| sf.write("hello.wav", wav, sr) |
| |
| # another voice from a reference clip, a little slower, with clear pauses at punctuation |
| wav, sr = tts.synthesize("Your balance is Rs. 250. धन्यवाद।", |
| voice="my_reference.wav", speed=0.95, pause_floor_ms=300) |
| ``` |
|
|
| From the command line, inside the downloaded folder: |
|
|
| ```bash |
| python -m swar.inference "नमस्ते, यहाँलाई स्वागत छ।" -o hello.wav |
| ``` |
|
|
| ### Options |
|
|
| | argument | meaning | |
| |---|---| |
| | `voice` | a name from `voices/` (default `bijaya`), a `.pt` of style vectors, or an audio clip (wav / flac / mp3 / ogg) | |
| | `speed` | 0.5-2.0, scales the predicted durations (1.0 = as predicted) | |
| | `pause_floor_ms` | every comma / full stop / question mark *inside* a sentence lasts at least this long (0 = as predicted) | |
| | `loanwords` | in a mostly-Nepali sentence, common English words are said the Nepali way ("recharge" -> रिचार्ज, "balance" -> ब्यालेन्स; list in `swar/loanwords.py`). Default on | |
| | `sentence_pause_ms` | silence between two sentences (default 350); the start and end of each sentence are trimmed, so the first sound comes within about 0.1 s | |
| | `return_info` | also return timing and the exact letters the model read | |
|
|
| ## What is inside |
|
|
| | part | params | role | |
| |---|---|---| |
| | PL-BERT (ALBERT, letters) + projection | 3.5 M | contextual letter features | |
| | prosody predictor | 9.7 M | duration per letter, pitch and energy at 80 Hz | |
| | prosodic style encoder | 3.5 M | manner of speaking from a reference clip | |
| | text encoder | 3.2 M | letter features for the decoder | |
| | decoder (iSTFTNet with harmonic source) | 15.1 M | waveform at 16 kHz | |
| | acoustic style encoder | 3.5 M | timbre from a reference clip | |
|
|
| A voice is two 128-d vectors: acoustic (timbre, read by the decoder) and prosodic (manner, |
| read by the predictor). `voices/bijaya.pt` stores them as `{"s_ac", "s_pr"}`. |
|
|
| Files: `swar.safetensors` (inference weights only), `config.json`, `swar/` (model, |
| text front-end, mel, inference; Kokoro's layer code is vendored under `swar/kokoro/` with its |
| Apache-2.0 licence), `voices/`. |
|
|
| ## Training |
|
|
| Trained from scratch in five stages, after the StyleTTS2 / Kokoro recipe adapted to letters at |
| 16 kHz: |
|
|
| 1. **PL-BERT** on Nepali and English text (masked letters). |
| 2. **Letter aligner** (CTC) on both languages, for durations. |
| 3. **Acoustic model**: text encoder, decoder and both style encoders, adversarial training with |
| multi-period and multi-resolution discriminators (300 k steps). |
| 4. **Prosody**: duration, pitch and energy predictors and the prosodic style encoder (100 k steps). |
| 5. **Register**: a short fine-tune (6 k steps) of the prosody half on about 8 hours of |
| customer-service style Nepali recordings, with general data replayed so the rest of the model |
| does not drift. |
|
|
| Data (acoustic and prosody stages): about 800 hours of Nepali speech from public corpora |
| (including OpenSLR 43 and OpenSLR 143) and other licensed and in-house recordings, part of it |
| synthetic, and about 570 hours of English speech synthesised with Kokoro-82M (`af_heart`). |
|
|
| ## Evaluation |
|
|
| - **Copy-synthesis intelligibility** (stage 3, durations and pitch taken from the real clip, |
| recognised by the same ASR as the real recordings): Nepali character error rate through a |
| G.711 telephone channel 0.199 against 0.220 for the real recordings (285 clips); English word |
| error rate 0.097 against 0.098 (200 clips). |
| - **Customer-service register** (stage 5): median pitch stays within 15 Hz of the reference |
| voice. The pause rate (about 0.13 pauses per second, about 0.20 with `pause_floor_ms=300`) |
| is still below the 0.30-0.45 target band: pauses are only lengthened where the text has |
| punctuation. |
| - Full text-to-speech intelligibility has not yet been measured with the same protocol. |
|
|
| ## Limitations |
|
|
| - Experimental. Rare words, names and some English spellings can be mispronounced; Nepali |
| rhythm is less polished than English. |
| - 16 kHz, telephone-oriented; not a studio-quality voice. |
| - Voices from a reference clip copy that clip's recording conditions too; use clean audio. |
|
|
| ## Responsible use |
|
|
| Swar can speak in a voice taken from a short recording. Only use voices you have permission to |
| use, label synthetic speech as synthetic where people could mistake it for a person, and do not |
| use the model to impersonate anyone or to deceive. |
|
|
| ## Acknowledgements |
|
|
| Swar would not exist without **[Kokoro-82M](https://huggingface.co/hexgrad/Kokoro-82M)** by |
| hexgrad. Its architecture is Swar's blueprint: the text encoder, the prosody predictor (duration, |
| pitch and energy), the iSTFTNet decoder with its harmonic source and the split acoustic / prosodic |
| style design all come from Kokoro, and Kokoro's own layer code is vendored in `swar/kokoro/`. Swar |
| re-sizes that design for 16 kHz, reads letters instead of phonemes, adds trainable style encoders |
| and is trained from scratch; Kokoro (voice `af_heart`) also generated Swar's English training |
| speech. Kokoro in turn builds on [StyleTTS2](https://github.com/yl4579/StyleTTS2) (Li et al.), whose |
| style encoder and PL-BERT recipe Swar also uses. |
|
|
| ## License |
|
|
| The Swar weights (`swar.safetensors`, `voices/`) and Swar's own code are released under |
| **[CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/)**: free for research and other |
| non-commercial use, with attribution. The vendored Kokoro code in `swar/kokoro/` stays under its |
| original Apache-2.0 licence (`swar/kokoro/LICENSE`); the StyleTTS2 style encoder code is MIT. |
| Training data includes [OpenSLR](https://www.openslr.org/) corpora under their own licences. |
|
|
