README.md

---
license: cc-by-nc-4.0
language:
  - ne
  - en
pipeline_tag: text-to-speech
library_name: pytorch
tags:
  - text-to-speech
  - tts
  - nepali
  - styletts2
  - kokoro
  - customer-service
---

# Swar

**Swar** (स्वर, "voice") is a small, experimental text-to-speech model for **Nepali and
English**, including sentences that mix the two, tuned toward a calm customer-service manner
of speaking.

- **38.5 M parameters**, 16 kHz output. The architecture is derived from
  [Kokoro-82M](https://huggingface.co/hexgrad/Kokoro-82M) (StyleTTS2 family), built from its layer
  classes: non-autoregressive, so a sentence is generated in one pass.
- Reads **Devanagari and Latin letters directly**: no phonemiser, no language switch.
  Numbers, currency ("Rs. 250", "रु ५००"), times and phone numbers are written out in words in
  the language of their sentence.
- **Default voice: `bijaya`** (`voices/bijaya.pt`). Any 3-10 s clip of one speaker gives
  another voice.
- Fast: about **15x real time on a desktop CPU** with all cores, about 6x on a single thread,
  and about 30x on a consumer GPU (measured on a Ryzen 9 5900X and an RTX 3090).

Try it in the Space: [Nepali Call Center Experimental - Powered by Swar](https://huggingface.co/spaces/elitexp/Nepali-Swar-Call-Center-Experimental).

## Quick start

```bash
pip install torch torchaudio transformers safetensors huggingface_hub soundfile
```

```python
import sys, soundfile as sf
from huggingface_hub import snapshot_download

folder = snapshot_download("elitexp/Swar")
sys.path.insert(0, folder)
from swar import Swar

tts = Swar.from_pretrained(folder)                       # GPU if available, else CPU
wav, sr = tts.synthesize("नमस्ते, ग्राहक सेवा केन्द्रमा यहाँलाई स्वागत छ।")   # voice "bijaya"
sf.write("hello.wav", wav, sr)

# another voice from a reference clip, a little slower, with clear pauses at punctuation
wav, sr = tts.synthesize("Your balance is Rs. 250. धन्यवाद।",
                         voice="my_reference.wav", speed=0.95, pause_floor_ms=300)
```

From the command line, inside the downloaded folder:

```bash
python -m swar.inference "नमस्ते, यहाँलाई स्वागत छ।" -o hello.wav
```

### Options

| argument | meaning |
|---|---|
| `voice` | a name from `voices/` (default `bijaya`), a `.pt` of style vectors, or an audio clip (wav / flac / mp3 / ogg) |
| `speed` | 0.5-2.0, scales the predicted durations (1.0 = as predicted) |
| `pause_floor_ms` | every comma / full stop / question mark *inside* a sentence lasts at least this long (0 = as predicted) |
| `loanwords` | in a mostly-Nepali sentence, common English words are said the Nepali way ("recharge" -> रिचार्ज, "balance" -> ब्यालेन्स; list in `swar/loanwords.py`). Default on |
| `sentence_pause_ms` | silence between two sentences (default 350); the start and end of each sentence are trimmed, so the first sound comes within about 0.1 s |
| `return_info` | also return timing and the exact letters the model read |

## What is inside

| part | params | role |
|---|---|---|
| PL-BERT (ALBERT, letters) + projection | 3.5 M | contextual letter features |
| prosody predictor | 9.7 M | duration per letter, pitch and energy at 80 Hz |
| prosodic style encoder | 3.5 M | manner of speaking from a reference clip |
| text encoder | 3.2 M | letter features for the decoder |
| decoder (iSTFTNet with harmonic source) | 15.1 M | waveform at 16 kHz |
| acoustic style encoder | 3.5 M | timbre from a reference clip |

A voice is two 128-d vectors: acoustic (timbre, read by the decoder) and prosodic (manner,
read by the predictor). `voices/bijaya.pt` stores them as `{"s_ac", "s_pr"}`.

Files: `swar.safetensors` (inference weights only), `config.json`, `swar/` (model,
text front-end, mel, inference; Kokoro's layer code is vendored under `swar/kokoro/` with its
Apache-2.0 licence), `voices/`.

## Training

Trained from scratch in five stages, after the StyleTTS2 / Kokoro recipe adapted to letters at
16 kHz:

1. **PL-BERT** on Nepali and English text (masked letters).
2. **Letter aligner** (CTC) on both languages, for durations.
3. **Acoustic model**: text encoder, decoder and both style encoders, adversarial training with
   multi-period and multi-resolution discriminators (300 k steps).
4. **Prosody**: duration, pitch and energy predictors and the prosodic style encoder (100 k steps).
5. **Register**: a short fine-tune (6 k steps) of the prosody half on about 8 hours of
   customer-service style Nepali recordings, with general data replayed so the rest of the model
   does not drift.

Data (acoustic and prosody stages): about 800 hours of Nepali speech from public corpora
(including OpenSLR 43 and OpenSLR 143) and other licensed and in-house recordings, part of it
synthetic, and about 570 hours of English speech synthesised with Kokoro-82M (`af_heart`).

## Evaluation

- **Copy-synthesis intelligibility** (stage 3, durations and pitch taken from the real clip,
  recognised by the same ASR as the real recordings): Nepali character error rate through a
  G.711 telephone channel 0.199 against 0.220 for the real recordings (285 clips); English word
  error rate 0.097 against 0.098 (200 clips).
- **Customer-service register** (stage 5): median pitch stays within 15 Hz of the reference
  voice. The pause rate (about 0.13 pauses per second, about 0.20 with `pause_floor_ms=300`)
  is still below the 0.30-0.45 target band: pauses are only lengthened where the text has
  punctuation.
- Full text-to-speech intelligibility has not yet been measured with the same protocol.

## Limitations

- Experimental. Rare words, names and some English spellings can be mispronounced; Nepali
  rhythm is less polished than English.
- 16 kHz, telephone-oriented; not a studio-quality voice.
- Voices from a reference clip copy that clip's recording conditions too; use clean audio.

## Responsible use

Swar can speak in a voice taken from a short recording. Only use voices you have permission to
use, label synthetic speech as synthetic where people could mistake it for a person, and do not
use the model to impersonate anyone or to deceive.

## Acknowledgements

Swar would not exist without **[Kokoro-82M](https://huggingface.co/hexgrad/Kokoro-82M)** by
hexgrad. Its architecture is Swar's blueprint: the text encoder, the prosody predictor (duration,
pitch and energy), the iSTFTNet decoder with its harmonic source and the split acoustic / prosodic
style design all come from Kokoro, and Kokoro's own layer code is vendored in `swar/kokoro/`. Swar
re-sizes that design for 16 kHz, reads letters instead of phonemes, adds trainable style encoders
and is trained from scratch; Kokoro (voice `af_heart`) also generated Swar's English training
speech. Kokoro in turn builds on [StyleTTS2](https://github.com/yl4579/StyleTTS2) (Li et al.), whose
style encoder and PL-BERT recipe Swar also uses.

## License

The Swar weights (`swar.safetensors`, `voices/`) and Swar's own code are released under
**[CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/)**: free for research and other
non-commercial use, with attribution. The vendored Kokoro code in `swar/kokoro/` stays under its
original Apache-2.0 licence (`swar/kokoro/LICENSE`); the StyleTTS2 style encoder code is MIT.
Training data includes [OpenSLR](https://www.openslr.org/) corpora under their own licences.