File size: 18,956 Bytes
fab51bb
 
bbac2cd
 
 
 
 
 
 
e70f480
 
 
bbac2cd
 
e70f480
 
 
 
bbac2cd
 
fab51bb
bbac2cd
 
 
 
 
 
 
 
 
 
 
 
 
 
 
e70f480
bbac2cd
 
 
 
 
 
 
 
 
 
 
e70f480
 
bbac2cd
 
 
 
 
 
 
 
 
 
 
 
2e659f0
bbac2cd
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2e659f0
 
 
bbac2cd
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
e70f480
 
 
 
 
 
 
bbac2cd
 
 
 
 
 
 
 
e70f480
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
bbac2cd
e70f480
 
 
 
bbac2cd
 
 
 
 
 
e70f480
 
 
 
 
bbac2cd
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
---
license: apache-2.0
language:
  - bn
  - en
pipeline_tag: text-to-speech
tags:
  - text-to-speech
  - bengali
  - bengali-tts
  - indic
  - onnx
  - onnxruntime
  - tiny
  - int8
  - quantized
  - cpu
  - on-device
datasets:
  - ai4bharat/Rasa
---

# tiny-swara

**tiny-swara** is a tiny, absurdly fast Bengali text-to-speech model with two
voices, **BF** (female) and **BM** (male).

It packs the entire pipeline — phoneme encoder, duration predictor, prosody
stacks and a source-filter waveform generator — into **4.5M parameters and an
8.2 MiB INT8 file**. On just **2 CPU threads** it renders **4.75 seconds of
speech in 103 ms: 45.9× faster than real time**, with about 4.6 seconds of
headroom before playback even catches up. No GPU. No cloud. No waiting.

Built on the [tiny-voca](https://huggingface.co/n4ze3m/tiny-voca) architecture,
with a source-filter waveform generator instead of a neural vocoder.

<audio controls preload="none" src="https://huggingface.co/Xeven7/tiny-swara/resolve/main/samples/bm-4.mp3"></audio>

|                   |                                                |
| ----------------- | ---------------------------------------------- |
| Parameters        | 4,537,483                                      |
| Voices            | BF, BM (Bengali)                               |
| Input             | Bengali text, phonemized with espeak-ng (`bn`) |
| Output            | 24 kHz mono                                    |
| Format            | ONNX (opset 17)                                |
| Recommended model | `onnx/tiny-swara-int8.onnx`                    |
| License           | Apache-2.0                                     |

![architecture](<Tiny-Swara Bengali TTS Architecture.png>)

## Files

| File                        |     Size | Contents                                             |
| --------------------------- | -------: | ---------------------------------------------------- |
| `onnx/tiny-swara-int8.onnx` | 8.24 MiB | INT8-quantized model, recommended                    |
| `onnx/tiny-swara.onnx`      | 20.4 MiB | FP32 model, reference                                |
| `config.json`               |          | Model settings, phoneme vocabulary, voices           |
| `inventory.json`            |          | Phoneme inventory and fallback mappings              |
| `lexicon.json`              |          | English-to-Bengali word mappings for code-mixed text |
| `tiny_voca/text.py`         |          | Bengali-patched frontend: text to phoneme tokens     |
| `tiny_voca/__init__.py`     |          | Package marker                                       |
| `inference.py`              |          | ONNX inference, CLI and Python API                   |
| `SETUP.md`                  |          | Beginner setup and troubleshooting                   |
| `requirements.txt`          |          | Python dependencies                                  |
| `samples/`                  |          | MP3 samples used on this page (64 kbps mono)         |

SHA256:

- `onnx/tiny-swara.onnx`: `cef829cff4b6cb38554e32d8f96b9b51f96a4c0e9cb2e406b53ac79d5923d630`
- `onnx/tiny-swara-int8.onnx`: `f3080233715cf22ebf519cfc1f81a8ebe4c937c0cdce0285b2cf51998c64eabb`

## Samples

The first table uses recordings exported from the notebook. The second table was
generated with the packaged `inference.py` and the INT8 model, so it also serves
as an INT8 check. All files here are MP3 at 64 kbps mono to keep the repository
small; the model itself outputs 24 kHz mono WAV.

### Female (BF) and male (BM)

| Text                                                              | Female (BF)                                                                                                          | Male (BM)                                                                                                            |
| ----------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- |
| আমার order কোথায়?                                                | <audio controls preload="none" src="https://huggingface.co/Xeven7/tiny-swara/resolve/main/samples/bf-1.mp3"></audio> | <audio controls preload="none" src="https://huggingface.co/Xeven7/tiny-swara/resolve/main/samples/bm-1.mp3"></audio> |
| গতকাল আমি খুব ব্যস্ত ছিলাম, তাই সন্ধ্যায় তোমাকে ফোন করতে পারিনি। | <audio controls preload="none" src="https://huggingface.co/Xeven7/tiny-swara/resolve/main/samples/bf-2.mp3"></audio> | <audio controls preload="none" src="https://huggingface.co/Xeven7/tiny-swara/resolve/main/samples/bm-2.mp3"></audio> |
| Hello বন্ধুরা, আপনারা কেমন আছেন?                                  | <audio controls preload="none" src="https://huggingface.co/Xeven7/tiny-swara/resolve/main/samples/bf-3.mp3"></audio> | <audio controls preload="none" src="https://huggingface.co/Xeven7/tiny-swara/resolve/main/samples/bm-3.mp3"></audio> |
| বাহ, এই ছোট মডেলটি তো দারুণ! আমি মুগ্ধ।                           | <audio controls preload="none" src="https://huggingface.co/Xeven7/tiny-swara/resolve/main/samples/bf-4.mp3"></audio> | <audio controls preload="none" src="https://huggingface.co/Xeven7/tiny-swara/resolve/main/samples/bm-4.mp3"></audio> |
| আমি তোমাকে পছন্দ করি।                                             | <audio controls preload="none" src="https://huggingface.co/Xeven7/tiny-swara/resolve/main/samples/bf-5.mp3"></audio> | <audio controls preload="none" src="https://huggingface.co/Xeven7/tiny-swara/resolve/main/samples/bm-5.mp3"></audio> |

### INT8 model

| Text                                                        | Voice | Audio                                                                                                                           |
| ----------------------------------------------------------- | ----- | ------------------------------------------------------------------------------------------------------------------------------- |
| আমার সোনার বাংলা, আমি তোমায় ভালোবাসি।                      | BF    | <audio controls preload="none" src="https://huggingface.co/Xeven7/tiny-swara/resolve/main/samples/int8-female.mp3"></audio>     |
| আজকের তারিখ ২০ নভেম্বর ২০২৬, তাপমাত্রা ৩২ ডিগ্রি সেলসিয়াস। | BM    | <audio controls preload="none" src="https://huggingface.co/Xeven7/tiny-swara/resolve/main/samples/int8-male.mp3"></audio>       |
| Hello, আমি আপনার order টি confirm করেছি।                    | BF    | <audio controls preload="none" src="https://huggingface.co/Xeven7/tiny-swara/resolve/main/samples/int8-code-mixed.mp3"></audio> |

## Usage

First time here? [`SETUP.md`](SETUP.md) has copy-paste steps and a
troubleshooting table.

### Python script

```bash
pip install -r requirements.txt
sudo apt install espeak-ng          # or: brew install espeak-ng

python inference.py \
  --text "আমি ভালো আছি, আপনার অর্ডার কনফার্ম হয়েছে।" \
  --voice BF \
  --output hello.wav
```

| Option          | Default                     | Meaning                                         |
| --------------- | --------------------------- | ----------------------------------------------- |
| `--text`        |                             | Text to speak                                   |
| `--text-file`   |                             | Read the text from a file instead of `--text`   |
| `--voice`       | `BF`                        | `BF` (female) or `BM` (male)                    |
| `--output`      | `tiny-swara.wav`            | Output WAV path                                 |
| `--model`       | `onnx/tiny-swara-int8.onnx` | Path to an ONNX file                            |
| `--speed`       | `1.0`                       | Speaking rate; larger is faster                 |
| `--pitch-shift` | `0.0`                       | Added to log F0; 0.058 is about one semitone    |
| `--pitch-range` | `1.0`                       | Scales pitch movement around the utterance mean |

Text is read one sentence at a time, so text of any length works. Output is
peak-normalized to 0.9 before it is written.

### Python API

```python
from inference import TinySwara

tts = TinySwara(voice="BF")        # INT8 model by default
tts.save("আমি ভালো আছি।", "hello.wav")

# Sentence-level streaming: each chunk is complete before it is yielded.
for audio in tts.stream("প্রথম বাক্য। দ্বিতীয় বাক্য।"):
    play(audio)                    # float32 mono at 24 kHz
```

`inference.py` sets `TINY_VOCA_INVENTORY`, `TINY_VOCA_LEXICON` and
`TINY_VOCA_FRONTEND` before importing `tiny_voca.text`, so the patched frontend
is wired up automatically. The frontend must be the Bengali-patched version
shipped here; the upstream tiny-voca `text.py` has no Bengali lexicon support.

### ONNX directly

The graph takes phoneme ids and stress values; text is converted to ids outside
the graph by `tiny_voca/text.py`. The ids are the positions in `vocab` in
`config.json`.

```python
import numpy as np
import onnxruntime as ort

session = ort.InferenceSession("onnx/tiny-swara-int8.onnx")
one = lambda value: np.array([value], dtype=np.float32)

audio, durations = session.run(None, {
    "tokens": np.array([[1, 62, 23, 64, 6, 2]], dtype=np.int64),  # <bos> ami . <eos>
    "stress": np.array([[0, 1, 0, 0, 0, 0]], dtype=np.int64),
    "voice": np.array([0], dtype=np.int64),      # 0 = BF, 1 = BM
    "speed": one(1.0),
    "pitch_shift": one(0.0),
    "pitch_range": one(1.0),
})
```

| Input         | Type    | Shape  | Meaning                                    |
| ------------- | ------- | ------ | ------------------------------------------ |
| `tokens`      | int64   | [1, N] | Phoneme ids, including `<bos>` and `<eos>` |
| `stress`      | int64   | [1, N] | 0 none, 1 primary, 2 secondary             |
| `voice`       | int64   | [1]    | 0 = BF, 1 = BM                             |
| `speed`       | float32 | [1]    | 1.0 as predicted; larger is faster         |
| `pitch_shift` | float32 | [1]    | Added to log F0                            |
| `pitch_range` | float32 | [1]    | Scales pitch movement                      |

| Output      | Type    | Shape        | Meaning                                       |
| ----------- | ------- | ------------ | --------------------------------------------- |
| `audio`     | float32 | [1, samples] | Waveform at 24 kHz                            |
| `durations` | int64   | [1, N]       | Length of each token in frames of 256 samples |

The model reads one utterance per call, up to 512 tokens and 32 seconds. Split
longer text into sentences. The raw graph output can be quiet, so scale it to a
usable level before saving 16-bit samples:

```python
audio = audio[0]
audio *= 0.9 / np.abs(audio).max()
```

## Model

Text is converted to phonemes with [espeak-ng](https://github.com/espeak-ng/espeak-ng)
(`-v bn --ipa`) by `tiny_voca/text.py` and mapped onto a Bengali phoneme
inventory. Phonemes that are not in the inventory are rewritten with the
fallback rules in `inventory.json`. Known English words in code-mixed text are
replaced with their Bengali spelling using `lexicon.json`; other Latin words are
phonemized with the English (`en-us`) voice and mapped back through the same
fallbacks. The token stream matches training: `<bos>`, `<sp>` between words,
explicit punctuation tokens, and `<eos>`.

A transformer encoder (4 blocks) reads the phonemes and a duration predictor
gives each one a length in frames. Two frame-level transformer stacks (2 and 5
blocks) predict pitch, voicing and energy and then the parameters of the
waveform generator. All blocks use 192 channels, 4 heads and rotary position
embeddings. The waveform is produced by a source-filter synthesizer instead of a
neural vocoder: a band-limited pulse train at the predicted pitch and a noise
signal are each shaped by a time-varying filter predicted once per frame
(256 samples, 10.7 ms).

|                |                                                               |
| -------------- | ------------------------------------------------------------- |
| Vocabulary     | 77 tokens (Bengali IPA phonemes, punctuation, special tokens) |
| Encoder        | 4 blocks, 192 channels, 4 heads                               |
| Frame stacks   | 2 + 5 blocks, 192 channels, 4 heads                           |
| Waveform       | Source-filter, hop 256                                        |
| Training steps | 60,000                                                        |

## Training

|               |                                                                                                                                    |
| ------------- | ---------------------------------------------------------------------------------------------------------------------------------- |
| Dataset       | [ai4bharat/Rasa](https://huggingface.co/datasets/ai4bharat/Rasa), Bengali subset, styles WIKI, CONV, BOOK, NEWS, SANGRAH, INDIC    |
| Hours / clips | 11.5 h, 6,275 train / 112 validation / 61 test clips after filtering                                                               |
| Steps         | 60,000                                                                                                                             |
| Hardware      | 1 × Tesla T4 on Kaggle, in two sessions (about 10.5 h and 7 h)                                                                     |
| Alignment     | `facebook/wav2vec2-lv-60-espeak-cv-ft`, relaxed gates (weak-phone fraction 0.5, token probability 0.3); 97% of training clips kept |

- Architecture and recipe from tiny-voca: multi-resolution spectral losses for
  the first 8,000 steps, then adversarial (GAN) training with multi-period and
  multi-resolution discriminators.
- All 60,000 steps ran on one Kaggle Tesla T4 across the two sessions above.

## Evaluation

tiny-swara was evaluated using **AI4Bharat Indic-Transcribe Core** as the Bengali
automatic speech recognition (ASR) evaluator.

### Quantitative results

| Dataset    | Clips | Synthesized WER | Synthesized CER | Reference WER | Reference CER |
| ---------- | ----- | --------------- | --------------- | ------------- | ------------- |
| Validation | 100   | 13.85%          | 5.66%           | 12.10%        | 2.22%         |
| Test       | 64    | 13.28%          | 4.39%           | 13.16%        | 2.38%         |

### Speaker-wise results

| Dataset    | Speaker | Clips | WER    | CER   |
| ---------- | ------- | ----- | ------ | ----- |
| Validation | BF      | 46    | 18.04% | 8.69% |
| Validation | BM      | 54    | 10.24% | 2.99% |
| Test       | BF      | 29    | 12.39% | 4.91% |
| Test       | BM      | 35    | 14.00% | 3.95% |

Test WER is very close to the reference (13.28% vs 13.16%), but CER is noticeably
higher for generated speech (4.39% vs 2.38%). The female voice has the largest
validation gap, while the test split shows the opposite WER ordering — too early
to call one voice consistently worse. The splits also have different difficulty
distributions, so their scores should not be read as a direct improvement or
regression.

### Evaluation methodology

- ASR evaluator: AI4Bharat Indic-Transcribe Core.
- Reference audio: Original Bengali Rasa dataset recordings.
- Synthesized audio: Generated by tiny-swara.
- Metrics: Word Error Rate (WER) and Character Error Rate (CER).
- Text normalization: Unicode NFC normalization and punctuation removal.

### Internal acoustic metrics

These are internal validation numbers on held-out Rasa clips.

| Metric                                     | Value |
| ------------------------------------------ | ----: |
| Mel loss, teacher-forced (lower is better) |  0.70 |
| Mel loss, predicted durations              |  0.72 |
| Pitch error (semitones)                    |  2.07 |
| Voicing accuracy                           |  0.93 |
| Pause accuracy                             |  0.90 |

**Limitations:** Results depend on the ASR evaluator and text normalization.
These metrics measure transcription accuracy rather than subjective speech
naturalness, prosody or speaker similarity. The evaluation covers 100 validation
and 64 test clips and should be treated as a preliminary benchmark.

## Performance

This is the fun part. Measured with the INT8 model on CPU using 2 ONNX Runtime
threads:

|                        |        |
| ---------------------- | -----: |
| Inference (2 threads)  | 103 ms |
| Generated audio        | 4.75 s |
| Real-time factor       |  45.9× |
| Headroom over playback | ~4.6 s |

The test sentence is the 66-token line "গতকাল আমি খুব ব্যস্ত ছিলাম, তাই সন্ধ্যায়
তোমাকে ফোন করতে পারিনি।". The 103 ms covers the ONNX graph only; phonemization
with espeak-ng adds about 70 ms on a desktop CPU, and writing the WAV is
negligible. Speed depends on the CPU and thread count.

## Intended use

Research and personal projects. Not for impersonating any real person.

## Limitations

- Bengali with limited code-mixed English; two fixed voices, no voice cloning.
- Code-mixed English is handled word by word through a small lexicon (213
  entries) plus fallback rules. Foreign names, unusual spellings and words
  outside the lexicon are often mispronounced.
- Numbers and abbreviations depend on espeak-ng's Bengali frontend.
- The same text is always read the same way. There is no sampling of prosody.
- No frame-level streaming; `inference.py` streams sentence by sentence.
- The INT8 model is smaller and faster, but its output differs slightly from
  the FP32 reference.

## License

The weights and code are released under Apache-2.0. tiny-swara is built on the
tiny-voca architecture by Muhammed Nazeem; see
[n4ze3m/tiny-voca](https://huggingface.co/n4ze3m/tiny-voca). The phoneme
inventory and lexicon were produced for this model. The training recordings
come from AI4Bharat Rasa; follow its terms when redistributing derived audio.

## Credits

- tiny-voca architecture and training code by n4ze3m (Apache-2.0).
- Bengali recordings from AI4Bharat Rasa.
- Aligner: `facebook/wav2vec2-lv-60-espeak-cv-ft`.
- Phonemizer: [espeak-ng](https://github.com/espeak-ng/espeak-ng).