|
Download README.md from Xeven7/tiny-swara: direct link, hf CLI and curl.
- Browser
- Download file 19 kB
-
https://huggingface.co/Xeven7/tiny-swara/resolve/main/README.md
- Command line
-
hf download hf://Xeven7/tiny-swara/README.md
-
curl -L -o README.md https://huggingface.co/Xeven7/tiny-swara/resolve/main/README.md
19 kB
| license: apache-2.0 | |
| language: | |
| - bn | |
| - en | |
| pipeline_tag: text-to-speech | |
| tags: | |
| - text-to-speech | |
| - bengali | |
| - bengali-tts | |
| - indic | |
| - onnx | |
| - onnxruntime | |
| - tiny | |
| - int8 | |
| - quantized | |
| - cpu | |
| - on-device | |
| datasets: | |
| - ai4bharat/Rasa | |
| # tiny-swara | |
| **tiny-swara** is a tiny, absurdly fast Bengali text-to-speech model with two | |
| voices, **BF** (female) and **BM** (male). | |
| It packs the entire pipeline — phoneme encoder, duration predictor, prosody | |
| stacks and a source-filter waveform generator — into **4.5M parameters and an | |
| 8.2 MiB INT8 file**. On just **2 CPU threads** it renders **4.75 seconds of | |
| speech in 103 ms: 45.9× faster than real time**, with about 4.6 seconds of | |
| headroom before playback even catches up. No GPU. No cloud. No waiting. | |
| Built on the [tiny-voca](https://huggingface.co/n4ze3m/tiny-voca) architecture, | |
| with a source-filter waveform generator instead of a neural vocoder. | |
| <audio controls preload="none" src="https://huggingface.co/Xeven7/tiny-swara/resolve/main/samples/bm-4.mp3"></audio> | |
| | | | | |
| | ----------------- | ---------------------------------------------- | | |
| | Parameters | 4,537,483 | | |
| | Voices | BF, BM (Bengali) | | |
| | Input | Bengali text, phonemized with espeak-ng (`bn`) | | |
| | Output | 24 kHz mono | | |
| | Format | ONNX (opset 17) | | |
| | Recommended model | `onnx/tiny-swara-int8.onnx` | | |
| | License | Apache-2.0 | | |
|  | |
| ## Files | |
| | File | Size | Contents | | |
| | --------------------------- | -------: | ---------------------------------------------------- | | |
| | `onnx/tiny-swara-int8.onnx` | 8.24 MiB | INT8-quantized model, recommended | | |
| | `onnx/tiny-swara.onnx` | 20.4 MiB | FP32 model, reference | | |
| | `config.json` | | Model settings, phoneme vocabulary, voices | | |
| | `inventory.json` | | Phoneme inventory and fallback mappings | | |
| | `lexicon.json` | | English-to-Bengali word mappings for code-mixed text | | |
| | `tiny_voca/text.py` | | Bengali-patched frontend: text to phoneme tokens | | |
| | `tiny_voca/__init__.py` | | Package marker | | |
| | `inference.py` | | ONNX inference, CLI and Python API | | |
| | `SETUP.md` | | Beginner setup and troubleshooting | | |
| | `requirements.txt` | | Python dependencies | | |
| | `samples/` | | MP3 samples used on this page (64 kbps mono) | | |
| SHA256: | |
| - `onnx/tiny-swara.onnx`: `cef829cff4b6cb38554e32d8f96b9b51f96a4c0e9cb2e406b53ac79d5923d630` | |
| - `onnx/tiny-swara-int8.onnx`: `f3080233715cf22ebf519cfc1f81a8ebe4c937c0cdce0285b2cf51998c64eabb` | |
| ## Samples | |
| The first table uses recordings exported from the notebook. The second table was | |
| generated with the packaged `inference.py` and the INT8 model, so it also serves | |
| as an INT8 check. All files here are MP3 at 64 kbps mono to keep the repository | |
| small; the model itself outputs 24 kHz mono WAV. | |
| ### Female (BF) and male (BM) | |
| | Text | Female (BF) | Male (BM) | | |
| | ----------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- | | |
| | আমার order কোথায়? | <audio controls preload="none" src="https://huggingface.co/Xeven7/tiny-swara/resolve/main/samples/bf-1.mp3"></audio> | <audio controls preload="none" src="https://huggingface.co/Xeven7/tiny-swara/resolve/main/samples/bm-1.mp3"></audio> | | |
| | গতকাল আমি খুব ব্যস্ত ছিলাম, তাই সন্ধ্যায় তোমাকে ফোন করতে পারিনি। | <audio controls preload="none" src="https://huggingface.co/Xeven7/tiny-swara/resolve/main/samples/bf-2.mp3"></audio> | <audio controls preload="none" src="https://huggingface.co/Xeven7/tiny-swara/resolve/main/samples/bm-2.mp3"></audio> | | |
| | Hello বন্ধুরা, আপনারা কেমন আছেন? | <audio controls preload="none" src="https://huggingface.co/Xeven7/tiny-swara/resolve/main/samples/bf-3.mp3"></audio> | <audio controls preload="none" src="https://huggingface.co/Xeven7/tiny-swara/resolve/main/samples/bm-3.mp3"></audio> | | |
| | বাহ, এই ছোট মডেলটি তো দারুণ! আমি মুগ্ধ। | <audio controls preload="none" src="https://huggingface.co/Xeven7/tiny-swara/resolve/main/samples/bf-4.mp3"></audio> | <audio controls preload="none" src="https://huggingface.co/Xeven7/tiny-swara/resolve/main/samples/bm-4.mp3"></audio> | | |
| | আমি তোমাকে পছন্দ করি। | <audio controls preload="none" src="https://huggingface.co/Xeven7/tiny-swara/resolve/main/samples/bf-5.mp3"></audio> | <audio controls preload="none" src="https://huggingface.co/Xeven7/tiny-swara/resolve/main/samples/bm-5.mp3"></audio> | | |
| ### INT8 model | |
| | Text | Voice | Audio | | |
| | ----------------------------------------------------------- | ----- | ------------------------------------------------------------------------------------------------------------------------------- | | |
| | আমার সোনার বাংলা, আমি তোমায় ভালোবাসি। | BF | <audio controls preload="none" src="https://huggingface.co/Xeven7/tiny-swara/resolve/main/samples/int8-female.mp3"></audio> | | |
| | আজকের তারিখ ২০ নভেম্বর ২০২৬, তাপমাত্রা ৩২ ডিগ্রি সেলসিয়াস। | BM | <audio controls preload="none" src="https://huggingface.co/Xeven7/tiny-swara/resolve/main/samples/int8-male.mp3"></audio> | | |
| | Hello, আমি আপনার order টি confirm করেছি। | BF | <audio controls preload="none" src="https://huggingface.co/Xeven7/tiny-swara/resolve/main/samples/int8-code-mixed.mp3"></audio> | | |
| ## Usage | |
| First time here? [`SETUP.md`](SETUP.md) has copy-paste steps and a | |
| troubleshooting table. | |
| ### Python script | |
| ```bash | |
| pip install -r requirements.txt | |
| sudo apt install espeak-ng # or: brew install espeak-ng | |
| python inference.py \ | |
| --text "আমি ভালো আছি, আপনার অর্ডার কনফার্ম হয়েছে।" \ | |
| --voice BF \ | |
| --output hello.wav | |
| ``` | |
| | Option | Default | Meaning | | |
| | --------------- | --------------------------- | ----------------------------------------------- | | |
| | `--text` | | Text to speak | | |
| | `--text-file` | | Read the text from a file instead of `--text` | | |
| | `--voice` | `BF` | `BF` (female) or `BM` (male) | | |
| | `--output` | `tiny-swara.wav` | Output WAV path | | |
| | `--model` | `onnx/tiny-swara-int8.onnx` | Path to an ONNX file | | |
| | `--speed` | `1.0` | Speaking rate; larger is faster | | |
| | `--pitch-shift` | `0.0` | Added to log F0; 0.058 is about one semitone | | |
| | `--pitch-range` | `1.0` | Scales pitch movement around the utterance mean | | |
| Text is read one sentence at a time, so text of any length works. Output is | |
| peak-normalized to 0.9 before it is written. | |
| ### Python API | |
| ```python | |
| from inference import TinySwara | |
| tts = TinySwara(voice="BF") # INT8 model by default | |
| tts.save("আমি ভালো আছি।", "hello.wav") | |
| # Sentence-level streaming: each chunk is complete before it is yielded. | |
| for audio in tts.stream("প্রথম বাক্য। দ্বিতীয় বাক্য।"): | |
| play(audio) # float32 mono at 24 kHz | |
| ``` | |
| `inference.py` sets `TINY_VOCA_INVENTORY`, `TINY_VOCA_LEXICON` and | |
| `TINY_VOCA_FRONTEND` before importing `tiny_voca.text`, so the patched frontend | |
| is wired up automatically. The frontend must be the Bengali-patched version | |
| shipped here; the upstream tiny-voca `text.py` has no Bengali lexicon support. | |
| ### ONNX directly | |
| The graph takes phoneme ids and stress values; text is converted to ids outside | |
| the graph by `tiny_voca/text.py`. The ids are the positions in `vocab` in | |
| `config.json`. | |
| ```python | |
| import numpy as np | |
| import onnxruntime as ort | |
| session = ort.InferenceSession("onnx/tiny-swara-int8.onnx") | |
| one = lambda value: np.array([value], dtype=np.float32) | |
| audio, durations = session.run(None, { | |
| "tokens": np.array([[1, 62, 23, 64, 6, 2]], dtype=np.int64), # <bos> ami . <eos> | |
| "stress": np.array([[0, 1, 0, 0, 0, 0]], dtype=np.int64), | |
| "voice": np.array([0], dtype=np.int64), # 0 = BF, 1 = BM | |
| "speed": one(1.0), | |
| "pitch_shift": one(0.0), | |
| "pitch_range": one(1.0), | |
| }) | |
| ``` | |
| | Input | Type | Shape | Meaning | | |
| | ------------- | ------- | ------ | ------------------------------------------ | | |
| | `tokens` | int64 | [1, N] | Phoneme ids, including `<bos>` and `<eos>` | | |
| | `stress` | int64 | [1, N] | 0 none, 1 primary, 2 secondary | | |
| | `voice` | int64 | [1] | 0 = BF, 1 = BM | | |
| | `speed` | float32 | [1] | 1.0 as predicted; larger is faster | | |
| | `pitch_shift` | float32 | [1] | Added to log F0 | | |
| | `pitch_range` | float32 | [1] | Scales pitch movement | | |
| | Output | Type | Shape | Meaning | | |
| | ----------- | ------- | ------------ | --------------------------------------------- | | |
| | `audio` | float32 | [1, samples] | Waveform at 24 kHz | | |
| | `durations` | int64 | [1, N] | Length of each token in frames of 256 samples | | |
| The model reads one utterance per call, up to 512 tokens and 32 seconds. Split | |
| longer text into sentences. The raw graph output can be quiet, so scale it to a | |
| usable level before saving 16-bit samples: | |
| ```python | |
| audio = audio[0] | |
| audio *= 0.9 / np.abs(audio).max() | |
| ``` | |
| ## Model | |
| Text is converted to phonemes with [espeak-ng](https://github.com/espeak-ng/espeak-ng) | |
| (`-v bn --ipa`) by `tiny_voca/text.py` and mapped onto a Bengali phoneme | |
| inventory. Phonemes that are not in the inventory are rewritten with the | |
| fallback rules in `inventory.json`. Known English words in code-mixed text are | |
| replaced with their Bengali spelling using `lexicon.json`; other Latin words are | |
| phonemized with the English (`en-us`) voice and mapped back through the same | |
| fallbacks. The token stream matches training: `<bos>`, `<sp>` between words, | |
| explicit punctuation tokens, and `<eos>`. | |
| A transformer encoder (4 blocks) reads the phonemes and a duration predictor | |
| gives each one a length in frames. Two frame-level transformer stacks (2 and 5 | |
| blocks) predict pitch, voicing and energy and then the parameters of the | |
| waveform generator. All blocks use 192 channels, 4 heads and rotary position | |
| embeddings. The waveform is produced by a source-filter synthesizer instead of a | |
| neural vocoder: a band-limited pulse train at the predicted pitch and a noise | |
| signal are each shaped by a time-varying filter predicted once per frame | |
| (256 samples, 10.7 ms). | |
| | | | | |
| | -------------- | ------------------------------------------------------------- | | |
| | Vocabulary | 77 tokens (Bengali IPA phonemes, punctuation, special tokens) | | |
| | Encoder | 4 blocks, 192 channels, 4 heads | | |
| | Frame stacks | 2 + 5 blocks, 192 channels, 4 heads | | |
| | Waveform | Source-filter, hop 256 | | |
| | Training steps | 60,000 | | |
| ## Training | |
| | | | | |
| | ------------- | ---------------------------------------------------------------------------------------------------------------------------------- | | |
| | Dataset | [ai4bharat/Rasa](https://huggingface.co/datasets/ai4bharat/Rasa), Bengali subset, styles WIKI, CONV, BOOK, NEWS, SANGRAH, INDIC | | |
| | Hours / clips | 11.5 h, 6,275 train / 112 validation / 61 test clips after filtering | | |
| | Steps | 60,000 | | |
| | Hardware | 1 × Tesla T4 on Kaggle, in two sessions (about 10.5 h and 7 h) | | |
| | Alignment | `facebook/wav2vec2-lv-60-espeak-cv-ft`, relaxed gates (weak-phone fraction 0.5, token probability 0.3); 97% of training clips kept | | |
| - Architecture and recipe from tiny-voca: multi-resolution spectral losses for | |
| the first 8,000 steps, then adversarial (GAN) training with multi-period and | |
| multi-resolution discriminators. | |
| - All 60,000 steps ran on one Kaggle Tesla T4 across the two sessions above. | |
| ## Evaluation | |
| tiny-swara was evaluated using **AI4Bharat Indic-Transcribe Core** as the Bengali | |
| automatic speech recognition (ASR) evaluator. | |
| ### Quantitative results | |
| | Dataset | Clips | Synthesized WER | Synthesized CER | Reference WER | Reference CER | | |
| | ---------- | ----- | --------------- | --------------- | ------------- | ------------- | | |
| | Validation | 100 | 13.85% | 5.66% | 12.10% | 2.22% | | |
| | Test | 64 | 13.28% | 4.39% | 13.16% | 2.38% | | |
| ### Speaker-wise results | |
| | Dataset | Speaker | Clips | WER | CER | | |
| | ---------- | ------- | ----- | ------ | ----- | | |
| | Validation | BF | 46 | 18.04% | 8.69% | | |
| | Validation | BM | 54 | 10.24% | 2.99% | | |
| | Test | BF | 29 | 12.39% | 4.91% | | |
| | Test | BM | 35 | 14.00% | 3.95% | | |
| Test WER is very close to the reference (13.28% vs 13.16%), but CER is noticeably | |
| higher for generated speech (4.39% vs 2.38%). The female voice has the largest | |
| validation gap, while the test split shows the opposite WER ordering — too early | |
| to call one voice consistently worse. The splits also have different difficulty | |
| distributions, so their scores should not be read as a direct improvement or | |
| regression. | |
| ### Evaluation methodology | |
| - ASR evaluator: AI4Bharat Indic-Transcribe Core. | |
| - Reference audio: Original Bengali Rasa dataset recordings. | |
| - Synthesized audio: Generated by tiny-swara. | |
| - Metrics: Word Error Rate (WER) and Character Error Rate (CER). | |
| - Text normalization: Unicode NFC normalization and punctuation removal. | |
| ### Internal acoustic metrics | |
| These are internal validation numbers on held-out Rasa clips. | |
| | Metric | Value | | |
| | ------------------------------------------ | ----: | | |
| | Mel loss, teacher-forced (lower is better) | 0.70 | | |
| | Mel loss, predicted durations | 0.72 | | |
| | Pitch error (semitones) | 2.07 | | |
| | Voicing accuracy | 0.93 | | |
| | Pause accuracy | 0.90 | | |
| **Limitations:** Results depend on the ASR evaluator and text normalization. | |
| These metrics measure transcription accuracy rather than subjective speech | |
| naturalness, prosody or speaker similarity. The evaluation covers 100 validation | |
| and 64 test clips and should be treated as a preliminary benchmark. | |
| ## Performance | |
| This is the fun part. Measured with the INT8 model on CPU using 2 ONNX Runtime | |
| threads: | |
| | | | | |
| | ---------------------- | -----: | | |
| | Inference (2 threads) | 103 ms | | |
| | Generated audio | 4.75 s | | |
| | Real-time factor | 45.9× | | |
| | Headroom over playback | ~4.6 s | | |
| The test sentence is the 66-token line "গতকাল আমি খুব ব্যস্ত ছিলাম, তাই সন্ধ্যায় | |
| তোমাকে ফোন করতে পারিনি।". The 103 ms covers the ONNX graph only; phonemization | |
| with espeak-ng adds about 70 ms on a desktop CPU, and writing the WAV is | |
| negligible. Speed depends on the CPU and thread count. | |
| ## Intended use | |
| Research and personal projects. Not for impersonating any real person. | |
| ## Limitations | |
| - Bengali with limited code-mixed English; two fixed voices, no voice cloning. | |
| - Code-mixed English is handled word by word through a small lexicon (213 | |
| entries) plus fallback rules. Foreign names, unusual spellings and words | |
| outside the lexicon are often mispronounced. | |
| - Numbers and abbreviations depend on espeak-ng's Bengali frontend. | |
| - The same text is always read the same way. There is no sampling of prosody. | |
| - No frame-level streaming; `inference.py` streams sentence by sentence. | |
| - The INT8 model is smaller and faster, but its output differs slightly from | |
| the FP32 reference. | |
| ## License | |
| The weights and code are released under Apache-2.0. tiny-swara is built on the | |
| tiny-voca architecture by Muhammed Nazeem; see | |
| [n4ze3m/tiny-voca](https://huggingface.co/n4ze3m/tiny-voca). The phoneme | |
| inventory and lexicon were produced for this model. The training recordings | |
| come from AI4Bharat Rasa; follow its terms when redistributing derived audio. | |
| ## Credits | |
| - tiny-voca architecture and training code by n4ze3m (Apache-2.0). | |
| - Bengali recordings from AI4Bharat Rasa. | |
| - Aligner: `facebook/wav2vec2-lv-60-espeak-cv-ft`. | |
| - Phonemizer: [espeak-ng](https://github.com/espeak-ng/espeak-ng). | |