Upload README.md
Browse files
README.md
CHANGED
|
@@ -5,6 +5,7 @@ language:
|
|
| 5 |
- ko
|
| 6 |
- zh
|
| 7 |
- ja
|
|
|
|
| 8 |
- en
|
| 9 |
pipeline_tag: text-to-speech
|
| 10 |
tags:
|
|
@@ -15,6 +16,8 @@ tags:
|
|
| 15 |
- chinese
|
| 16 |
- mandarin
|
| 17 |
- japanese
|
|
|
|
|
|
|
| 18 |
- english
|
| 19 |
- bilingual
|
| 20 |
- code-switching
|
|
@@ -34,7 +37,7 @@ tags:
|
|
| 34 |
|
| 35 |
Tiny int8 voices for [vtts](https://github.com/tuanpmt/vtts), a text-to-speech engine that runs entirely on ESP32 chips: text normalization, grapheme-to-phoneme, the acoustic model and the iSTFT vocoder all run on the microcontroller, with no network. Listen to samples generated on the chip on the [website](https://tuanpmt.github.io/vtts/).
|
| 36 |
|
| 37 |
-
Every vtts model is **bilingual: one main language + English**. One voice reads the main language, English, and sentences that mix both ("Đã kết nối Bluetooth với AirPods", "스마트폰을 Wi-Fi에 연결했어요", "已经通过Bluetooth连接到AirPods", "このWi-Fiはpasswordが必要ですか?"); the on-device front-end picks the language per word. Released pairs: Vietnamese + English (v0.4.0), Korean + English and Chinese (Mandarin) + English (v0.5.0), Japanese + English (v0.6.0).
|
| 38 |
|
| 39 |
> These files are not PyTorch or Transformers checkpoints. They are `.vtts` files for the vtts C runtime ([format](https://github.com/tuanpmt/vtts/blob/main/docs/model-format.md)), meant to be flashed with the [ESP32 firmware](https://github.com/tuanpmt/vtts/tree/main/firmware/esp32) or run with the host tools.
|
| 40 |
|
|
@@ -49,27 +52,30 @@ One folder per voice. Each voice comes in two sizes, and both run on the ESP32-P
|
|
| 49 |
| `ko-en-nu-ko/` | `nu-ko` (female, about 40, Seoul) | Korean + English, code-switching | `vtts-koen-nuko-m-v0.5.q8.vtts`, `vtts-koen-nuko-s-v0.5.q8.vtts` | v0.5.0 |
|
| 50 |
| `zh-en-nu-zh/` | `nu-zh` (female, about 40, Beijing Mandarin) | Chinese (Mandarin) + English, code-switching | `vtts-zhen-nuzh-m-v0.5.q8.vtts`, `vtts-zhen-nuzh-s-v0.5.q8.vtts` | v0.5.0 |
|
| 51 |
| `ja-en-nu-ja/` | `nu-ja` (female, about 40, Tokyo) | Japanese + English, code-switching | `vtts-jaen-nuja-m-v0.6.q8.vtts`, `vtts-jaen-nuja-s-v0.6.q8.vtts` | v0.6.0 |
|
|
|
|
| 52 |
|
| 53 |
-
Each folder has a `SHA256SUMS` file. More pairs (Spanish,
|
| 54 |
|
| 55 |
## Models
|
| 56 |
|
| 57 |
| File | Tier | Params | Size | ESP32-P4 RTF | ESP32-S3 RTF |
|
| 58 |
|---|---|---:|---:|---:|---:|
|
| 59 |
-
| `vtts-vien-nam1-m-v0.4.q8.vtts` | Quality (M) | 1.83M | 2.6 MB | 0.26 | 0.
|
| 60 |
-
| `vtts-vien-nam1-s-v0.4.q8.vtts` | Lite (S) | 0.82M | 1.2 MB | 0.18 | 0.
|
| 61 |
-
| `vtts-vien-nu10-m-v0.4.q8.vtts` | Quality (M) | 1.83M | 2.6 MB | 0.26 | 0.
|
| 62 |
-
| `vtts-vien-nu10-s-v0.4.q8.vtts` | Lite (S) | 0.82M | 1.2 MB | 0.18 | 0.
|
| 63 |
-
| `vtts-koen-nuko-m-v0.5.q8.vtts` | Quality (M) | 1.84M | 2.6 MB | 0.26 | 0.
|
| 64 |
-
| `vtts-koen-nuko-s-v0.5.q8.vtts` | Lite (S) | 0.83M | 1.2 MB | 0.17 | 0.
|
| 65 |
-
| `vtts-zhen-nuzh-m-v0.5.q8.vtts` | Quality (M) | 1.84M | 2.6 MB | 0.26 | 0.
|
| 66 |
-
| `vtts-zhen-nuzh-s-v0.5.q8.vtts` | Lite (S) | 0.83M | 1.2 MB | 0.18 | 0.
|
| 67 |
-
| `vtts-jaen-nuja-m-v0.6.q8.vtts` | Quality (M) | 1.84M | 2.6 MB | 0.26 | 0.
|
| 68 |
-
| `vtts-jaen-nuja-s-v0.6.q8.vtts` | Lite (S) | 0.83M | 1.2 MB | 0.17 | 0.
|
| 69 |
-
|
| 70 |
-
|
| 71 |
-
|
| 72 |
-
|
|
|
|
|
|
|
| 73 |
|
| 74 |
Vietnamese + English:
|
| 75 |
|
|
@@ -105,7 +111,17 @@ Japanese + English (`nu-ja`). Japanese CER on readings: reference and transcript
|
|
| 105 |
| `nu-ja` S | 0.053 | 0.019 | 0.049 | 2.30 / 2.76 / 2.28 |
|
| 106 |
| VoxCPM2 `nu-ja` (reference voice) | 0.023 | — | 0.049 | 3.45 / — / 3.43 |
|
| 107 |
|
| 108 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 109 |
|
| 110 |
## Usage
|
| 111 |
|
|
@@ -119,24 +135,25 @@ BOARD=esp32-p4-nano ./build.sh ../../../models/vi-en-nu-10/vtts-vien-nu10-s-v0.4
|
|
| 119 |
../../../models/vi-en-nu-10/vtts-vien-nu10-m-v0.4.q8.vtts -p PORT flash monitor
|
| 120 |
```
|
| 121 |
|
| 122 |
-
For the Korean + English voice download `ko-en-nu-ko/*` and pass `vtts-koen-nuko-s-v0.5.q8.vtts` and `vtts-koen-nuko-m-v0.5.q8.vtts` instead; for Chinese + English download `zh-en-nu-zh/*` and pass `vtts-zhen-nuzh-s-v0.5.q8.vtts` and `vtts-zhen-nuzh-m-v0.5.q8.vtts` (`build.sh` embeds the 116 KB Mandarin reading table they need); for Japanese + English download `ja-en-nu-ja/*` and pass `vtts-jaen-nuja-s-v0.6.q8.vtts` and `vtts-jaen-nuja-m-v0.6.q8.vtts` (`build.sh` embeds the 721 KB Japanese reading table they need).
|
| 123 |
|
| 124 |
-
Type a sentence on the serial console (main language, English or mixed) and the board speaks it; `model s` / `model m` switch between the two sizes and `lang vi|en|auto` sets the text mode of vi-en models (ko-en, zh-en and
|
| 125 |
|
| 126 |
## How the models are made
|
| 127 |
|
| 128 |
-
- **Training data:** speech synthesized with [VoxCPM2](https://github.com/OpenBMB/VoxCPM) (Apache-2.0) from designed voices (not cloned from real people): about 98 h per Vietnamese + English voice, with sentences chosen to cover rare Vietnamese syllables, numbers, English and code-switched text; about 43,600 sentences (Korean, Korean with English words, English) for `nu-ko`; 54,636 sentences (Mandarin, Mandarin with English terms, English) for `nu-zh`; 49,591 sentences (Japanese, Japanese with English words, English) for `nu-ja`. No recorded corpus is used.
|
| 129 |
- **Teacher:** a VITS model (~24M parameters) per voice, trained from random initialization on that synthetic audio only.
|
| 130 |
- **Distillation:** the teacher produces durations, latents and audio; a small student (duration + acoustic model with a pitch/energy predictor + Vocos-style ConvNeXt/iSTFT decoder) learns from it in three stages and is quantized to int8.
|
| 131 |
-
- **Front-end:** rule-based, in Python (training) and C (device) with identical output: a spelling-based Vietnamese G2P, a Korean G2P (Hangul decomposition, the standard pronunciation rules, Sino-Korean and native numbers), a Mandarin front-end (pinyin readings with a polyphone phrase table, tones, Chinese number reading), a Japanese front-end (kana and kanji readings by longest match over a word table, numbers with counters), an English front-end (CMU Pronouncing Dictionary, learned letter-to-sound rules, normalizer) and per-word language tagging. The English lexicon (1.9 MB) is embedded in the firmware, for zh-en also the Mandarin reading table (116 KB), for ja-en the Japanese reading table (721 KB).
|
| 132 |
|
| 133 |
## License and limitations
|
| 134 |
|
| 135 |
- **Apache-2.0** for these model files and for the vtts code.
|
| 136 |
- Naturalness is below the teacher (about one UTMOS point), the usual gap for models this small. Brand names inside Vietnamese sentences and unusual names are the hardest cases.
|
| 137 |
-
- Korean + English: "Bluetooth" and "Google Meet" are read poorly (write 블루투스); numbers are always read in Korean, also inside English sentences
|
| 138 |
-
- Chinese + English: digits inside English sentences are read in Mandarin (write those numbers as words)
|
| 139 |
-
- Japanese + English: word-initial *r* is weak (リビング can sound like イビング); digits inside English sentences are read in Japanese and kanji numerals before 分 are read ぶん (write digits: 30分); "Bluetooth" is read poorly (write ブルートゥース)
|
|
|
|
| 140 |
- Label audio produced with these models as AI-generated, and do not use them to impersonate real people.
|
| 141 |
|
| 142 |
## Links
|
|
|
|
| 5 |
- ko
|
| 6 |
- zh
|
| 7 |
- ja
|
| 8 |
+
- pt
|
| 9 |
- en
|
| 10 |
pipeline_tag: text-to-speech
|
| 11 |
tags:
|
|
|
|
| 16 |
- chinese
|
| 17 |
- mandarin
|
| 18 |
- japanese
|
| 19 |
+
- portuguese
|
| 20 |
+
- brazilian-portuguese
|
| 21 |
- english
|
| 22 |
- bilingual
|
| 23 |
- code-switching
|
|
|
|
| 37 |
|
| 38 |
Tiny int8 voices for [vtts](https://github.com/tuanpmt/vtts), a text-to-speech engine that runs entirely on ESP32 chips: text normalization, grapheme-to-phoneme, the acoustic model and the iSTFT vocoder all run on the microcontroller, with no network. Listen to samples generated on the chip on the [website](https://tuanpmt.github.io/vtts/).
|
| 39 |
|
| 40 |
+
Every vtts model is **bilingual: one main language + English**. One voice reads the main language, English, and sentences that mix both ("Đã kết nối Bluetooth với AirPods", "스마트폰을 Wi-Fi에 연결했어요", "已经通过Bluetooth连接到AirPods", "このWi-Fiはpasswordが必要ですか?", "O sinal do Wi-Fi está fraco"); the on-device front-end picks the language per word. Released pairs: Vietnamese + English (v0.4.0), Korean + English and Chinese (Mandarin) + English (v0.5.0), Japanese + English (v0.6.0), Portuguese (Brazil) + English (v0.7.0).
|
| 41 |
|
| 42 |
> These files are not PyTorch or Transformers checkpoints. They are `.vtts` files for the vtts C runtime ([format](https://github.com/tuanpmt/vtts/blob/main/docs/model-format.md)), meant to be flashed with the [ESP32 firmware](https://github.com/tuanpmt/vtts/tree/main/firmware/esp32) or run with the host tools.
|
| 43 |
|
|
|
|
| 52 |
| `ko-en-nu-ko/` | `nu-ko` (female, about 40, Seoul) | Korean + English, code-switching | `vtts-koen-nuko-m-v0.5.q8.vtts`, `vtts-koen-nuko-s-v0.5.q8.vtts` | v0.5.0 |
|
| 53 |
| `zh-en-nu-zh/` | `nu-zh` (female, about 40, Beijing Mandarin) | Chinese (Mandarin) + English, code-switching | `vtts-zhen-nuzh-m-v0.5.q8.vtts`, `vtts-zhen-nuzh-s-v0.5.q8.vtts` | v0.5.0 |
|
| 54 |
| `ja-en-nu-ja/` | `nu-ja` (female, about 40, Tokyo) | Japanese + English, code-switching | `vtts-jaen-nuja-m-v0.6.q8.vtts`, `vtts-jaen-nuja-s-v0.6.q8.vtts` | v0.6.0 |
|
| 55 |
+
| `pt-en-nu-pt/` | `nu-pt` (female, about 40, Brazilian Portuguese) | Portuguese (Brazil) + English, code-switching | `vtts-pten-nupt-m-v0.7.q8.vtts`, `vtts-pten-nupt-s-v0.7.q8.vtts` | v0.7.0 |
|
| 56 |
|
| 57 |
+
Each folder has a `SHA256SUMS` file. More pairs (Spanish, Indonesian, Hindi, Thai and others, each + English) will follow in their own folders.
|
| 58 |
|
| 59 |
## Models
|
| 60 |
|
| 61 |
| File | Tier | Params | Size | ESP32-P4 RTF | ESP32-S3 RTF |
|
| 62 |
|---|---|---:|---:|---:|---:|
|
| 63 |
+
| `vtts-vien-nam1-m-v0.4.q8.vtts` | Quality (M) | 1.83M | 2.6 MB | 0.26 | 0.70 |
|
| 64 |
+
| `vtts-vien-nam1-s-v0.4.q8.vtts` | Lite (S) | 0.82M | 1.2 MB | 0.18 | 0.42 |
|
| 65 |
+
| `vtts-vien-nu10-m-v0.4.q8.vtts` | Quality (M) | 1.83M | 2.6 MB | 0.26 | 0.70 |
|
| 66 |
+
| `vtts-vien-nu10-s-v0.4.q8.vtts` | Lite (S) | 0.82M | 1.2 MB | 0.18 | 0.42 |
|
| 67 |
+
| `vtts-koen-nuko-m-v0.5.q8.vtts` | Quality (M) | 1.84M | 2.6 MB | 0.26 | 0.70 |
|
| 68 |
+
| `vtts-koen-nuko-s-v0.5.q8.vtts` | Lite (S) | 0.83M | 1.2 MB | 0.17 | 0.41 |
|
| 69 |
+
| `vtts-zhen-nuzh-m-v0.5.q8.vtts` | Quality (M) | 1.84M | 2.6 MB | 0.26 | 0.69 |
|
| 70 |
+
| `vtts-zhen-nuzh-s-v0.5.q8.vtts` | Lite (S) | 0.83M | 1.2 MB | 0.18 | 0.41 |
|
| 71 |
+
| `vtts-jaen-nuja-m-v0.6.q8.vtts` | Quality (M) | 1.84M | 2.6 MB | 0.26 | 0.69 |
|
| 72 |
+
| `vtts-jaen-nuja-s-v0.6.q8.vtts` | Lite (S) | 0.83M | 1.2 MB | 0.17 | 0.41 |
|
| 73 |
+
| `vtts-pten-nupt-m-v0.7.q8.vtts` | Quality (M) | 1.84M | 2.6 MB | 0.28 | 0.69 |
|
| 74 |
+
| `vtts-pten-nupt-s-v0.7.q8.vtts` | Lite (S) | 0.83M | 1.2 MB | 0.19 | 0.42 |
|
| 75 |
+
|
| 76 |
+
Token set `vi-en` (107 tokens: Vietnamese spelling-based phonemes with 6 tones, English ARPAbet with stress), `ko-en` (153 tokens: the same 107 plus 46 Korean onset, vowel and coda tokens) `zh-en` (171 tokens: the vi-en set plus Mandarin initials, finals and tones) `ja-en` (144 tokens: the v2 set plus 37 Japanese phonemes) or `pt-en` (150 tokens: the v2 set plus 43 Brazilian Portuguese phonemes), 22.05 kHz output. RTF (`bench`) below 1 is faster than real time; with streaming, speech starts after 0.06–0.15 s (S) / 0.13–0.34 s (M) on the ESP32-P4 and 0.21–0.48 s (S) / 0.28–0.93 s (M) on the ESP32-S3, for all pairs. Measured on a Waveshare ESP32-P4-86-Panel-ETH-2RO and an ESP32-S3-Touch-LCD-2.8 (8 MB PSRAM required on the S3). Details: [performance report](https://github.com/tuanpmt/vtts/blob/main/docs/reports/performance.md).
|
| 77 |
+
|
| 78 |
+
Quality on three out-of-domain sets of 50 sentences per pair (not in the training text), audio from the int8 C runtime. CER is the character error rate of an ASR transcript (PhoWhisper-large for Vietnamese, tones included; Whisper large-v3 for English, Korean, Mandarin, Japanese, Portuguese and mixed sentences); UTMOS predicts naturalness.
|
| 79 |
|
| 80 |
Vietnamese + English:
|
| 81 |
|
|
|
|
| 111 |
| `nu-ja` S | 0.053 | 0.019 | 0.049 | 2.30 / 2.76 / 2.28 |
|
| 112 |
| VoxCPM2 `nu-ja` (reference voice) | 0.023 | — | 0.049 | 3.45 / — / 3.43 |
|
| 113 |
|
| 114 |
+
Portuguese (Brazil) + English (`nu-pt`), Whisper large-v3 for all three sets:
|
| 115 |
+
|
| 116 |
+
| Model | Portuguese CER | English CER | Mixed CER | UTMOS (pt / en / mixed) |
|
| 117 |
+
|---|---:|---:|---:|---|
|
| 118 |
+
| `nu-pt` M | 0.008 | 0.089 | 0.026 | 2.36 / 2.71 / 2.54 |
|
| 119 |
+
| `nu-pt` S | 0.019 | 0.080 | 0.022 | 2.63 / 2.69 / 2.69 |
|
| 120 |
+
| VoxCPM2 `nu-pt` (reference voice) | 0.044 | — | 0.044 | 3.99 / — / 3.93 |
|
| 121 |
+
|
| 122 |
+
The reference voice misreads raw digits ("R$ 1.250,00"); without the numbers group it scores 0.001 / 0.016 against 0.005 / 0.027 (M) and 0.013 / 0.023 (S). The English CER is higher than for the other pairs because the pt mode tags each Latin word by spelling, and short or common English words that fit Portuguese spelling (I, a, an, time, temperature) are read as Portuguese in whole English sentences; Portuguese sentences with English words are not affected.
|
| 123 |
+
|
| 124 |
+
Reports: [bilingual-student-eval.md](https://github.com/tuanpmt/vtts/blob/main/docs/reports/bilingual-student-eval.md) (vi-en), [ko-student-eval.md](https://github.com/tuanpmt/vtts/blob/main/docs/reports/ko-student-eval.md) (ko-en), [zh-student-eval.md](https://github.com/tuanpmt/vtts/blob/main/docs/reports/zh-student-eval.md) (zh-en), [ja-student-eval.md](https://github.com/tuanpmt/vtts/blob/main/docs/reports/ja-student-eval.md) (ja-en), [pt-student-eval.md](https://github.com/tuanpmt/vtts/blob/main/docs/reports/pt-student-eval.md) (pt-en).
|
| 125 |
|
| 126 |
## Usage
|
| 127 |
|
|
|
|
| 135 |
../../../models/vi-en-nu-10/vtts-vien-nu10-m-v0.4.q8.vtts -p PORT flash monitor
|
| 136 |
```
|
| 137 |
|
| 138 |
+
For the Korean + English voice download `ko-en-nu-ko/*` and pass `vtts-koen-nuko-s-v0.5.q8.vtts` and `vtts-koen-nuko-m-v0.5.q8.vtts` instead; for Chinese + English download `zh-en-nu-zh/*` and pass `vtts-zhen-nuzh-s-v0.5.q8.vtts` and `vtts-zhen-nuzh-m-v0.5.q8.vtts` (`build.sh` embeds the 116 KB Mandarin reading table they need); for Japanese + English download `ja-en-nu-ja/*` and pass `vtts-jaen-nuja-s-v0.6.q8.vtts` and `vtts-jaen-nuja-m-v0.6.q8.vtts` (`build.sh` embeds the 721 KB Japanese reading table they need); for Portuguese (Brazil) + English download `pt-en-nu-pt/*` and pass `vtts-pten-nupt-s-v0.7.q8.vtts` and `vtts-pten-nupt-m-v0.7.q8.vtts` (no blob beyond the English lexicon).
|
| 139 |
|
| 140 |
+
Type a sentence on the serial console (main language, English or mixed) and the board speaks it; `model s` / `model m` switch between the two sizes and `lang vi|en|auto` sets the text mode of vi-en models (ko-en, zh-en, ja-en and pt-en models have one mode each, `ko`, `zh`, `ja` and `pt`, which also read English). The same files are attached to the GitHub releases [v0.4.0](https://github.com/tuanpmt/vtts/releases/tag/v0.4.0) (vi-en), [v0.5.0](https://github.com/tuanpmt/vtts/releases/tag/v0.5.0) (ko-en and zh-en), [v0.6.0](https://github.com/tuanpmt/vtts/releases/tag/v0.6.0) (ja-en) and [v0.7.0](https://github.com/tuanpmt/vtts/releases/tag/v0.7.0) (pt-en).
|
| 141 |
|
| 142 |
## How the models are made
|
| 143 |
|
| 144 |
+
- **Training data:** speech synthesized with [VoxCPM2](https://github.com/OpenBMB/VoxCPM) (Apache-2.0) from designed voices (not cloned from real people): about 98 h per Vietnamese + English voice, with sentences chosen to cover rare Vietnamese syllables, numbers, English and code-switched text; about 43,600 sentences (Korean, Korean with English words, English) for `nu-ko`; 54,636 sentences (Mandarin, Mandarin with English terms, English) for `nu-zh`; 49,591 sentences (Japanese, Japanese with English words, English) for `nu-ja`; 45,191 sentences (Brazilian Portuguese, Portuguese with English words, English) for `nu-pt`. No recorded corpus is used.
|
| 145 |
- **Teacher:** a VITS model (~24M parameters) per voice, trained from random initialization on that synthetic audio only.
|
| 146 |
- **Distillation:** the teacher produces durations, latents and audio; a small student (duration + acoustic model with a pitch/energy predictor + Vocos-style ConvNeXt/iSTFT decoder) learns from it in three stages and is quantized to int8.
|
| 147 |
+
- **Front-end:** rule-based, in Python (training) and C (device) with identical output: a spelling-based Vietnamese G2P, a Korean G2P (Hangul decomposition, the standard pronunciation rules, Sino-Korean and native numbers), a Mandarin front-end (pinyin readings with a polyphone phrase table, tones, Chinese number reading), a Japanese front-end (kana and kanji readings by longest match over a word table, numbers with counters), a Brazilian Portuguese front-end (spelling rules with stress, nasal vowels and the two r sounds; Latin words tagged Portuguese or English per word), an English front-end (CMU Pronouncing Dictionary, learned letter-to-sound rules, normalizer) and per-word language tagging. The English lexicon (1.9 MB) is embedded in the firmware, for zh-en also the Mandarin reading table (116 KB), for ja-en the Japanese reading table (721 KB).
|
| 148 |
|
| 149 |
## License and limitations
|
| 150 |
|
| 151 |
- **Apache-2.0** for these model files and for the vtts code.
|
| 152 |
- Naturalness is below the teacher (about one UTMOS point), the usual gap for models this small. Brand names inside Vietnamese sentences and unusual names are the hardest cases.
|
| 153 |
+
- Korean + English: "Bluetooth" and "Google Meet" are read poorly (write 블루투스); numbers are always read in Korean, also inside English sentences.
|
| 154 |
+
- Chinese + English: digits inside English sentences are read in Mandarin (write those numbers as words).
|
| 155 |
+
- Japanese + English: word-initial *r* is weak (リビング can sound like イビング); digits inside English sentences are read in Japanese and kanji numerals before 分 are read ぶん (write digits: 30分); "Bluetooth" is read poorly (write ブルートゥース).
|
| 156 |
+
- Portuguese (Brazil) + English: whole English sentences are partly read as Portuguese (short words such as I, a, an, and words that fit Portuguese spelling), English CER about 0.08–0.09; acronyms such as ESP32 are spelled with Portuguese letter names; numbers are always read in Portuguese.
|
| 157 |
- Label audio produced with these models as AI-generated, and do not use them to impersonate real people.
|
| 158 |
|
| 159 |
## Links
|