v0.1.1: a cleaner voice from the same weights
Browse filesSame weights file as v0.1. The way the output is turned into sound is corrected while speaking (alpa/model.py, Polish); new browser file onnx/alpa-v0.1.1.onnx beside the v0.1 one; samples re-made; card updated. `--plain` gives the v0.1 sound.
- .gitattributes +2 -0
- README.md +42 -13
- THIRD_PARTY_NOTICES.md +2 -2
- alpa/export.py +322 -0
- alpa/infer.py +11 -6
- alpa/model.py +163 -5
- onnx/alpa-v0.1.1.json +83 -0
- onnx/alpa-v0.1.1.onnx +3 -0
- samples/1_gita_1_1.wav +1 -1
- samples/2_retroflex_and_dental.wav +1 -1
- samples/3_everyday.wav +1 -1
- samples/4_aspirates.wav +1 -1
- samples/5_subhashita.wav +1 -1
- samples/after_v0.1.1.wav +3 -0
- samples/before_v0.1.wav +3 -0
.gitattributes
CHANGED
|
@@ -40,3 +40,5 @@ samples/4_aspirates.wav filter=lfs diff=lfs merge=lfs -text
|
|
| 40 |
samples/5_subhashita.wav filter=lfs diff=lfs merge=lfs -text
|
| 41 |
assets/banner.png filter=lfs diff=lfs merge=lfs -text
|
| 42 |
assets/how.png filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
|
|
|
| 40 |
samples/5_subhashita.wav filter=lfs diff=lfs merge=lfs -text
|
| 41 |
assets/banner.png filter=lfs diff=lfs merge=lfs -text
|
| 42 |
assets/how.png filter=lfs diff=lfs merge=lfs -text
|
| 43 |
+
samples/after_v0.1.1.wav filter=lfs diff=lfs merge=lfs -text
|
| 44 |
+
samples/before_v0.1.wav filter=lfs diff=lfs merge=lfs -text
|
README.md
CHANGED
|
@@ -22,7 +22,9 @@ tags:
|
|
| 22 |
|
| 23 |
**An English voice model, converted to Sanskrit: two tables edited by hand, then 100 minutes on a CPU. No GPU.**
|
| 24 |
|
| 25 |
-
> **Early preview, v0.1.** One trained voice, 3.5 hours of speech, checked by one listener and one recogniser. See [Limits](#limits).
|
|
|
|
|
|
|
| 26 |
|
| 27 |
## Listen
|
| 28 |
|
|
@@ -36,6 +38,27 @@ tags:
|
|
| 36 |
|
| 37 |
Each sample was generated once; nothing was re-rolled or selected. Verse is read aloud as text; Alpa does not chant.
|
| 38 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 39 |
## From English to Sanskrit, without a GPU
|
| 40 |
|
| 41 |

|
|
@@ -102,13 +125,15 @@ The whole path, from typed Devanagari to audio, runs inside the page: the text r
|
|
| 102 |
| CPU slowed 4× | 2.9× real time | 2.7 s |
|
| 103 |
| CPU slowed 6× | 1.9× real time | 4.3 s |
|
| 104 |
|
| 105 |
-
The
|
|
|
|
|
|
|
| 106 |
|
| 107 |
The only other in-browser Sanskrit voice we know of is the port of Vāgdhenu, a chant system. By its own published benchmark it downloads 296 MB and, on a Pixel 10 Pro XL with six threads, needs about 15.5 seconds to chant a 7.5-second verse, with the first quarter-verse arriving after about 3.8 seconds.
|
| 108 |
|
| 109 |
These two are not the same test. Theirs is a real flagship phone; ours is a desktop browser on a 2.1 GHz cloud CPU, slowed with Chrome's own CPU throttle to imitate slower hardware. **Alpa has not yet been timed on a real phone.**
|
| 110 |
|
| 111 |
-
The file is `onnx/alpa-v0.1
|
| 112 |
|
| 113 |
```html
|
| 114 |
<script src="https://cdn.jsdelivr.net/npm/onnxruntime-web@1.20.1/dist/ort.wasm.min.js"></script>
|
|
@@ -118,7 +143,7 @@ The file is `onnx/alpa-v0.1-preview.onnx`; `onnx/alpa-v0.1-preview.json` lists i
|
|
| 118 |
const int64 = (v) => new ort.Tensor("int64", BigInt64Array.from(v.map(BigInt)), [1, v.length]);
|
| 119 |
const one = (x) => new ort.Tensor("float32", Float32Array.from([x]), [1]);
|
| 120 |
ort.env.wasm.numThreads = 1;
|
| 121 |
-
const session = await ort.InferenceSession.create("alpa-v0.1
|
| 122 |
const out = await session.run({ tokens: int64(ids), weight: int64(weight),
|
| 123 |
voice: new ort.Tensor("int64", BigInt64Array.from([1n]), [1]), // 1 = male
|
| 124 |
speed: one(1), pitch_shift: one(0), pitch_range: one(1) });
|
|
@@ -130,14 +155,16 @@ The file is `onnx/alpa-v0.1-preview.onnx`; `onnx/alpa-v0.1-preview.json` lists i
|
|
| 130 |
|
| 131 |
The judge is [Su-śrotā](https://huggingface.co/prathoshap/sushrota-sanskrit-asr), a Sanskrit speech recogniser. It listened to 148 sentences held out of training, spoken once by Alpa from text alone and once by the original voice artist.
|
| 132 |
|
| 133 |
-
| | Alpa, `male` | Human recordings |
|
| 134 |
|---|---:|---:|
|
| 135 |
-
| Phonemes misread | **3.
|
| 136 |
-
| ट heard as ट |
|
| 137 |
-
| ड heard as ड |
|
| 138 |
-
| ण heard as ण |
|
| 139 |
| त, द, न heard correctly | 98–100% | 98–100% |
|
| 140 |
|
|
|
|
|
|
|
| 141 |
Read these with two cautions. The recogniser knows Sanskrit words and partly guesses from context, which flatters both columns; the stricter between-vowels figures are in the table further up. And a recogniser is not an ear: one human listener has checked a handful of lines.
|
| 142 |
|
| 143 |
**Against other systems.** There is no shared benchmark for Sanskrit speech synthesis yet, and the other systems have not been run through this judge, so we make no quality claim against them. Vāgdhenu's authors report a mean opinion score of about 4.6 from an expert listener, for chant. Alpa has no listener-panel score. On desktop speed the published figures are: Alpa about 100 times faster than real time on four CPU threads; Vāgdhenu about 3.5 times faster than real time on an M5 Max.
|
|
@@ -163,12 +190,12 @@ audio, phonemes, _ = speak(model, "सत्यमेव जयते नान
|
|
| 163 |
sf.write("out.wav", audio.numpy(), 24000)
|
| 164 |
```
|
| 165 |
|
| 166 |
-
`speak` also takes `speed`, `pitch_shift` and `pitch_range`. Text in Kannada, Telugu, Bengali or Malayalam script is read as well if you `pip install indic-transliteration`. The model file holds plain tensors and settings only and is loaded with `weights_only=True`.
|
| 167 |
|
| 168 |
| | |
|
| 169 |
|---|---|
|
| 170 |
| Parameters | 4,531,723 |
|
| 171 |
-
| Files | `alpa-v0.1-preview.pt` (18 MB), `onnx/alpa-v0.1-preview.onnx` (21 MB) |
|
| 172 |
| Audio | 24 kHz, mono |
|
| 173 |
| Input | Sanskrit in Devanagari |
|
| 174 |
| Voices | `male` is trained. `female` is present but was never trained on Sanskrit |
|
|
@@ -185,8 +212,10 @@ sf.write("out.wav", audio.numpy(), 24000)
|
|
| 185 |
|
| 186 |
## Limits
|
| 187 |
|
| 188 |
-
- **One trained voice.** With `female`, ट and ड are noticeably weaker.
|
| 189 |
-
- **Little data and a short run.** It learned from 3.5 hours, and the adversarial stage of tiny-voca's recipe, which adds much of the naturalness, has not been run. The voice can sound flatter
|
|
|
|
|
|
|
| 190 |
- **ण and न** are not yet fully distinct to a human listener.
|
| 191 |
- **Word-final visarga is sometimes weak.** The recogniser missed it in three of the five samples above.
|
| 192 |
- **It speaks; it does not chant.** Vedic accent marks are ignored.
|
|
|
|
| 22 |
|
| 23 |
**An English voice model, converted to Sanskrit: two tables edited by hand, then 100 minutes on a CPU. No GPU.**
|
| 24 |
|
| 25 |
+
> **Early preview, v0.1.1.** One trained voice, 3.5 hours of speech, checked by one listener and one recogniser. See [Limits](#limits).
|
| 26 |
+
>
|
| 27 |
+
> **New in v0.1.1:** a cleaner voice from the same weights. See [What changed in v0.1.1](#what-changed-in-v011).
|
| 28 |
|
| 29 |
## Listen
|
| 30 |
|
|
|
|
| 38 |
|
| 39 |
Each sample was generated once; nothing was re-rolled or selected. Verse is read aloud as text; Alpa does not chant.
|
| 40 |
|
| 41 |
+
## What changed in v0.1.1
|
| 42 |
+
|
| 43 |
+
The weights are the same file as v0.1. What changed is how the model's output is turned into sound, and again nothing was trained.
|
| 44 |
+
|
| 45 |
+
| | Audio |
|
| 46 |
+
|---|---|
|
| 47 |
+
| v0.1 | <audio controls src="https://huggingface.co/77ethers/alpa/resolve/main/samples/before_v0.1.wav"></audio> |
|
| 48 |
+
| v0.1.1 | <audio controls src="https://huggingface.co/77ethers/alpa/resolve/main/samples/after_v0.1.1.wav"></audio> |
|
| 49 |
+
|
| 50 |
+
*श्रीः स्त्री शास्त्रं स्पष्टं पश्यति। क्षत्रियः राष्ट्रं रक्षति।*
|
| 51 |
+
|
| 52 |
+
After the release, listeners reported a buzz. Our listener narrowed it down: in श, ष and क्ष, and then in ज, घ and ध, it sounded like "two metallic people saying at once". The cause was in the inherited sound generator. It makes the voice from pitched pulses, and the model had learnt to build its hiss, breath and bursts out of those same pulses, so every noisy sound carried a second, pitched voice inside it.
|
| 53 |
+
|
| 54 |
+
v0.1.1 corrects this while speaking:
|
| 55 |
+
|
| 56 |
+
- **Each sound gets a frequency limit.** Below it the generator is fed the voice's pulses; above it, plain noise of the same loudness. Sibilants and च are noise throughout; stops, ज and ह keep the voice only below 1 kHz; vowels keep it up to 4.5 kHz, as real voices do.
|
| 57 |
+
- **Sharper vowels.** The resonances that tell one vowel from another were flatter than in the recordings; they are restored to the recordings' level.
|
| 58 |
+
- **Smaller touches:** a steadier pitch, less whisper inside vowels, and a slightly brighter top.
|
| 59 |
+
|
| 60 |
+
Measured against the recordings, the noisy sounds now have no pitch above 1 kHz, as in real speech, and the recogniser's score is unchanged (3.0% of phonemes misread, 3.1% before). On a side-by-side of six sentences our listener judged the new voice clearly better. `--plain` gives the earlier sound.
|
| 61 |
+
|
| 62 |
## From English to Sanskrit, without a GPU
|
| 63 |
|
| 64 |

|
|
|
|
| 125 |
| CPU slowed 4× | 2.9× real time | 2.7 s |
|
| 126 |
| CPU slowed 6× | 1.9× real time | 4.3 s |
|
| 127 |
|
| 128 |
+
These timings are of the v0.1 file. The v0.1.1 file, measured beside it on the same day, is about 5% slower (14.0× against 14.5× real time on that day's machine).
|
| 129 |
+
|
| 130 |
+
The model file for the browser is 23 MB; with both runtimes a page needs 47 MB in total.
|
| 131 |
|
| 132 |
The only other in-browser Sanskrit voice we know of is the port of Vāgdhenu, a chant system. By its own published benchmark it downloads 296 MB and, on a Pixel 10 Pro XL with six threads, needs about 15.5 seconds to chant a 7.5-second verse, with the first quarter-verse arriving after about 3.8 seconds.
|
| 133 |
|
| 134 |
These two are not the same test. Theirs is a real flagship phone; ours is a desktop browser on a 2.1 GHz cloud CPU, slowed with Chrome's own CPU throttle to imitate slower hardware. **Alpa has not yet been timed on a real phone.**
|
| 135 |
|
| 136 |
+
The file is `onnx/alpa-v0.1.1.onnx`; `onnx/alpa-v0.1.1.json` lists its inputs and the phoneme table. It takes the same inputs as the v0.1 file, `onnx/alpa-v0.1-preview.onnx`, which is still here.
|
| 137 |
|
| 138 |
```html
|
| 139 |
<script src="https://cdn.jsdelivr.net/npm/onnxruntime-web@1.20.1/dist/ort.wasm.min.js"></script>
|
|
|
|
| 143 |
const int64 = (v) => new ort.Tensor("int64", BigInt64Array.from(v.map(BigInt)), [1, v.length]);
|
| 144 |
const one = (x) => new ort.Tensor("float32", Float32Array.from([x]), [1]);
|
| 145 |
ort.env.wasm.numThreads = 1;
|
| 146 |
+
const session = await ort.InferenceSession.create("alpa-v0.1.1.onnx", { executionProviders: ["wasm"] });
|
| 147 |
const out = await session.run({ tokens: int64(ids), weight: int64(weight),
|
| 148 |
voice: new ort.Tensor("int64", BigInt64Array.from([1n]), [1]), // 1 = male
|
| 149 |
speed: one(1), pitch_shift: one(0), pitch_range: one(1) });
|
|
|
|
| 155 |
|
| 156 |
The judge is [Su-śrotā](https://huggingface.co/prathoshap/sushrota-sanskrit-asr), a Sanskrit speech recogniser. It listened to 148 sentences held out of training, spoken once by Alpa from text alone and once by the original voice artist.
|
| 157 |
|
| 158 |
+
| | Alpa v0.1.1, `male` | Human recordings |
|
| 159 |
|---|---:|---:|
|
| 160 |
+
| Phonemes misread | **3.0%** | 3.3% |
|
| 161 |
+
| ट heard as ट | 94% | 97% |
|
| 162 |
+
| ड heard as ड | 87% | 94% |
|
| 163 |
+
| ण heard as ण | 97% | 99% |
|
| 164 |
| त, द, न heard correctly | 98–100% | 98–100% |
|
| 165 |
|
| 166 |
+
v0.1, with the same weights, scored 3.1%, 93%, 89% and 98% on the first four rows: the cleaner voice left pronunciation where it was.
|
| 167 |
+
|
| 168 |
Read these with two cautions. The recogniser knows Sanskrit words and partly guesses from context, which flatters both columns; the stricter between-vowels figures are in the table further up. And a recogniser is not an ear: one human listener has checked a handful of lines.
|
| 169 |
|
| 170 |
**Against other systems.** There is no shared benchmark for Sanskrit speech synthesis yet, and the other systems have not been run through this judge, so we make no quality claim against them. Vāgdhenu's authors report a mean opinion score of about 4.6 from an expert listener, for chant. Alpa has no listener-panel score. On desktop speed the published figures are: Alpa about 100 times faster than real time on four CPU threads; Vāgdhenu about 3.5 times faster than real time on an M5 Max.
|
|
|
|
| 190 |
sf.write("out.wav", audio.numpy(), 24000)
|
| 191 |
```
|
| 192 |
|
| 193 |
+
`speak` also takes `speed`, `pitch_shift` and `pitch_range`. For the sound of v0.1 as first published, add `--plain` to the command or pass `polish=None` to `speak`. `python -m alpa.export` writes the browser file (it needs `pip install onnx onnxruntime`). Text in Kannada, Telugu, Bengali or Malayalam script is read as well if you `pip install indic-transliteration`. The model file holds plain tensors and settings only and is loaded with `weights_only=True`.
|
| 194 |
|
| 195 |
| | |
|
| 196 |
|---|---|
|
| 197 |
| Parameters | 4,531,723 |
|
| 198 |
+
| Files | `alpa-v0.1-preview.pt` (18 MB, unchanged since v0.1), `onnx/alpa-v0.1.1.onnx` (23 MB), `onnx/alpa-v0.1-preview.onnx` (21 MB, the v0.1 sound) |
|
| 199 |
| Audio | 24 kHz, mono |
|
| 200 |
| Input | Sanskrit in Devanagari |
|
| 201 |
| Voices | `male` is trained. `female` is present but was never trained on Sanskrit |
|
|
|
|
| 212 |
|
| 213 |
## Limits
|
| 214 |
|
| 215 |
+
- **One trained voice.** With `female`, ट and ड are noticeably weaker, the visarga fades, and our listener judged the voice not good enough to use.
|
| 216 |
+
- **Little data and a short run.** It learned from 3.5 hours, and the adversarial stage of tiny-voca's recipe, which adds much of the naturalness, has not been run. The voice can sound flatter than larger systems.
|
| 217 |
+
- **The cleaner voice is a correction, not a cure.** v0.1.1 fixes the sound while speaking; the model itself still makes the faults, and `--plain` lets you hear them.
|
| 218 |
+
- **A few vowels come out hoarse.** The ए of ते in कर्मण्येवाधिकारस्ते… मा ते सङ्गोऽस्त्वकर्मणि is the known case; it is not fixed.
|
| 219 |
- **ण and न** are not yet fully distinct to a human listener.
|
| 220 |
- **Word-final visarga is sometimes weak.** The recogniser missed it in three of the five samples above.
|
| 221 |
- **It speaks; it does not chant.** Vedic accent marks are ignored.
|
THIRD_PARTY_NOTICES.md
CHANGED
|
@@ -8,8 +8,8 @@ licensed under the Apache License, Version 2.0. A copy of that licence is in `LI
|
|
| 8 |
|
| 9 |
- **Weights.** `alpa-v0.1-preview.pt` was trained starting from tiny-voca's published weights. Its phoneme
|
| 10 |
table and its stress table were rewritten for Sanskrit before training; every other layer started as a copy.
|
| 11 |
-
- **Browser
|
| 12 |
-
- **Code.** `alpa/model.py`, `alpa/config.py` and `alpa/
|
| 13 |
same names. Each states its changes at its top. `alpa/text.py` is original to Alpa.
|
| 14 |
|
| 15 |
## Vāgdhenu
|
|
|
|
| 8 |
|
| 9 |
- **Weights.** `alpa-v0.1-preview.pt` was trained starting from tiny-voca's published weights. Its phoneme
|
| 10 |
table and its stress table were rewritten for Sanskrit before training; every other layer started as a copy.
|
| 11 |
+
- **Browser files.** `onnx/alpa-v0.1-preview.onnx` was exported with tiny-voca's exporter (`src/tiny_voca/export.py`), unchanged. `onnx/alpa-v0.1.1.onnx` was exported with `alpa/export.py`, which is adapted from it.
|
| 12 |
+
- **Code.** `alpa/model.py`, `alpa/config.py`, `alpa/infer.py` and `alpa/export.py` are adapted from tiny-voca's files of the
|
| 13 |
same names. Each states its changes at its top. `alpa/text.py` is original to Alpa.
|
| 14 |
|
| 15 |
## Vāgdhenu
|
alpa/export.py
ADDED
|
@@ -0,0 +1,322 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Adapted from tiny-voca by Muhammed Nazeem, commit 28e0722 (Apache-2.0; see LICENSES/Apache-2.0.txt
|
| 2 |
+
# and THIRD_PARTY_NOTICES.md). File: src/tiny_voca/export.py. Changes for Alpa: Alpa's model and syllable weight; the
|
| 3 |
+
# second input is named ``weight``; the corrections made while speaking (model.Polish) are part of the graph.
|
| 4 |
+
|
| 5 |
+
"""ONNX export of free-running synthesis.
|
| 6 |
+
|
| 7 |
+
python -m alpa.export --checkpoint inference.pt --output alpa.onnx
|
| 8 |
+
|
| 9 |
+
The exported graph speaks one utterance: phoneme ids and syllable weights in,
|
| 10 |
+
waveform out. Text is turned into ids outside the graph (``alpa.text.phonemize``).
|
| 11 |
+
|
| 12 |
+
``Exportable`` computes the same audio as ``Alpa.synthesize`` for a single
|
| 13 |
+
utterance, written with operations ONNX has: the short-time Fourier transforms
|
| 14 |
+
of the source-filter head become real convolutions with fixed kernels, the
|
| 15 |
+
complex filter responses are kept as separate real and imaginary parts, and
|
| 16 |
+
the frame expansion uses a comparison instead of a sorted search.
|
| 17 |
+
|
| 18 |
+
Inputs
|
| 19 |
+
tokens int64 [1, N] phoneme ids, with the edge tokens
|
| 20 |
+
weight int64 [1, N] 0 none, 1 laghu, 2 guru
|
| 21 |
+
voice int64 [1] index into ``voices`` in the metadata
|
| 22 |
+
speed float32 [1] 1.0 = as predicted, larger is faster
|
| 23 |
+
pitch_shift float32 [1] added to log F0 (0.058 is about a semitone)
|
| 24 |
+
pitch_range float32 [1] scales pitch movement around the mean
|
| 25 |
+
Outputs
|
| 26 |
+
audio float32 [1, samples]
|
| 27 |
+
durations int64 [1, N] frames per token (``hop`` samples each)
|
| 28 |
+
"""
|
| 29 |
+
|
| 30 |
+
import argparse
|
| 31 |
+
import dataclasses
|
| 32 |
+
import json
|
| 33 |
+
import math
|
| 34 |
+
from pathlib import Path
|
| 35 |
+
|
| 36 |
+
import torch
|
| 37 |
+
from torch import nn
|
| 38 |
+
import torch.nn.functional as F
|
| 39 |
+
|
| 40 |
+
from .infer import load_model
|
| 41 |
+
from .model import ALL_PULSES, NO_PULSES, Polish, treble_kernel
|
| 42 |
+
from .text import SYMBOLS, frontend_fingerprint
|
| 43 |
+
|
| 44 |
+
INPUTS = ("tokens", "weight", "voice", "speed", "pitch_shift", "pitch_range")
|
| 45 |
+
OUTPUTS = ("audio", "durations")
|
| 46 |
+
|
| 47 |
+
|
| 48 |
+
def _cepstrum_matrices(positions, fft):
|
| 49 |
+
"""Real and imaginary parts of rfft for unit coefficients at ``positions``."""
|
| 50 |
+
buffer = torch.zeros(len(positions), fft, dtype=torch.float64)
|
| 51 |
+
buffer[torch.arange(len(positions)), torch.tensor(positions)] = 1
|
| 52 |
+
spectrum = torch.fft.rfft(buffer, dim=-1)
|
| 53 |
+
return spectrum.real.float(), spectrum.imag.float()
|
| 54 |
+
|
| 55 |
+
|
| 56 |
+
def _analysis_kernels(window):
|
| 57 |
+
"""Convolution kernels [bins, 1, fft] giving the real and imaginary STFT."""
|
| 58 |
+
fft = len(window)
|
| 59 |
+
spectrum = torch.fft.rfft(torch.eye(fft, dtype=torch.float64) * window.double(), dim=-1)
|
| 60 |
+
return spectrum.real.T[:, None].float().contiguous(), spectrum.imag.T[:, None].float().contiguous()
|
| 61 |
+
|
| 62 |
+
|
| 63 |
+
def _synthesis_kernel(window):
|
| 64 |
+
"""Transposed-convolution kernel [2 * bins, 1, fft]: windowed inverse rfft
|
| 65 |
+
of the real parts (first half of the channels) and imaginary parts."""
|
| 66 |
+
fft = len(window)
|
| 67 |
+
bins = fft // 2 + 1
|
| 68 |
+
unit = torch.eye(bins, dtype=torch.complex128)
|
| 69 |
+
real = torch.fft.irfft(unit, n=fft, dim=-1)
|
| 70 |
+
imaginary = torch.fft.irfft(unit * 1j, n=fft, dim=-1)
|
| 71 |
+
return (torch.cat((real, imaginary)) * window.double())[:, None].float().contiguous()
|
| 72 |
+
|
| 73 |
+
|
| 74 |
+
class Exportable(nn.Module):
|
| 75 |
+
"""Single-utterance synthesis in ONNX-friendly operations. ``polish`` as in ``Alpa.synthesize``."""
|
| 76 |
+
|
| 77 |
+
def __init__(self, model, external_noise=False, polish=Polish()):
|
| 78 |
+
super().__init__()
|
| 79 |
+
self.model = model
|
| 80 |
+
self.external_noise = external_noise
|
| 81 |
+
self.polish = polish
|
| 82 |
+
c = model.config
|
| 83 |
+
v = model.vocoder
|
| 84 |
+
causal = list(range(c.cep_causal))
|
| 85 |
+
anti = [c.synth_fft - 1 - i for i in range(c.cep_anti)]
|
| 86 |
+
for name, (real, imaginary) in (("harmonic", _cepstrum_matrices(causal + anti, c.synth_fft)),
|
| 87 |
+
("noise", _cepstrum_matrices(list(range(c.noise_cep)), c.noise_fft))):
|
| 88 |
+
self.register_buffer(f"{name}_real", real, persistent=False)
|
| 89 |
+
self.register_buffer(f"{name}_imaginary", imaginary, persistent=False)
|
| 90 |
+
for name, window in (("synth", v.synth_window), ("noise", v.noise_window)):
|
| 91 |
+
real, imaginary = _analysis_kernels(window)
|
| 92 |
+
self.register_buffer(f"{name}_analysis_real", real, persistent=False)
|
| 93 |
+
self.register_buffer(f"{name}_analysis_imaginary", imaginary, persistent=False)
|
| 94 |
+
self.register_buffer(f"{name}_synthesis", _synthesis_kernel(window), persistent=False)
|
| 95 |
+
self.register_buffer(f"{name}_envelope", (window.double() ** 2).float()[None, None].contiguous(), persistent=False)
|
| 96 |
+
self.register_buffer("ramp", torch.arange(c.hop, dtype=torch.float32) / c.hop, persistent=False)
|
| 97 |
+
self.phase_steps = 1 << 24
|
| 98 |
+
if polish is not None:
|
| 99 |
+
# Which cepstral coefficients sharpening stretches, in the order causal then anticausal.
|
| 100 |
+
chosen = torch.zeros(c.cep_causal)
|
| 101 |
+
chosen[polish.sharpen_from:polish.sharpen_to] = 1
|
| 102 |
+
mirror = torch.zeros(c.cep_anti)
|
| 103 |
+
mirror[max(polish.sharpen_from - 1, 0):max(polish.sharpen_to - 1, 0)] = 1
|
| 104 |
+
self.register_buffer("stretched", torch.cat((chosen, mirror)), persistent=False)
|
| 105 |
+
triangle = torch.cat((torch.arange(1, polish.pitch_smooth + 2), torch.arange(polish.pitch_smooth, 0, -1))).float()
|
| 106 |
+
self.register_buffer("pitch_kernel", (triangle / triangle.sum())[None, None], persistent=False)
|
| 107 |
+
self.register_buffer("rumble_cut", torch.sigmoid((v.synth_frequency - 90.0) / 12.0)[None, :, None], persistent=False)
|
| 108 |
+
self.register_buffer("treble", treble_kernel(polish.treble, c.sample_rate)[None, None], persistent=False)
|
| 109 |
+
|
| 110 |
+
def _analysis(self, source, name, fft, hop):
|
| 111 |
+
"""Real and imaginary STFT [1, bins, frames] of ``source`` [1, samples]."""
|
| 112 |
+
padded = F.pad(source[:, None], (fft // 2, fft))
|
| 113 |
+
return (F.conv1d(padded, getattr(self, f"{name}_analysis_real"), stride=hop),
|
| 114 |
+
F.conv1d(padded, getattr(self, f"{name}_analysis_imaginary"), stride=hop))
|
| 115 |
+
|
| 116 |
+
def _synthesis(self, s_real, s_imaginary, real, imaginary, name, fft, hop, samples):
|
| 117 |
+
"""Multiply an STFT by a response given as real and imaginary parts
|
| 118 |
+
[1, frames, bins], and resynthesize ``samples`` samples."""
|
| 119 |
+
real, imaginary = real.transpose(1, 2), imaginary.transpose(1, 2)
|
| 120 |
+
product = torch.cat((s_real * real - s_imaginary * imaginary, s_real * imaginary + s_imaginary * real), dim=1)
|
| 121 |
+
wave = F.conv_transpose1d(product, getattr(self, f"{name}_synthesis"), stride=hop)
|
| 122 |
+
envelope = F.conv_transpose1d(torch.ones_like(s_real[:, :1]), getattr(self, f"{name}_envelope"), stride=hop)
|
| 123 |
+
return (wave / envelope)[:, 0, fft // 2:fft // 2 + samples]
|
| 124 |
+
|
| 125 |
+
def _filter(self, source, real, imaginary, name, fft, hop):
|
| 126 |
+
s_real, s_imaginary = self._analysis(source, name, fft, hop)
|
| 127 |
+
return self._synthesis(s_real, s_imaginary, real, imaginary, name, fft, hop, source.shape[1])
|
| 128 |
+
|
| 129 |
+
@staticmethod
|
| 130 |
+
def _response(cepstrum, real, imaginary):
|
| 131 |
+
magnitude = torch.exp((cepstrum @ real).clamp(-14, 6))
|
| 132 |
+
angle = cepstrum @ imaginary
|
| 133 |
+
return magnitude * torch.cos(angle), magnitude * torch.sin(angle)
|
| 134 |
+
|
| 135 |
+
def _pulses(self, f0):
|
| 136 |
+
"""``pulse_train`` for one utterance, without 64-bit floats (the web
|
| 137 |
+
runtime has none). The phase is summed inside each frame in float32,
|
| 138 |
+
where it stays below a few cycles, and across frames as 24-bit integer
|
| 139 |
+
fractions of a cycle, which wrap exactly however long the utterance is."""
|
| 140 |
+
c = self.model.config
|
| 141 |
+
knots = torch.log(f0.clamp_min(1.0))
|
| 142 |
+
knots = torch.cat((knots, knots[:, -1:]), dim=1)
|
| 143 |
+
low, high = knots[:, :-1, None], knots[:, 1:, None]
|
| 144 |
+
frequency = torch.exp(low * (1 - self.ramp) + high * self.ramp)
|
| 145 |
+
within = torch.cumsum(frequency / c.sample_rate, dim=2)
|
| 146 |
+
whole_frame = within[:, :, -1]
|
| 147 |
+
steps = torch.round((whole_frame - torch.floor(whole_frame)) * self.phase_steps).long()
|
| 148 |
+
start = torch.remainder(torch.cumsum(steps, dim=1) - steps, self.phase_steps).to(within.dtype) / self.phase_steps
|
| 149 |
+
cycles = within + start[:, :, None]
|
| 150 |
+
theta = (2 * math.pi * (cycles - torch.floor(cycles))).reshape(1, -1)
|
| 151 |
+
frequency = frequency.reshape(1, -1)
|
| 152 |
+
harmonics = (c.harmonic_limit * c.sample_rate / frequency).clamp_min(1.0)
|
| 153 |
+
whole = torch.floor(harmonics)
|
| 154 |
+
denominator = 2 * torch.sin(theta / 2)
|
| 155 |
+
tiny = denominator.abs() < 1e-6
|
| 156 |
+
core = torch.where(tiny, whole, torch.sin((whole + 0.5) * theta) / torch.where(tiny, torch.ones_like(denominator), denominator) - 0.5)
|
| 157 |
+
wave = core + (harmonics - whole) * torch.cos((whole + 1) * theta)
|
| 158 |
+
return wave * torch.sqrt(2.0 / harmonics)
|
| 159 |
+
|
| 160 |
+
def _vocoder(self, hidden, f0, total, noise, breath=None, limits=None, open_=None, vowel=None):
|
| 161 |
+
c = self.model.config
|
| 162 |
+
v = self.model.vocoder
|
| 163 |
+
p = self.polish
|
| 164 |
+
last = total - 1
|
| 165 |
+
# Harmonic branch.
|
| 166 |
+
h = v.harmonic(hidden)
|
| 167 |
+
split = c.cep_causal + c.cep_anti
|
| 168 |
+
steps = v.harmonic_steps
|
| 169 |
+
sub = torch.minimum(torch.arange(steps * total + 3, device=hidden.device), steps * total - 1)
|
| 170 |
+
here = torch.div(sub, steps, rounding_mode="floor")
|
| 171 |
+
following = torch.minimum(torch.div(sub + 1, steps, rounding_mode="floor"), last)
|
| 172 |
+
envelope = 0.5 * (h[..., :split].index_select(1, here) + h[..., :split].index_select(1, following))
|
| 173 |
+
envelope = envelope * torch.cat((v.causal_lifter, v.anti_lifter))
|
| 174 |
+
gain = h[..., split:].flatten(1).index_select(1, sub)[..., None]
|
| 175 |
+
envelope = torch.cat((envelope[..., :1] + gain, envelope[..., 1:]), dim=-1)
|
| 176 |
+
if p is not None and p.sharpen != 1.0:
|
| 177 |
+
# Sharper formants in vowels, nasals and semivowels, with the loudness of each broad band put back.
|
| 178 |
+
beta = (1.0 + (p.sharpen - 1.0) * open_.to(envelope.dtype)).index_select(1, here)[..., None]
|
| 179 |
+
sharp = envelope * (1 + self.stretched * (beta - 1))
|
| 180 |
+
flat_level, sharp_level = (envelope @ self.harmonic_real).clamp(-14, 6), (sharp @ self.harmonic_real).clamp(-14, 6)
|
| 181 |
+
power = lambda level: torch.exp(2 * level) @ v.band_mask.T + 1e-20
|
| 182 |
+
magnitude = torch.exp(sharp_level + (0.5 * torch.log(power(flat_level) / power(sharp_level))) @ v.band_weight)
|
| 183 |
+
angle = sharp @ self.harmonic_imaginary
|
| 184 |
+
harmonic_real, harmonic_imaginary = magnitude * torch.cos(angle), magnitude * torch.sin(angle)
|
| 185 |
+
else:
|
| 186 |
+
harmonic_real, harmonic_imaginary = self._response(envelope, self.harmonic_real, self.harmonic_imaginary)
|
| 187 |
+
# Noise branch.
|
| 188 |
+
n = v.noise(hidden)
|
| 189 |
+
steps = v.noise_steps
|
| 190 |
+
sub = torch.minimum(torch.arange(steps * total + 3, device=hidden.device), steps * total - 1)
|
| 191 |
+
shape = n[..., :c.noise_cep].index_select(1, torch.div(sub, steps, rounding_mode="floor")) * v.noise_lifter
|
| 192 |
+
gain = n[..., c.noise_cep:].flatten(1).index_select(1, sub)[..., None]
|
| 193 |
+
shape = torch.cat((shape[..., :1] + gain, shape[..., 1:]), dim=-1)
|
| 194 |
+
noise_real, noise_imaginary = self._response(shape, self.noise_real, self.noise_imaginary)
|
| 195 |
+
pulses = self._pulses(f0)
|
| 196 |
+
size = torch.arange(c.hop * total, device=hidden.device)
|
| 197 |
+
noise = torch.randn_like(pulses) if noise is None else noise.index_select(1, size)
|
| 198 |
+
hiss = self._filter(noise, noise_real, noise_imaginary, "noise", c.noise_fft, c.noise_hop)
|
| 199 |
+
if p is None:
|
| 200 |
+
return self._filter(pulses, harmonic_real, harmonic_imaginary, "synth", c.synth_fft, c.synth_hop) + hiss
|
| 201 |
+
# The mixed source: pulses below each sub-frame's limit, noise of the same power above it.
|
| 202 |
+
breath = torch.randn_like(pulses) if breath is None else breath.index_select(1, size)
|
| 203 |
+
limit = limits.index_select(1, here)
|
| 204 |
+
width = torch.where(limit > 2000, torch.full_like(limit, 350.0), torch.full_like(limit, 120.0))
|
| 205 |
+
keep = torch.sigmoid((limit[:, None] - v.synth_frequency[None, :, None]) / width[:, None])
|
| 206 |
+
keep = F.avg_pool1d(F.pad(keep, (1, 1), mode="replicate"), 3, 1)
|
| 207 |
+
other = torch.sqrt((1 - keep ** 2).clamp_min(0)) * self.rumble_cut
|
| 208 |
+
p_real, p_imaginary = self._analysis(pulses, "synth", c.synth_fft, c.synth_hop)
|
| 209 |
+
b_real, b_imaginary = self._analysis(breath * math.sqrt(0.5 / c.harmonic_limit), "synth", c.synth_fft, c.synth_hop)
|
| 210 |
+
harmonic = self._synthesis(p_real * keep + b_real * other, p_imaginary * keep + b_imaginary * other, harmonic_real, harmonic_imaginary,
|
| 211 |
+
"synth", c.synth_fft, c.synth_hop, pulses.shape[1])
|
| 212 |
+
if p.noise_cap is not None:
|
| 213 |
+
level = lambda wave: wave.reshape(1, -1, c.hop).pow(2).mean(-1)
|
| 214 |
+
quieter = (level(harmonic) * 10 ** (p.noise_cap / 10) / level(hiss).clamp_min(1e-20)).clamp(max=1.0).sqrt()
|
| 215 |
+
quieter = torch.where(vowel, quieter, torch.ones_like(quieter))
|
| 216 |
+
quieter = F.avg_pool1d(F.pad(quieter[:, None], (1, 1), mode="replicate"), 3, 1)
|
| 217 |
+
hiss = hiss * F.interpolate(quieter, scale_factor=float(c.hop), mode="linear", align_corners=False)[:, 0]
|
| 218 |
+
wave = harmonic + hiss
|
| 219 |
+
if p.treble:
|
| 220 |
+
wave = F.conv1d(wave[:, None], self.treble, padding=self.treble.shape[-1] // 2)[:, 0]
|
| 221 |
+
return wave
|
| 222 |
+
|
| 223 |
+
def forward(self, tokens, weight, voice, speed, pitch_shift, pitch_range, noise=None, breath=None):
|
| 224 |
+
m = self.model
|
| 225 |
+
c = m.config
|
| 226 |
+
p = self.polish
|
| 227 |
+
token_mask = torch.ones_like(tokens, dtype=torch.bool)
|
| 228 |
+
hidden = m.encoder(m.tokens(tokens) + m.syllable_weight(weight) + m.voices(voice)[:, None], token_mask)
|
| 229 |
+
# Durations.
|
| 230 |
+
output = m.duration(hidden, token_mask)
|
| 231 |
+
frames = torch.round((torch.exp(output[..., 0].clamp(0, math.log1p(400))) - 1) / speed).long().clamp_min(1)
|
| 232 |
+
present = m.phone_tokens[tokens] | (output[..., 1] > 0)
|
| 233 |
+
durations = torch.where(present, frames, torch.zeros_like(frames))
|
| 234 |
+
# Repeat token states by duration.
|
| 235 |
+
ends = durations.cumsum(1)
|
| 236 |
+
total = ends[0, -1]
|
| 237 |
+
frame = torch.arange(total, device=tokens.device)[None]
|
| 238 |
+
index = (ends[:, None, :] <= frame[:, :, None]).sum(-1)
|
| 239 |
+
owned = durations.gather(1, index).clamp_min(1)
|
| 240 |
+
offset = frame - (ends.gather(1, index) - durations.gather(1, index))
|
| 241 |
+
relative = (offset.to(hidden.dtype) + 0.5) / owned
|
| 242 |
+
features = torch.stack((relative, 1 - relative, torch.log1p(owned.to(hidden.dtype)) / 4), dim=-1)
|
| 243 |
+
expanded = hidden.gather(1, index[..., None].expand(-1, -1, hidden.shape[-1]))
|
| 244 |
+
expanded = expanded + m.frame_position(features) + m.frame_voices(voice)[:, None]
|
| 245 |
+
frame_mask = torch.ones_like(frame, dtype=torch.bool)
|
| 246 |
+
# Pitch, voicing, energy.
|
| 247 |
+
prosody_hidden = m.prosody_stack(expanded, frame_mask)
|
| 248 |
+
prosody = m.prosody_head(prosody_hidden)
|
| 249 |
+
predicted = prosody[..., 0]
|
| 250 |
+
if p is not None and p.pitch_smooth:
|
| 251 |
+
predicted = F.conv1d(F.pad(predicted[:, None], (p.pitch_smooth, p.pitch_smooth), mode="replicate"), self.pitch_kernel)[:, 0]
|
| 252 |
+
vowel = m.vowel_tokens[tokens].gather(1, index)
|
| 253 |
+
voicing = torch.where(vowel, prosody[..., 1].clamp_min(2.0), prosody[..., 1])
|
| 254 |
+
voiced = (voicing > 0).to(prosody.dtype)
|
| 255 |
+
mean = (predicted * voiced).sum(1, keepdim=True) / voiced.sum(1, keepdim=True).clamp_min(1)
|
| 256 |
+
pitch = mean + (predicted - mean) * pitch_range
|
| 257 |
+
condition = torch.stack((pitch.clamp(-2.0, 1.5), voiced, prosody[..., 2].clamp(-4.5, 3.0)), dim=-1)
|
| 258 |
+
acoustic = m.acoustic_stack(prosody_hidden + m.prosody_projection(condition), frame_mask)
|
| 259 |
+
f0 = (c.pitch_reference_hz * torch.exp(pitch + pitch_shift)).clamp(c.f0_min, c.f0_max)
|
| 260 |
+
if p is None:
|
| 261 |
+
return self._vocoder(acoustic, f0, total, noise), durations
|
| 262 |
+
# A frequency limit per frame, from the sound it belongs to (see ``Polish``).
|
| 263 |
+
kind = m.sound_kind[tokens].gather(1, index)
|
| 264 |
+
is_voiced = voicing > 0
|
| 265 |
+
fill = lambda value: torch.full_like(pitch, value)
|
| 266 |
+
limits = torch.where(kind == 1, fill(p.open_limit), fill(ALL_PULSES))
|
| 267 |
+
limits = torch.where(kind == 2, fill(p.consonant_limit), limits)
|
| 268 |
+
limits = torch.where(kind == 3, fill(NO_PULSES), limits)
|
| 269 |
+
limits = torch.where(kind == 4, torch.where(is_voiced, fill(p.consonant_limit), fill(NO_PULSES)), limits)
|
| 270 |
+
limits = torch.where((kind == 5) & ~is_voiced, fill(NO_PULSES), limits)
|
| 271 |
+
return self._vocoder(acoustic, f0, total, noise, breath, limits, kind == 1, vowel), durations
|
| 272 |
+
|
| 273 |
+
|
| 274 |
+
def example_inputs(length=24):
|
| 275 |
+
tokens = torch.randint(5, len(SYMBOLS), (1, length))
|
| 276 |
+
one = lambda value: torch.tensor([value], dtype=torch.float32)
|
| 277 |
+
return tokens, torch.zeros_like(tokens), torch.tensor([0]), one(1.0), one(0.0), one(1.0)
|
| 278 |
+
|
| 279 |
+
|
| 280 |
+
def export(model, path, external_noise=False, opset=17, polish=Polish()):
|
| 281 |
+
wrapper = Exportable(model, external_noise, polish).eval()
|
| 282 |
+
inputs = example_inputs()
|
| 283 |
+
names = list(INPUTS)
|
| 284 |
+
axes = {"tokens": {1: "tokens"}, "weight": {1: "tokens"}, "audio": {1: "samples"}, "durations": {1: "tokens"}}
|
| 285 |
+
if external_noise:
|
| 286 |
+
for name in ("noise",) if polish is None else ("noise", "breath"):
|
| 287 |
+
inputs = (*inputs, torch.randn(1, model.config.hop * model.config.max_frames))
|
| 288 |
+
names.append(name)
|
| 289 |
+
axes[name] = {1: f"{name}_samples"}
|
| 290 |
+
with torch.no_grad():
|
| 291 |
+
torch.onnx.export(wrapper, inputs, str(path), input_names=names, output_names=list(OUTPUTS), dynamic_axes=axes,
|
| 292 |
+
opset_version=opset, do_constant_folding=True, dynamo=False)
|
| 293 |
+
return path
|
| 294 |
+
|
| 295 |
+
|
| 296 |
+
def metadata(model, checkpoint, polish=Polish()):
|
| 297 |
+
c = model.config
|
| 298 |
+
state = torch.load(checkpoint, map_location="cpu", weights_only=True)
|
| 299 |
+
return {"sample_rate": c.sample_rate, "hop": c.hop, "voices": list(c.speakers), "symbols": list(SYMBOLS),
|
| 300 |
+
"max_tokens": c.max_tokens, "max_frames": c.max_frames, "step": state.get("step"),
|
| 301 |
+
"frontend_fingerprint": frontend_fingerprint(), "inputs": list(INPUTS), "outputs": list(OUTPUTS),
|
| 302 |
+
"polish": None if polish is None else dataclasses.asdict(polish)}
|
| 303 |
+
|
| 304 |
+
|
| 305 |
+
def main():
|
| 306 |
+
parser = argparse.ArgumentParser(description="Export an Alpa inference checkpoint to ONNX")
|
| 307 |
+
parser.add_argument("--checkpoint", type=Path, required=True)
|
| 308 |
+
parser.add_argument("--output", type=Path, required=True)
|
| 309 |
+
parser.add_argument("--plain", action="store_true", help="Leave out the corrections made while speaking (the sound of v0.1 as first published)")
|
| 310 |
+
parser.add_argument("--external-noise", action="store_true", help="Take the noise signals as inputs (for exact comparisons)")
|
| 311 |
+
parser.add_argument("--opset", type=int, default=17)
|
| 312 |
+
args = parser.parse_args()
|
| 313 |
+
model = load_model(args.checkpoint, "cpu")
|
| 314 |
+
polish = None if args.plain else Polish()
|
| 315 |
+
args.output.parent.mkdir(parents=True, exist_ok=True)
|
| 316 |
+
export(model, args.output, args.external_noise, args.opset, polish)
|
| 317 |
+
args.output.with_suffix(".json").write_text(json.dumps(metadata(model, args.checkpoint, polish), ensure_ascii=False, indent=1) + "\n", encoding="utf-8")
|
| 318 |
+
print(f"wrote {args.output} ({args.output.stat().st_size / 1e6:.1f} MB) and {args.output.with_suffix('.json').name}")
|
| 319 |
+
|
| 320 |
+
|
| 321 |
+
if __name__ == "__main__":
|
| 322 |
+
main()
|
alpa/infer.py
CHANGED
|
@@ -1,12 +1,13 @@
|
|
| 1 |
# Adapted from tiny-voca by Muhammed Nazeem, commit 28e0722 (Apache-2.0; see LICENSES/Apache-2.0.txt
|
| 2 |
# and THIRD_PARTY_NOTICES.md). File: src/tiny_voca/infer.py. Changes for Alpa: Alpa's model and syllable weight; one
|
| 3 |
-
# front end, so the options for choosing between two are removed; the voice is checked against the checkpoint's own list
|
|
|
|
| 4 |
|
| 5 |
import argparse
|
| 6 |
from pathlib import Path
|
| 7 |
import torch
|
| 8 |
from .config import CHECKPOINT_FORMAT, Config
|
| 9 |
-
from .model import Alpa
|
| 10 |
from .text import frontend_fingerprint, phonemize
|
| 11 |
|
| 12 |
|
|
@@ -21,8 +22,9 @@ def load_model(path, device="cpu"):
|
|
| 21 |
return model.to(device).eval()
|
| 22 |
|
| 23 |
|
| 24 |
-
def speak(model, text, voice, speed=1.0, pitch_shift=0.0, seed=0, pitch_range=1.0):
|
| 25 |
-
"""Text -> (float waveform tensor, phonemized text). Same front end as training.
|
|
|
|
| 26 |
device = next(model.parameters()).device
|
| 27 |
phones = phonemize(text)
|
| 28 |
if len(phones) > model.config.max_tokens:
|
|
@@ -32,7 +34,8 @@ def speak(model, text, voice, speed=1.0, pitch_shift=0.0, seed=0, pitch_range=1.
|
|
| 32 |
if voice not in model.config.speakers:
|
| 33 |
raise ValueError(f"unknown voice {voice!r}; this model has {model.config.speakers}")
|
| 34 |
speaker = torch.tensor([model.config.speakers.index(voice)], device=device)
|
| 35 |
-
audio, lengths, details = model.synthesize(tokens, weight, torch.tensor([len(phones)], device=device), speaker, speed, pitch_shift, seed, pitch_range
|
|
|
|
| 36 |
if not torch.isfinite(audio).all():
|
| 37 |
raise RuntimeError("model produced non-finite audio")
|
| 38 |
return audio[0, :int(lengths[0])].cpu(), phones, details
|
|
@@ -47,11 +50,13 @@ def main():
|
|
| 47 |
parser.add_argument("--speed", type=float, default=1)
|
| 48 |
parser.add_argument("--pitch-shift", type=float, default=0, help="Added to log F0 (0.058 is about one semitone)")
|
| 49 |
parser.add_argument("--pitch-range", type=float, default=1, help="Scale pitch movement around the utterance mean (1 = as predicted)")
|
|
|
|
| 50 |
parser.add_argument("--device", default="cpu")
|
| 51 |
args = parser.parse_args()
|
| 52 |
import soundfile as sf
|
| 53 |
model = load_model(args.checkpoint, args.device)
|
| 54 |
-
wave, phones, _ = speak(model, args.text, args.voice or model.config.speakers[0], args.speed, args.pitch_shift, pitch_range=args.pitch_range
|
|
|
|
| 55 |
args.output.parent.mkdir(parents=True, exist_ok=True)
|
| 56 |
# Floating WAV preserves output for inspection; no silent clipping or normalization.
|
| 57 |
sf.write(args.output, wave.numpy(), model.config.sample_rate, subtype="FLOAT")
|
|
|
|
| 1 |
# Adapted from tiny-voca by Muhammed Nazeem, commit 28e0722 (Apache-2.0; see LICENSES/Apache-2.0.txt
|
| 2 |
# and THIRD_PARTY_NOTICES.md). File: src/tiny_voca/infer.py. Changes for Alpa: Alpa's model and syllable weight; one
|
| 3 |
+
# front end, so the options for choosing between two are removed; the voice is checked against the checkpoint's own list;
|
| 4 |
+
# speech is polished (model.Polish) unless asked otherwise.
|
| 5 |
|
| 6 |
import argparse
|
| 7 |
from pathlib import Path
|
| 8 |
import torch
|
| 9 |
from .config import CHECKPOINT_FORMAT, Config
|
| 10 |
+
from .model import Alpa, Polish
|
| 11 |
from .text import frontend_fingerprint, phonemize
|
| 12 |
|
| 13 |
|
|
|
|
| 22 |
return model.to(device).eval()
|
| 23 |
|
| 24 |
|
| 25 |
+
def speak(model, text, voice, speed=1.0, pitch_shift=0.0, seed=0, pitch_range=1.0, polish=Polish()):
|
| 26 |
+
"""Text -> (float waveform tensor, phonemized text). Same front end as training. ``polish=None`` gives the
|
| 27 |
+
sound exactly as the network was trained to make it."""
|
| 28 |
device = next(model.parameters()).device
|
| 29 |
phones = phonemize(text)
|
| 30 |
if len(phones) > model.config.max_tokens:
|
|
|
|
| 34 |
if voice not in model.config.speakers:
|
| 35 |
raise ValueError(f"unknown voice {voice!r}; this model has {model.config.speakers}")
|
| 36 |
speaker = torch.tensor([model.config.speakers.index(voice)], device=device)
|
| 37 |
+
audio, lengths, details = model.synthesize(tokens, weight, torch.tensor([len(phones)], device=device), speaker, speed, pitch_shift, seed, pitch_range,
|
| 38 |
+
polish=polish)
|
| 39 |
if not torch.isfinite(audio).all():
|
| 40 |
raise RuntimeError("model produced non-finite audio")
|
| 41 |
return audio[0, :int(lengths[0])].cpu(), phones, details
|
|
|
|
| 50 |
parser.add_argument("--speed", type=float, default=1)
|
| 51 |
parser.add_argument("--pitch-shift", type=float, default=0, help="Added to log F0 (0.058 is about one semitone)")
|
| 52 |
parser.add_argument("--pitch-range", type=float, default=1, help="Scale pitch movement around the utterance mean (1 = as predicted)")
|
| 53 |
+
parser.add_argument("--plain", action="store_true", help="Leave out the corrections made while speaking (the sound of v0.1 as first published)")
|
| 54 |
parser.add_argument("--device", default="cpu")
|
| 55 |
args = parser.parse_args()
|
| 56 |
import soundfile as sf
|
| 57 |
model = load_model(args.checkpoint, args.device)
|
| 58 |
+
wave, phones, _ = speak(model, args.text, args.voice or model.config.speakers[0], args.speed, args.pitch_shift, pitch_range=args.pitch_range,
|
| 59 |
+
polish=None if args.plain else Polish())
|
| 60 |
args.output.parent.mkdir(parents=True, exist_ok=True)
|
| 61 |
# Floating WAV preserves output for inspection; no silent clipping or normalization.
|
| 62 |
sf.write(args.output, wave.numpy(), model.config.sample_rate, subtype="FLOAT")
|
alpa/model.py
CHANGED
|
@@ -1,6 +1,7 @@
|
|
| 1 |
# Adapted from tiny-voca by Muhammed Nazeem, commit 28e0722 (Apache-2.0; see LICENSES/Apache-2.0.txt
|
| 2 |
# and THIRD_PARTY_NOTICES.md). File: src/tiny_voca/model.py. Changes for Alpa: the model is named Alpa; the per-token
|
| 3 |
-
# stress input is the syllable weight (0 none, 1 laghu, 2 guru); the docstring and one comment are reworded
|
|
|
|
| 4 |
|
| 5 |
"""Alpa: feed-forward transformer with a pitch-driven source-filter head.
|
| 6 |
|
|
@@ -21,6 +22,8 @@ is what keeps the voice from sounding buzzy or unstable at this size.
|
|
| 21 |
"""
|
| 22 |
|
| 23 |
import math
|
|
|
|
|
|
|
| 24 |
import torch
|
| 25 |
from torch import nn
|
| 26 |
from torch.nn import functional as F
|
|
@@ -162,6 +165,61 @@ def cepstrum_response(causal, anticausal, fft):
|
|
| 162 |
return torch.polar(torch.exp(spectrum.real.clamp(-14, 6)), spectrum.imag)
|
| 163 |
|
| 164 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 165 |
class SourceFilter(nn.Module):
|
| 166 |
"""Harmonic-plus-noise synthesis controlled per frame by two linear heads."""
|
| 167 |
|
|
@@ -186,6 +244,19 @@ class SourceFilter(nn.Module):
|
|
| 186 |
self.register_buffer("noise_lifter", lifter(c.noise_cep, 0), persistent=False)
|
| 187 |
self.register_buffer("synth_window", torch.hann_window(c.synth_fft), persistent=False)
|
| 188 |
self.register_buffer("noise_window", torch.hann_window(c.noise_fft), persistent=False)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 189 |
|
| 190 |
@staticmethod
|
| 191 |
def _take(values, index):
|
|
@@ -204,7 +275,7 @@ class SourceFilter(nn.Module):
|
|
| 204 |
# then the same whether it stands alone or is padded inside a batch.
|
| 205 |
return filtered(F.pad(source, (0, fft // 2)), response, fft, hop, window)[:, :source.shape[-1]]
|
| 206 |
|
| 207 |
-
def forward(self, hidden, f0, lengths, noise=None, phase=None):
|
| 208 |
c = self.config
|
| 209 |
frames = hidden.shape[1]
|
| 210 |
samples = frames * c.hop
|
|
@@ -233,10 +304,58 @@ class SourceFilter(nn.Module):
|
|
| 233 |
pulses = pulse_train(f0, c.sample_rate, c.hop, c.harmonic_limit, phase) * audible
|
| 234 |
if noise is None:
|
| 235 |
noise = torch.randn(hidden.shape[0], samples, device=hidden.device)
|
|
|
|
|
|
|
| 236 |
wave = (self._shape(pulses, harmonic_response, c.synth_fft, c.synth_hop, self.synth_window)
|
| 237 |
+ self._shape(noise * audible, noise_response, c.noise_fft, c.noise_hop, self.noise_window))
|
| 238 |
return wave * audible
|
| 239 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 240 |
|
| 241 |
class Alpa(nn.Module):
|
| 242 |
def __init__(self, config=Config()):
|
|
@@ -264,6 +383,10 @@ class Alpa(nn.Module):
|
|
| 264 |
vowels = torch.zeros(c.vocab_size, dtype=torch.bool)
|
| 265 |
vowels[[SYMBOL_TO_ID[v] for v in VOWELS]] = True
|
| 266 |
self.register_buffer("vowel_tokens", vowels, persistent=False)
|
|
|
|
|
|
|
|
|
|
|
|
|
| 267 |
|
| 268 |
def encode(self, tokens, weight, lengths, speakers):
|
| 269 |
if tokens.ndim != 2 or weight.shape != tokens.shape or lengths.shape != (tokens.shape[0],) or speakers.shape != lengths.shape:
|
|
@@ -344,16 +467,50 @@ class Alpa(nn.Module):
|
|
| 344 |
slow = slow / slow.std().clamp_min(1e-6)
|
| 345 |
return ((2 * slow + fast) / 5 ** 0.5 * semitones * math.log(2) / 12).to(device)
|
| 346 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 347 |
@torch.inference_mode()
|
| 348 |
def synthesize(self, tokens, weight, lengths, speakers, speed=1.0, pitch_shift=0.0, seed=0, pitch_range=1.0, pitch_jitter=0.0,
|
| 349 |
-
voiced_vowels=True):
|
| 350 |
-
"""Free-running synthesis: predicted durations, pitch, voicing and energy.
|
|
|
|
| 351 |
if not math.isfinite(speed) or not 0.25 <= speed <= 4 or not math.isfinite(pitch_shift) or not 0 <= pitch_range <= 3:
|
| 352 |
raise ValueError("speed must be between 0.25 and 4, pitch_shift finite and pitch_range between 0 and 3")
|
| 353 |
hidden, mask = self.encode(tokens, weight, lengths, speakers)
|
| 354 |
durations = self.predict_durations(hidden, mask, tokens, speed)
|
| 355 |
frames, frame_mask, frame_lengths = self.expand(hidden, durations, mask, speakers)
|
| 356 |
prosody_hidden, prosody = self.predict_prosody(frames, frame_mask)
|
|
|
|
|
|
|
| 357 |
if voiced_vowels:
|
| 358 |
# A pitch tracker can miss creaky vowels in the recordings, and the
|
| 359 |
# voicing predictor then learns to drop whole vowels (heard as a whispered
|
|
@@ -373,5 +530,6 @@ class Alpa(nn.Module):
|
|
| 373 |
pitch = prosody[..., 0] + pitch_shift
|
| 374 |
if pitch_jitter > 0:
|
| 375 |
pitch = pitch + self.pitch_wobble(pitch.shape, pitch_jitter, seed, pitch.device)
|
| 376 |
-
|
|
|
|
| 377 |
return wave, frame_lengths * self.config.hop, {"durations": durations, "prosody": prosody, "frame_mask": frame_mask}
|
|
|
|
| 1 |
# Adapted from tiny-voca by Muhammed Nazeem, commit 28e0722 (Apache-2.0; see LICENSES/Apache-2.0.txt
|
| 2 |
# and THIRD_PARTY_NOTICES.md). File: src/tiny_voca/model.py. Changes for Alpa: the model is named Alpa; the per-token
|
| 3 |
+
# stress input is the syllable weight (0 none, 1 laghu, 2 guru); the docstring and one comment are reworded; ``Polish``,
|
| 4 |
+
# ``Shaping`` and the code that applies them when speaking are Alpa's own.
|
| 5 |
|
| 6 |
"""Alpa: feed-forward transformer with a pitch-driven source-filter head.
|
| 7 |
|
|
|
|
| 22 |
"""
|
| 23 |
|
| 24 |
import math
|
| 25 |
+
from dataclasses import dataclass
|
| 26 |
+
from typing import NamedTuple
|
| 27 |
import torch
|
| 28 |
from torch import nn
|
| 29 |
from torch.nn import functional as F
|
|
|
|
| 165 |
return torch.polar(torch.exp(spectrum.real.clamp(-14, 6)), spectrum.imag)
|
| 166 |
|
| 167 |
|
| 168 |
+
# Sounds by how they are voiced, for ``Polish``: vowels, nasals and semivowels; stops and ज; sibilants and च.
|
| 169 |
+
OPEN = VOWELS + tuple("NYRnmyrlvM")
|
| 170 |
+
STOPS = tuple("kgjwqtdpb")
|
| 171 |
+
HISS = tuple("cSzs")
|
| 172 |
+
NO_PULSES, ALL_PULSES = -1e5, 1e6 # frequency limits (Hz) that mean "noise everywhere" and "pulses everywhere"
|
| 173 |
+
|
| 174 |
+
|
| 175 |
+
@dataclass(frozen=True)
|
| 176 |
+
class Polish:
|
| 177 |
+
"""Corrections made while speaking; the weights are not touched.
|
| 178 |
+
|
| 179 |
+
The harmonic branch was trained with a pulse train as its only source, and the network learnt to build hiss,
|
| 180 |
+
breath and bursts out of those pulses as well, which a listener hears as a second, metallic voice inside the
|
| 181 |
+
consonants. Here every frame gets a frequency limit from the sound it belongs to. Below the limit the branch
|
| 182 |
+
is fed the pulses, above it white noise of the same power:
|
| 183 |
+
|
| 184 |
+
* sibilants and च: noise at every frequency;
|
| 185 |
+
* stops and ज: pulses below ``consonant_limit`` only; ह the same where the frame is voiced, else noise;
|
| 186 |
+
* visarga: pulses where the frame is voiced, else noise;
|
| 187 |
+
* vowels, nasals and semivowels: pulses up to ``open_limit``, above which a real voice is breath.
|
| 188 |
+
|
| 189 |
+
The other settings: the predicted pitch is averaged over ``pitch_smooth`` frames either side (triangular
|
| 190 |
+
weights); in vowels, nasals and semivowels the peaks and valleys of the spectral envelope are stretched by
|
| 191 |
+
``sharpen`` (cepstral coefficients ``sharpen_from`` up to ``sharpen_to``, with the loudness of each broad
|
| 192 |
+
band put back); in vowels the noise branch is held ``noise_cap`` dB under the harmonic branch; and the top
|
| 193 |
+
of the spectrum is lifted by ``treble`` dB (from 2 kHz, in full from 5 kHz). The values were chosen against
|
| 194 |
+
the recordings and by ear: docs/runs, runs 20 to 22."""
|
| 195 |
+
pitch_smooth: int = 3
|
| 196 |
+
consonant_limit: float = 1000.0
|
| 197 |
+
open_limit: float = 4500.0
|
| 198 |
+
sharpen: float = 1.6
|
| 199 |
+
sharpen_from: int = 8
|
| 200 |
+
sharpen_to: int = 40
|
| 201 |
+
noise_cap: float = -15.0
|
| 202 |
+
treble: float = 4.0
|
| 203 |
+
|
| 204 |
+
|
| 205 |
+
class Shaping(NamedTuple):
|
| 206 |
+
"""What ``SourceFilter`` needs to apply a ``Polish``: per-frame controls and the replacement noise."""
|
| 207 |
+
polish: Polish
|
| 208 |
+
limits: torch.Tensor # [B, frames] Hz: pulses below, noise above
|
| 209 |
+
open: torch.Tensor # [B, frames] bool: vowel, nasal or semivowel
|
| 210 |
+
vowel: torch.Tensor # [B, frames] bool
|
| 211 |
+
breath: torch.Tensor # [B, samples] white noise, unit variance
|
| 212 |
+
|
| 213 |
+
|
| 214 |
+
def treble_kernel(gain_db, sample_rate, taps=129):
|
| 215 |
+
"""Zero-phase filter that lifts the top of the spectrum: nothing below 2 kHz, ``gain_db`` from 5 kHz up."""
|
| 216 |
+
size = 1024
|
| 217 |
+
frequency = torch.fft.rfftfreq(size, 1 / sample_rate).clamp_min(1.0)
|
| 218 |
+
rise = (torch.log2(frequency / 2000.0) / math.log2(2.5)).clamp(0, 1)
|
| 219 |
+
kernel = torch.fft.irfft(10 ** (gain_db * rise / 20), n=size).roll(taps // 2)[:taps]
|
| 220 |
+
return kernel * torch.hann_window(taps, periodic=False)
|
| 221 |
+
|
| 222 |
+
|
| 223 |
class SourceFilter(nn.Module):
|
| 224 |
"""Harmonic-plus-noise synthesis controlled per frame by two linear heads."""
|
| 225 |
|
|
|
|
| 244 |
self.register_buffer("noise_lifter", lifter(c.noise_cep, 0), persistent=False)
|
| 245 |
self.register_buffer("synth_window", torch.hann_window(c.synth_fft), persistent=False)
|
| 246 |
self.register_buffer("noise_window", torch.hann_window(c.noise_fft), persistent=False)
|
| 247 |
+
# For ``Polish``: four broad bands of the harmonic filter, and weights that turn one gain per band into a
|
| 248 |
+
# smooth gain per frequency (straight lines between the band centres, level outside them).
|
| 249 |
+
frequency = torch.arange(c.synth_fft // 2 + 1) * (c.sample_rate / c.synth_fft)
|
| 250 |
+
edges, centres = (0.0, 1000.0, 2500.0, 5000.0, float("inf")), torch.tensor([500.0, 1750.0, 3750.0, 8500.0])
|
| 251 |
+
self.register_buffer("band_mask", torch.stack([((frequency >= lo) & (frequency < hi)).float() for lo, hi in zip(edges, edges[1:])]), persistent=False)
|
| 252 |
+
inside = frequency.clamp(float(centres[0]), float(centres[-1]))
|
| 253 |
+
upper = torch.bucketize(inside, centres).clamp(1, len(centres) - 1)
|
| 254 |
+
share = (inside - centres[upper - 1]) / (centres[upper] - centres[upper - 1])
|
| 255 |
+
weight = torch.zeros(len(centres), len(frequency))
|
| 256 |
+
weight[upper - 1, torch.arange(len(frequency))] = 1 - share
|
| 257 |
+
weight[upper, torch.arange(len(frequency))] += share
|
| 258 |
+
self.register_buffer("band_weight", weight, persistent=False)
|
| 259 |
+
self.register_buffer("synth_frequency", frequency, persistent=False)
|
| 260 |
|
| 261 |
@staticmethod
|
| 262 |
def _take(values, index):
|
|
|
|
| 275 |
# then the same whether it stands alone or is padded inside a batch.
|
| 276 |
return filtered(F.pad(source, (0, fft // 2)), response, fft, hop, window)[:, :source.shape[-1]]
|
| 277 |
|
| 278 |
+
def forward(self, hidden, f0, lengths, noise=None, phase=None, shaping=None):
|
| 279 |
c = self.config
|
| 280 |
frames = hidden.shape[1]
|
| 281 |
samples = frames * c.hop
|
|
|
|
| 304 |
pulses = pulse_train(f0, c.sample_rate, c.hop, c.harmonic_limit, phase) * audible
|
| 305 |
if noise is None:
|
| 306 |
noise = torch.randn(hidden.shape[0], samples, device=hidden.device)
|
| 307 |
+
if shaping is not None:
|
| 308 |
+
return self._polished(shaping, causal, anti, harmonic_response, here, pulses, noise * audible, noise_response, last, audible)
|
| 309 |
wave = (self._shape(pulses, harmonic_response, c.synth_fft, c.synth_hop, self.synth_window)
|
| 310 |
+ self._shape(noise * audible, noise_response, c.noise_fft, c.noise_hop, self.noise_window))
|
| 311 |
return wave * audible
|
| 312 |
|
| 313 |
+
def _polished(self, shaping, causal, anti, response, here, pulses, noise, noise_response, last, audible):
|
| 314 |
+
"""The same synthesis with a ``Polish`` applied. ``here`` is the frame of each harmonic sub-frame."""
|
| 315 |
+
c, p = self.config, shaping.polish
|
| 316 |
+
frames = shaping.limits.shape[1]
|
| 317 |
+
samples = frames * c.hop
|
| 318 |
+
frequency = self.synth_frequency
|
| 319 |
+
if p.sharpen != 1.0:
|
| 320 |
+
# Stretch the envelope's detail by ``beta`` (in dB) where the frame is a vowel, nasal or semivowel. The lowest
|
| 321 |
+
# coefficients (loudness and overall slope) and the highest (finer than a formant) are left alone.
|
| 322 |
+
beta = (1.0 + (p.sharpen - 1.0) * shaping.open.float()).gather(1, here)[..., None]
|
| 323 |
+
chosen = torch.zeros(c.cep_causal, device=causal.device)
|
| 324 |
+
chosen[p.sharpen_from:p.sharpen_to] = 1
|
| 325 |
+
mirror = torch.zeros(c.cep_anti, device=causal.device)
|
| 326 |
+
mirror[max(p.sharpen_from - 1, 0):max(p.sharpen_to - 1, 0)] = 1
|
| 327 |
+
sharp = cepstrum_response(causal * (1 + chosen * (beta - 1)), anti * (1 + mirror * (beta - 1)), c.synth_fft)
|
| 328 |
+
# Sharper peaks would move loudness between broad bands; give each band back what the network predicted.
|
| 329 |
+
power = lambda r: r.abs().pow(2) @ self.band_mask.T + 1e-20
|
| 330 |
+
response = sharp * torch.exp(0.5 * torch.log(power(response) / power(sharp)) @ self.band_weight)
|
| 331 |
+
# The mixed source, built in the short-time frames the filter works in: pulses below each sub-frame's
|
| 332 |
+
# limit, noise above it. The switch is 120 Hz wide for a consonant's limit, 350 Hz for the higher one of open
|
| 333 |
+
# sounds, and is softened over three sub-frames in time.
|
| 334 |
+
limit = shaping.limits.gather(1, here)
|
| 335 |
+
keep = torch.sigmoid((limit[:, None] - frequency[None, :, None]) / torch.where(limit > 2000, 350.0, 120.0)[:, None])
|
| 336 |
+
keep = F.avg_pool1d(F.pad(keep, (1, 1), mode="replicate"), 3, 1)
|
| 337 |
+
fft, hop, window = c.synth_fft, c.synth_hop, self.synth_window
|
| 338 |
+
spectrum = lambda source: torch.stft(F.pad(source, (0, fft // 2)), fft, hop, window=window, center=True, pad_mode="constant", return_complex=True)
|
| 339 |
+
# White noise with the pulse train's power per frequency. The pulses carry nothing below the pitch, so the
|
| 340 |
+
# noise that replaces them is cut below 90 Hz, or it would rumble.
|
| 341 |
+
breath = spectrum(shaping.breath * audible * math.sqrt(0.5 / c.harmonic_limit)) * torch.sigmoid((frequency - 90.0) / 12.0)[:, None]
|
| 342 |
+
mixed = spectrum(pulses) * keep + breath * torch.sqrt((1 - keep ** 2).clamp_min(0))
|
| 343 |
+
harmonic = torch.istft(mixed * response.transpose(1, 2), fft, hop, window=window, center=True, length=samples + fft // 2)[:, :samples]
|
| 344 |
+
hiss = self._shape(noise, noise_response, c.noise_fft, c.noise_hop, self.noise_window)
|
| 345 |
+
if p.noise_cap is not None:
|
| 346 |
+
# In a vowel the noise branch sometimes whispers the vowel as loudly as the harmonic branch voices it.
|
| 347 |
+
level = lambda wave: wave.unflatten(1, (frames, c.hop)).pow(2).mean(-1)
|
| 348 |
+
gain = (level(harmonic) * 10 ** (p.noise_cap / 10) / level(hiss).clamp_min(1e-20)).clamp(max=1.0).sqrt()
|
| 349 |
+
gain = torch.where(shaping.vowel, gain, torch.ones_like(gain))
|
| 350 |
+
gain = gain.gather(1, torch.minimum(torch.arange(frames, device=gain.device)[None], last)) # padding repeats the last frame
|
| 351 |
+
gain = F.avg_pool1d(F.pad(gain[:, None], (1, 1), mode="replicate"), 3, 1)
|
| 352 |
+
hiss = hiss * F.interpolate(gain, size=samples, mode="linear", align_corners=False)[:, 0]
|
| 353 |
+
wave = (harmonic + hiss) * audible
|
| 354 |
+
if p.treble:
|
| 355 |
+
kernel = treble_kernel(p.treble, c.sample_rate).to(wave.device)
|
| 356 |
+
wave = F.conv1d(wave[:, None], kernel[None, None], padding=len(kernel) // 2)[:, 0] * audible
|
| 357 |
+
return wave
|
| 358 |
+
|
| 359 |
|
| 360 |
class Alpa(nn.Module):
|
| 361 |
def __init__(self, config=Config()):
|
|
|
|
| 383 |
vowels = torch.zeros(c.vocab_size, dtype=torch.bool)
|
| 384 |
vowels[[SYMBOL_TO_ID[v] for v in VOWELS]] = True
|
| 385 |
self.register_buffer("vowel_tokens", vowels, persistent=False)
|
| 386 |
+
kinds = torch.zeros(c.vocab_size, dtype=torch.long) # 0 pause or mark, then the groups ``Polish`` treats alike
|
| 387 |
+
for kind, symbols in enumerate((OPEN, STOPS, HISS, ("h",), ("H",)), start=1):
|
| 388 |
+
kinds[[SYMBOL_TO_ID[symbol] for symbol in symbols]] = kind
|
| 389 |
+
self.register_buffer("sound_kind", kinds, persistent=False)
|
| 390 |
|
| 391 |
def encode(self, tokens, weight, lengths, speakers):
|
| 392 |
if tokens.ndim != 2 or weight.shape != tokens.shape or lengths.shape != (tokens.shape[0],) or speakers.shape != lengths.shape:
|
|
|
|
| 467 |
slow = slow / slow.std().clamp_min(1e-6)
|
| 468 |
return ((2 * slow + fast) / 5 ** 0.5 * semitones * math.log(2) / 12).to(device)
|
| 469 |
|
| 470 |
+
@staticmethod
|
| 471 |
+
def frame_owner(durations, size):
|
| 472 |
+
"""Index of the token each of ``size`` frames belongs to, [B, size]."""
|
| 473 |
+
frame = torch.arange(size, device=durations.device)[None].expand(durations.shape[0], -1).contiguous()
|
| 474 |
+
return torch.searchsorted(durations.cumsum(1), frame, right=True).clamp(max=durations.shape[1] - 1)
|
| 475 |
+
|
| 476 |
+
@staticmethod
|
| 477 |
+
def steady_pitch(prosody, frame_lengths, frame_mask, reach):
|
| 478 |
+
"""Triangular moving average of the predicted pitch, ``reach`` frames either side. The prediction flickers
|
| 479 |
+
from frame to frame more than a voice does. Each row is extended with its own last value, so padding
|
| 480 |
+
inside a batch never changes it."""
|
| 481 |
+
last = (frame_lengths - 1).clamp_min(0)[:, None]
|
| 482 |
+
pitch = prosody[..., 0].gather(1, torch.minimum(torch.arange(prosody.shape[1], device=prosody.device)[None], last))
|
| 483 |
+
kernel = torch.cat((torch.arange(1, reach + 2), torch.arange(reach, 0, -1))).to(pitch)
|
| 484 |
+
pitch = F.conv1d(F.pad(pitch[:, None], (reach, reach), mode="replicate"), (kernel / kernel.sum())[None, None])[:, 0]
|
| 485 |
+
return torch.cat((pitch[..., None], prosody[..., 1:]), dim=-1) * frame_mask[..., None]
|
| 486 |
+
|
| 487 |
+
def shaping(self, polish, tokens, durations, voiced, frame_lengths, seed):
|
| 488 |
+
"""Per-frame controls for the waveform head, from the sound each frame belongs to (see ``Polish``)."""
|
| 489 |
+
kind = self.sound_kind[tokens].gather(1, self.frame_owner(durations, voiced.shape[1]))
|
| 490 |
+
limits = torch.full(kind.shape, ALL_PULSES, device=kind.device)
|
| 491 |
+
limits = torch.where(kind == 1, polish.open_limit, limits)
|
| 492 |
+
limits = torch.where(kind == 2, polish.consonant_limit, limits)
|
| 493 |
+
limits = torch.where(kind == 3, NO_PULSES, limits)
|
| 494 |
+
limits = torch.where(kind == 4, torch.where(voiced, polish.consonant_limit, NO_PULSES), limits)
|
| 495 |
+
limits = torch.where((kind == 5) & ~voiced, NO_PULSES, limits)
|
| 496 |
+
vowel = self.vowel_tokens[tokens].gather(1, self.frame_owner(durations, voiced.shape[1]))
|
| 497 |
+
breath = nn.utils.rnn.pad_sequence([torch.randn(int(n) * self.config.hop, generator=torch.Generator().manual_seed(seed + 7919 + row))
|
| 498 |
+
for row, n in enumerate(frame_lengths)], batch_first=True)
|
| 499 |
+
return Shaping(polish, limits, kind == 1, vowel, breath.to(limits.device))
|
| 500 |
+
|
| 501 |
@torch.inference_mode()
|
| 502 |
def synthesize(self, tokens, weight, lengths, speakers, speed=1.0, pitch_shift=0.0, seed=0, pitch_range=1.0, pitch_jitter=0.0,
|
| 503 |
+
voiced_vowels=True, polish=None):
|
| 504 |
+
"""Free-running synthesis: predicted durations, pitch, voicing and energy. ``polish`` (a ``Polish``) applies
|
| 505 |
+
the corrections made while speaking; ``None`` gives the sound exactly as the network was trained to make it."""
|
| 506 |
if not math.isfinite(speed) or not 0.25 <= speed <= 4 or not math.isfinite(pitch_shift) or not 0 <= pitch_range <= 3:
|
| 507 |
raise ValueError("speed must be between 0.25 and 4, pitch_shift finite and pitch_range between 0 and 3")
|
| 508 |
hidden, mask = self.encode(tokens, weight, lengths, speakers)
|
| 509 |
durations = self.predict_durations(hidden, mask, tokens, speed)
|
| 510 |
frames, frame_mask, frame_lengths = self.expand(hidden, durations, mask, speakers)
|
| 511 |
prosody_hidden, prosody = self.predict_prosody(frames, frame_mask)
|
| 512 |
+
if polish is not None and polish.pitch_smooth:
|
| 513 |
+
prosody = self.steady_pitch(prosody, frame_lengths, frame_mask, polish.pitch_smooth)
|
| 514 |
if voiced_vowels:
|
| 515 |
# A pitch tracker can miss creaky vowels in the recordings, and the
|
| 516 |
# voicing predictor then learns to drop whole vowels (heard as a whispered
|
|
|
|
| 530 |
pitch = prosody[..., 0] + pitch_shift
|
| 531 |
if pitch_jitter > 0:
|
| 532 |
pitch = pitch + self.pitch_wobble(pitch.shape, pitch_jitter, seed, pitch.device)
|
| 533 |
+
shaping = None if polish is None else self.shaping(polish, tokens, durations, (prosody[..., 1] > 0) & frame_mask, frame_lengths, seed)
|
| 534 |
+
wave = self.vocoder(acoustic, self.frequency(pitch), frame_lengths, noise=noise.to(hidden.device), shaping=shaping)
|
| 535 |
return wave, frame_lengths * self.config.hop, {"durations": durations, "prosody": prosody, "frame_mask": frame_mask}
|
onnx/alpa-v0.1.1.json
ADDED
|
@@ -0,0 +1,83 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"model": "alpa-v0.1.1",
|
| 3 |
+
"sample_rate": 24000,
|
| 4 |
+
"hop": 256,
|
| 5 |
+
"voices": [
|
| 6 |
+
"female",
|
| 7 |
+
"male"
|
| 8 |
+
],
|
| 9 |
+
"symbols": [
|
| 10 |
+
"<pad>",
|
| 11 |
+
"<bos>",
|
| 12 |
+
"<eos>",
|
| 13 |
+
"<sp>",
|
| 14 |
+
"<unk>",
|
| 15 |
+
",",
|
| 16 |
+
"?",
|
| 17 |
+
"!",
|
| 18 |
+
"|",
|
| 19 |
+
"||",
|
| 20 |
+
"<yati>",
|
| 21 |
+
"a",
|
| 22 |
+
"A",
|
| 23 |
+
"i",
|
| 24 |
+
"I",
|
| 25 |
+
"u",
|
| 26 |
+
"U",
|
| 27 |
+
"f",
|
| 28 |
+
"e",
|
| 29 |
+
"E",
|
| 30 |
+
"o",
|
| 31 |
+
"O",
|
| 32 |
+
"k",
|
| 33 |
+
"g",
|
| 34 |
+
"N",
|
| 35 |
+
"c",
|
| 36 |
+
"j",
|
| 37 |
+
"Y",
|
| 38 |
+
"w",
|
| 39 |
+
"q",
|
| 40 |
+
"R",
|
| 41 |
+
"t",
|
| 42 |
+
"d",
|
| 43 |
+
"n",
|
| 44 |
+
"p",
|
| 45 |
+
"b",
|
| 46 |
+
"m",
|
| 47 |
+
"y",
|
| 48 |
+
"r",
|
| 49 |
+
"l",
|
| 50 |
+
"v",
|
| 51 |
+
"S",
|
| 52 |
+
"z",
|
| 53 |
+
"s",
|
| 54 |
+
"h",
|
| 55 |
+
"M",
|
| 56 |
+
"H"
|
| 57 |
+
],
|
| 58 |
+
"frontend_fingerprint": "605680d636eba1c6b48119392b97d43de2b71f8abfc52ead230d0f2074ef69f2",
|
| 59 |
+
"max_tokens": 512,
|
| 60 |
+
"inputs": {
|
| 61 |
+
"tokens": "int64 [1, N] phoneme ids from alpa/text.py, with the edge tokens",
|
| 62 |
+
"weight": "int64 [1, N] syllable weight: 0 none, 1 laghu, 2 guru",
|
| 63 |
+
"voice": "int64 [1] index into voices",
|
| 64 |
+
"speed": "float32 [1], 1.0 = as predicted",
|
| 65 |
+
"pitch_shift": "float32 [1], added to log F0",
|
| 66 |
+
"pitch_range": "float32 [1], 1.0 = as predicted"
|
| 67 |
+
},
|
| 68 |
+
"outputs": {
|
| 69 |
+
"audio": "float32 [1, samples]",
|
| 70 |
+
"durations": "int64 [1, N] frames per token"
|
| 71 |
+
},
|
| 72 |
+
"polish": {
|
| 73 |
+
"pitch_smooth": 3,
|
| 74 |
+
"consonant_limit": 1000.0,
|
| 75 |
+
"open_limit": 4500.0,
|
| 76 |
+
"sharpen": 1.6,
|
| 77 |
+
"sharpen_from": 8,
|
| 78 |
+
"sharpen_to": 40,
|
| 79 |
+
"noise_cap": -15.0,
|
| 80 |
+
"treble": 4.0
|
| 81 |
+
},
|
| 82 |
+
"note": "Same weights as alpa-v0.1-preview.onnx. The corrections made while speaking (alpa/model.py, Polish) are part of the graph."
|
| 83 |
+
}
|
onnx/alpa-v0.1.1.onnx
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:778b6cbbeea71093f23456c8a21548512dbbcf2d03a3fe7a3c8110464a202502
|
| 3 |
+
size 22550089
|
samples/1_gita_1_1.wav
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
size 375852
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:f36dc9ccbd2450188b375b51b35391f3fed8b4acd1366b39b13c15d1a139560e
|
| 3 |
size 375852
|
samples/2_retroflex_and_dental.wav
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
size 307756
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:6e18aa19c9c881b8654a793bcb715c9faf1b051ac6e92e8c0887152ce6e75d54
|
| 3 |
size 307756
|
samples/3_everyday.wav
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
size 343084
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:75691854bd6de908bfb5749df1752c888f34ac2aa6e47921803e597b1fdf8cd9
|
| 3 |
size 343084
|
samples/4_aspirates.wav
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
size 297516
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:4e690cbbbf345ecb38520b3de3e54c092c33bbed33269c522b1243834b8c19ef
|
| 3 |
size 297516
|
samples/5_subhashita.wav
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
size 200236
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:12ac7547d1e9906f846b76b2c9b061f98b9a019677e375a4be669a554a0cf2d7
|
| 3 |
size 200236
|
samples/after_v0.1.1.wav
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:89b7deb71e7f10aafa0e9a877669b1b2e50ebde3ca60e336d709929466ff3287
|
| 3 |
+
size 248364
|
samples/before_v0.1.wav
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:929368b57b2292d92c01df98d81dab719e1535f9a21eb820f57c5b7df9e2d630
|
| 3 |
+
size 248364
|