inoryQwQ commited on
Commit ยท
0c313fe
1
Parent(s): 6badca1
feat: add AX620E and AX637 support (encoder NPU + decoder ONNX fallback)
Browse files- AX620E: Encoder axmodel (U16, cosine 1.0), Decoder ONNX (CPU)
- AX637: Encoder axmodel (U16, cosine 1.0), Decoder ONNX (CPU)
- AX650: Full NPU (encoder + decoder axmodel)
- Multi-chip engine with auto-detection
- Note: HiFi-GAN decoder cannot compile on AX620E/AX637
(ConvTranspose stride=8 exceeds NPU tiling limits)
- .gitattributes +2 -1
- README.md +25 -38
- models/{dp.onnx โ ax620e/dp.onnx} +0 -0
- models/{emb.npy โ ax620e/emb.npy} +0 -0
- models/ax620e/inflect_decoder.onnx +3 -0
- models/ax620e/inflect_decoder.onnx.data +3 -0
- models/ax620e/inflect_encoder.axmodel +3 -0
- models/ax620e/model_meta.json +13 -0
- models/ax637/dp.onnx +3 -0
- models/ax637/emb.npy +0 -0
- models/ax637/inflect_decoder.onnx +3 -0
- models/ax637/inflect_decoder.onnx.data +3 -0
- models/ax637/inflect_encoder.axmodel +3 -0
- models/ax637/model_meta.json +13 -0
- models/ax650/dp.onnx +3 -0
- models/ax650/emb.npy +0 -0
- models/{inflect_decoder.axmodel โ ax650/inflect_decoder.axmodel} +0 -0
- models/{inflect_encoder.axmodel โ ax650/inflect_encoder.axmodel} +0 -0
- models/ax650/model_meta.json +13 -0
- models/model_meta.json +0 -58
- python/inflect_tts_sdk/__main__.py +5 -4
- python/inflect_tts_sdk/tts_engine.py +89 -60
- run.sh +4 -3
.gitattributes
CHANGED
|
@@ -1,4 +1,5 @@
|
|
| 1 |
-
models/*.axmodel filter=lfs diff=lfs merge=lfs -text
|
| 2 |
*.pth filter=lfs diff=lfs merge=lfs -text
|
| 3 |
*.onnx filter=lfs diff=lfs merge=lfs -text
|
| 4 |
*.wav filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
| 1 |
+
models/**/*.axmodel filter=lfs diff=lfs merge=lfs -text
|
| 2 |
*.pth filter=lfs diff=lfs merge=lfs -text
|
| 3 |
*.onnx filter=lfs diff=lfs merge=lfs -text
|
| 4 |
*.wav filter=lfs diff=lfs merge=lfs -text
|
| 5 |
+
*.onnx.data filter=lfs diff=lfs merge=lfs -text
|
README.md
CHANGED
|
@@ -1,65 +1,52 @@
|
|
| 1 |
---
|
| 2 |
license: mit
|
| 3 |
-
language:
|
| 4 |
-
|
| 5 |
-
tags:
|
| 6 |
-
- text-to-speech
|
| 7 |
-
- vits
|
| 8 |
-
- ax650
|
| 9 |
-
- npu
|
| 10 |
-
- onnx
|
| 11 |
pipeline_tag: text-to-speech
|
| 12 |
---
|
| 13 |
|
| 14 |
-
# Inflect-Micro-v2 for
|
| 15 |
|
| 16 |
-
High-quality English TTS (VITS
|
| 17 |
|
| 18 |
-
|
| 19 |
-
-
|
| 20 |
-
|
| 21 |
-
|
|
|
|
| 22 |
|
| 23 |
-
## ๐ง Audio Samples
|
| 24 |
|
| 25 |
-
*
|
| 26 |
-
|
| 27 |
-
**"Hello and welcome to Inflect Micro V2 text to speech."** (3.6s, RTF 0.05x)
|
| 28 |
<audio controls src="https://huggingface.co/AXERA-TECH/inflect_micro_v2/resolve/main/samples/en_hello.wav"></audio>
|
| 29 |
|
| 30 |
-
**"The weather today is sunny with a gentle breeze."**
|
| 31 |
<audio controls src="https://huggingface.co/AXERA-TECH/inflect_micro_v2/resolve/main/samples/en_weather.wav"></audio>
|
| 32 |
|
| 33 |
-
**"Once upon a time, there lived a curious robot who loved to sing."**
|
| 34 |
<audio controls src="https://huggingface.co/AXERA-TECH/inflect_micro_v2/resolve/main/samples/en_story.wav"></audio>
|
| 35 |
|
| 36 |
-
**"Neural voice synthesis is the next frontier of
|
| 37 |
<audio controls src="https://huggingface.co/AXERA-TECH/inflect_micro_v2/resolve/main/samples/en_tech.wav"></audio>
|
| 38 |
|
| 39 |
-
**"This audio was generated entirely on the AX650 NPU chip."**
|
| 40 |
<audio controls src="https://huggingface.co/AXERA-TECH/inflect_micro_v2/resolve/main/samples/en_intro.wav"></audio>
|
| 41 |
|
| 42 |
## ๐ Quick Start
|
| 43 |
|
| 44 |
```bash
|
| 45 |
bash setup.sh
|
| 46 |
-
bash run.sh "Hello world." output.wav
|
| 47 |
```
|
| 48 |
|
| 49 |
-
##
|
| 50 |
-
|
| 51 |
-
```bash
|
| 52 |
-
# CLI
|
| 53 |
-
python -m inflect_tts_sdk --text "Hello" --output out.wav
|
| 54 |
|
| 55 |
-
# OpenAI-compatible Server
|
| 56 |
-
python -m inflect_tts_sdk.server --port 8000
|
| 57 |
```
|
| 58 |
-
|
| 59 |
-
|
| 60 |
-
|
| 61 |
-
|
| 62 |
-
|
| 63 |
-
|
| 64 |
-
|
| 65 |
-
|
|
|
|
| 1 |
---
|
| 2 |
license: mit
|
| 3 |
+
language: [en]
|
| 4 |
+
tags: [text-to-speech, vits, ax650, ax620e, ax637, npu, onnx]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 5 |
pipeline_tag: text-to-speech
|
| 6 |
---
|
| 7 |
|
| 8 |
+
# Inflect-Micro-v2 for AXera NPU
|
| 9 |
|
| 10 |
+
High-quality English TTS (VITS) optimized for **AX650 / AX620E / AX637** NPUs.
|
| 11 |
|
| 12 |
+
| Chip | Encoder | Decoder | Precision |
|
| 13 |
+
|------|---------|---------|-----------|
|
| 14 |
+
| **AX650** | NPU (axmodel) | NPU (axmodel) | U16 |
|
| 15 |
+
| **AX620E** | NPU (axmodel) | CPU (onnxruntime) | U16 |
|
| 16 |
+
| **AX637** | NPU (axmodel) | CPU (onnxruntime) | U16 |
|
| 17 |
|
| 18 |
+
## ๐ง Audio Samples (AX650 NPU)
|
| 19 |
|
| 20 |
+
**"Hello and welcome to Inflect Micro V2 text to speech."**
|
|
|
|
|
|
|
| 21 |
<audio controls src="https://huggingface.co/AXERA-TECH/inflect_micro_v2/resolve/main/samples/en_hello.wav"></audio>
|
| 22 |
|
| 23 |
+
**"The weather today is sunny with a gentle breeze."**
|
| 24 |
<audio controls src="https://huggingface.co/AXERA-TECH/inflect_micro_v2/resolve/main/samples/en_weather.wav"></audio>
|
| 25 |
|
| 26 |
+
**"Once upon a time, there lived a curious robot who loved to sing."**
|
| 27 |
<audio controls src="https://huggingface.co/AXERA-TECH/inflect_micro_v2/resolve/main/samples/en_story.wav"></audio>
|
| 28 |
|
| 29 |
+
**"Neural voice synthesis is the next frontier of AI."**
|
| 30 |
<audio controls src="https://huggingface.co/AXERA-TECH/inflect_micro_v2/resolve/main/samples/en_tech.wav"></audio>
|
| 31 |
|
| 32 |
+
**"This audio was generated entirely on the AX650 NPU chip."**
|
| 33 |
<audio controls src="https://huggingface.co/AXERA-TECH/inflect_micro_v2/resolve/main/samples/en_intro.wav"></audio>
|
| 34 |
|
| 35 |
## ๐ Quick Start
|
| 36 |
|
| 37 |
```bash
|
| 38 |
bash setup.sh
|
| 39 |
+
bash run.sh "Hello world." output.wav cli ax650
|
| 40 |
```
|
| 41 |
|
| 42 |
+
## ๐ Structure
|
|
|
|
|
|
|
|
|
|
|
|
|
| 43 |
|
|
|
|
|
|
|
| 44 |
```
|
| 45 |
+
models/
|
| 46 |
+
ax650/ โ full NPU (encoder + decoder axmodel)
|
| 47 |
+
ax620e/ โ encoder axmodel + decoder ONNX
|
| 48 |
+
ax637/ โ encoder axmodel + decoder ONNX
|
| 49 |
+
python/ โ Python SDK
|
| 50 |
+
cpp/ โ C++ SDK
|
| 51 |
+
samples/ โ audio samples
|
| 52 |
+
```
|
models/{dp.onnx โ ax620e/dp.onnx}
RENAMED
|
File without changes
|
models/{emb.npy โ ax620e/emb.npy}
RENAMED
|
File without changes
|
models/ax620e/inflect_decoder.onnx
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:6ad0e403794f3e9b24be9bcc438785ff46b887f10bf90698411fc25edcc0504e
|
| 3 |
+
size 107178
|
models/ax620e/inflect_decoder.onnx.data
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:f378527dfcfb8b7069193559bde3702af3a1b889b0f3af403dbc3a3c4ed8c5ad
|
| 3 |
+
size 38338560
|
models/ax620e/inflect_encoder.axmodel
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:cc63f65ab37ad48156cbfc26c499f5ea5616ef0a0ce98d9ece7ecfcb3f56192b
|
| 3 |
+
size 1728627
|
models/ax620e/model_meta.json
ADDED
|
@@ -0,0 +1,13 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"model": "inflect_micro_v2",
|
| 3 |
+
"chip": "ax620e",
|
| 4 |
+
"encoder": "axmodel",
|
| 5 |
+
"decoder": "onnx",
|
| 6 |
+
"dp": "onnx",
|
| 7 |
+
"embedding": "npy",
|
| 8 |
+
"sample_rate": 24000,
|
| 9 |
+
"hop_length": 256,
|
| 10 |
+
"precision": "U16",
|
| 11 |
+
"max_tokens": 200,
|
| 12 |
+
"max_mel_frames": 500
|
| 13 |
+
}
|
models/ax637/dp.onnx
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:fb2f0f1138c7c409a89eaeaec81dec6720dfbc82d4f20d0159b1c92058527731
|
| 3 |
+
size 1105165
|
models/ax637/emb.npy
ADDED
|
Binary file (68.5 kB). View file
|
|
|
models/ax637/inflect_decoder.onnx
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:6ad0e403794f3e9b24be9bcc438785ff46b887f10bf90698411fc25edcc0504e
|
| 3 |
+
size 107178
|
models/ax637/inflect_decoder.onnx.data
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:f378527dfcfb8b7069193559bde3702af3a1b889b0f3af403dbc3a3c4ed8c5ad
|
| 3 |
+
size 38338560
|
models/ax637/inflect_encoder.axmodel
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:4a883cae3c3d0f8f654acd38c67c8c96994e130f19a53a28794b1fa3c8c590bd
|
| 3 |
+
size 1695942
|
models/ax637/model_meta.json
ADDED
|
@@ -0,0 +1,13 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"model": "inflect_micro_v2",
|
| 3 |
+
"chip": "ax637",
|
| 4 |
+
"encoder": "axmodel",
|
| 5 |
+
"decoder": "onnx",
|
| 6 |
+
"dp": "onnx",
|
| 7 |
+
"embedding": "npy",
|
| 8 |
+
"sample_rate": 24000,
|
| 9 |
+
"hop_length": 256,
|
| 10 |
+
"precision": "U16",
|
| 11 |
+
"max_tokens": 200,
|
| 12 |
+
"max_mel_frames": 500
|
| 13 |
+
}
|
models/ax650/dp.onnx
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:fb2f0f1138c7c409a89eaeaec81dec6720dfbc82d4f20d0159b1c92058527731
|
| 3 |
+
size 1105165
|
models/ax650/emb.npy
ADDED
|
Binary file (68.5 kB). View file
|
|
|
models/{inflect_decoder.axmodel โ ax650/inflect_decoder.axmodel}
RENAMED
|
File without changes
|
models/{inflect_encoder.axmodel โ ax650/inflect_encoder.axmodel}
RENAMED
|
File without changes
|
models/ax650/model_meta.json
ADDED
|
@@ -0,0 +1,13 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"model": "inflect_micro_v2",
|
| 3 |
+
"chip": "ax650",
|
| 4 |
+
"encoder": "axmodel",
|
| 5 |
+
"decoder": "axmodel",
|
| 6 |
+
"dp": "onnx",
|
| 7 |
+
"embedding": "npy",
|
| 8 |
+
"sample_rate": 24000,
|
| 9 |
+
"hop_length": 256,
|
| 10 |
+
"precision": "U16",
|
| 11 |
+
"max_tokens": 200,
|
| 12 |
+
"max_mel_frames": 500
|
| 13 |
+
}
|
models/model_meta.json
DELETED
|
@@ -1,58 +0,0 @@
|
|
| 1 |
-
{
|
| 2 |
-
"model_name": "inflect_micro_v2",
|
| 3 |
-
"model_type": "vits_tts",
|
| 4 |
-
"framework": "pytorch",
|
| 5 |
-
"task": "text_to_speech",
|
| 6 |
-
"encoder_decoder_split": true,
|
| 7 |
-
"config": {
|
| 8 |
-
"sampling_rate": 24000,
|
| 9 |
-
"hop_length": 256,
|
| 10 |
-
"inter_channels": 192,
|
| 11 |
-
"hidden_channels": 96,
|
| 12 |
-
"filter_channels": 768,
|
| 13 |
-
"n_vocab": 256,
|
| 14 |
-
"add_blank": true
|
| 15 |
-
},
|
| 16 |
-
"submodels": [
|
| 17 |
-
{
|
| 18 |
-
"name": "inflect_encoder",
|
| 19 |
-
"onnx_file": "inflect_encoder_static.onnx",
|
| 20 |
-
"description": "TextEncoder: tokens -> m_p, logs_p, x, x_mask",
|
| 21 |
-
"inputs": [
|
| 22 |
-
{"name": "tokens", "shape": [1, 200], "dtype": "int64", "dynamic_dim": 1},
|
| 23 |
-
{"name": "lengths", "shape": [1], "dtype": "int64"}
|
| 24 |
-
],
|
| 25 |
-
"outputs": [
|
| 26 |
-
{"name": "m_p", "shape": [1, 192, "token_len"], "dtype": "float32"},
|
| 27 |
-
{"name": "logs_p", "shape": [1, 192, "token_len"], "dtype": "float32"},
|
| 28 |
-
{"name": "x", "shape": [1, 96, "token_len"], "dtype": "float32"},
|
| 29 |
-
{"name": "x_mask", "shape": [1, 1, "token_len"], "dtype": "float32"}
|
| 30 |
-
]
|
| 31 |
-
},
|
| 32 |
-
{
|
| 33 |
-
"name": "inflect_decoder",
|
| 34 |
-
"onnx_file": "inflect_decoder_static.onnx",
|
| 35 |
-
"description": "Flow(reverse) + HiFi-GAN Generator: z_p -> waveform",
|
| 36 |
-
"inputs": [
|
| 37 |
-
{"name": "z_p", "shape": [1, 192, 500], "dtype": "float32", "dynamic_dim": 2},
|
| 38 |
-
{"name": "y_mask", "shape": [1, 1, 500], "dtype": "float32", "dynamic_dim": 2}
|
| 39 |
-
],
|
| 40 |
-
"outputs": [
|
| 41 |
-
{"name": "waveform", "shape": [1, 1, 128000], "dtype": "float32"}
|
| 42 |
-
]
|
| 43 |
-
}
|
| 44 |
-
],
|
| 45 |
-
"pipeline": {
|
| 46 |
-
"preprocessing": "text -> phonemes -> token_ids (CPU, Python frontend)",
|
| 47 |
-
"encoder": "tokens -> enc_p -> m_p, logs_p (NPU)",
|
| 48 |
-
"duration_alignment": "dp + generate_path + expand (CPU, Python)",
|
| 49 |
-
"decoder": "z_p -> flow(reverse) -> dec -> waveform (NPU)",
|
| 50 |
-
"postprocessing": "none (raw waveform output)"
|
| 51 |
-
},
|
| 52 |
-
"export_info": {
|
| 53 |
-
"opset_version": 18,
|
| 54 |
-
"torch_version": "2.x",
|
| 55 |
-
"dynamic_onnx_available": true,
|
| 56 |
-
"static_onnx_for_compile": true
|
| 57 |
-
}
|
| 58 |
-
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
python/inflect_tts_sdk/__main__.py
CHANGED
|
@@ -1,19 +1,20 @@
|
|
| 1 |
-
"""CLI: python -m inflect_tts_sdk --text "Hello" --output out.wav"""
|
| 2 |
import argparse
|
| 3 |
from pathlib import Path
|
| 4 |
from .tts_engine import InflectTTSEngine
|
| 5 |
|
| 6 |
def main():
|
| 7 |
-
ap = argparse.ArgumentParser(description="Inflect-Micro-v2 TTS (
|
| 8 |
-
ap.add_argument("--text", required=True
|
| 9 |
ap.add_argument("--output", default="output.wav")
|
| 10 |
ap.add_argument("--speed", type=float, default=1.0)
|
| 11 |
ap.add_argument("--variation", type=float, default=0.667)
|
| 12 |
ap.add_argument("--seed", type=int, default=0)
|
| 13 |
ap.add_argument("--model-dir", default=None)
|
|
|
|
| 14 |
args = ap.parse_args()
|
| 15 |
|
| 16 |
-
engine = InflectTTSEngine(model_dir=args.model_dir)
|
| 17 |
engine.save(args.text, args.output, speed=args.speed,
|
| 18 |
variation=args.variation, seed=args.seed)
|
| 19 |
print(f"๐ {args.output}")
|
|
|
|
| 1 |
+
"""CLI: python -m inflect_tts_sdk --text "Hello" --output out.wav [--chip ax650]"""
|
| 2 |
import argparse
|
| 3 |
from pathlib import Path
|
| 4 |
from .tts_engine import InflectTTSEngine
|
| 5 |
|
| 6 |
def main():
|
| 7 |
+
ap = argparse.ArgumentParser(description="Inflect-Micro-v2 TTS (AX NPU)")
|
| 8 |
+
ap.add_argument("--text", required=True)
|
| 9 |
ap.add_argument("--output", default="output.wav")
|
| 10 |
ap.add_argument("--speed", type=float, default=1.0)
|
| 11 |
ap.add_argument("--variation", type=float, default=0.667)
|
| 12 |
ap.add_argument("--seed", type=int, default=0)
|
| 13 |
ap.add_argument("--model-dir", default=None)
|
| 14 |
+
ap.add_argument("--chip", default="ax650", choices=["ax650", "ax620e", "ax637"])
|
| 15 |
args = ap.parse_args()
|
| 16 |
|
| 17 |
+
engine = InflectTTSEngine(model_dir=args.model_dir, chip=args.chip)
|
| 18 |
engine.save(args.text, args.output, speed=args.speed,
|
| 19 |
variation=args.variation, seed=args.seed)
|
| 20 |
print(f"๐ {args.output}")
|
python/inflect_tts_sdk/tts_engine.py
CHANGED
|
@@ -1,13 +1,13 @@
|
|
| 1 |
"""
|
| 2 |
-
Inflect-Micro-v2 TTS Engine
|
| 3 |
-
|
|
|
|
| 4 |
"""
|
| 5 |
from __future__ import annotations
|
| 6 |
-
import sys, os, io, math, re, json
|
| 7 |
from pathlib import Path
|
| 8 |
import numpy as np
|
| 9 |
import onnxruntime as ort
|
| 10 |
-
import axengine as axe
|
| 11 |
|
| 12 |
PKG = Path(__file__).resolve().parent
|
| 13 |
sys.path.insert(0, str(PKG))
|
|
@@ -18,70 +18,96 @@ from text.symbols import symbols
|
|
| 18 |
from text import cleaned_text_to_sequence
|
| 19 |
|
| 20 |
|
| 21 |
-
# โโ numpy
|
| 22 |
|
| 23 |
def _intersperse(lst, item):
|
| 24 |
result = [item] * (len(lst) * 2 + 1)
|
| 25 |
result[1::2] = lst
|
| 26 |
return result
|
| 27 |
|
| 28 |
-
|
| 29 |
def _sequence_mask(length, max_length=None):
|
| 30 |
if max_length is None:
|
| 31 |
max_length = int(length.max())
|
| 32 |
return np.arange(max_length, dtype=length.dtype)[None, :] < length[:, None]
|
| 33 |
|
| 34 |
-
|
| 35 |
def _generate_path(duration, mask):
|
| 36 |
-
"""numpy ็ๆฌ็ commons.generate_path
|
| 37 |
-
duration: [b, 1, t_x]
|
| 38 |
-
mask: [b, 1, t_y, t_x]
|
| 39 |
-
"""
|
| 40 |
b, _, t_y, t_x = mask.shape
|
| 41 |
-
cum_duration = np.cumsum(duration, axis=-1)
|
| 42 |
-
cum_duration_flat = cum_duration.reshape(b * t_x)
|
| 43 |
-
path = _sequence_mask(cum_duration_flat, t_y).astype(mask.dtype)
|
| 44 |
path = path.reshape(b, t_x, t_y)
|
| 45 |
-
padded = np.pad(path[:, :-1], ((0, 0), (1, 0), (0, 0)))
|
| 46 |
path = path - padded
|
| 47 |
-
path = path[:, np.newaxis, :, :].transpose(0, 1, 3, 2) * mask
|
| 48 |
return path
|
| 49 |
|
| 50 |
|
| 51 |
# โโ ๅผๆ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
|
| 52 |
|
| 53 |
class InflectTTSEngine:
|
| 54 |
-
"""Inflect-Micro-v2 TTS โ
|
| 55 |
|
| 56 |
-
def __init__(self, model_dir: str | Path | None = None):
|
| 57 |
if model_dir:
|
| 58 |
self.root = Path(model_dir)
|
| 59 |
else:
|
| 60 |
-
|
|
|
|
|
|
|
|
|
|
| 61 |
|
| 62 |
# ๅ ่ฝฝ้
็ฝฎ
|
| 63 |
-
|
| 64 |
-
with open(cfg_path) as f:
|
| 65 |
self.hps = json.load(f)
|
| 66 |
self.sample_rate = int(self.hps["data"]["sampling_rate"])
|
| 67 |
self.hop_length = int(self.hps["data"]["hop_length"])
|
| 68 |
self.hidden_channels = int(self.hps["model"]["hidden_channels"])
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 69 |
|
| 70 |
-
# Embedding
|
| 71 |
-
|
| 72 |
-
self._emb = np.load(str(emb_path))
|
| 73 |
self._emb_scale = math.sqrt(self.hidden_channels)
|
| 74 |
|
| 75 |
# Duration Predictor ONNX
|
| 76 |
-
|
| 77 |
-
|
| 78 |
-
|
| 79 |
-
|
| 80 |
-
#
|
| 81 |
-
|
| 82 |
-
|
| 83 |
-
|
| 84 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 85 |
|
| 86 |
def synthesize(self, text: str, speed: float = 1.0, variation: float = 0.667,
|
| 87 |
seed: int = 0):
|
|
@@ -96,8 +122,7 @@ class InflectTTSEngine:
|
|
| 96 |
pieces = []
|
| 97 |
for idx, chunk in enumerate(sentences):
|
| 98 |
if idx > 0:
|
| 99 |
-
|
| 100 |
-
pieces.append(np.zeros(pause, dtype=np.float32))
|
| 101 |
|
| 102 |
phonemes = run_vits_frontend(chunk).phoneme_text
|
| 103 |
seq = cleaned_text_to_sequence(phonemes)
|
|
@@ -106,64 +131,68 @@ class InflectTTSEngine:
|
|
| 106 |
if not seq:
|
| 107 |
continue
|
| 108 |
|
| 109 |
-
tokens = np.array(seq, dtype=np.int64)
|
| 110 |
tlen = len(tokens)
|
| 111 |
|
| 112 |
-
# โโ
|
| 113 |
-
x_emb = self._emb[tokens] * self._emb_scale
|
| 114 |
-
x_emb = x_emb.T[np.newaxis, :, :]
|
| 115 |
|
| 116 |
-
# โโ
|
| 117 |
MAX_TOK = 200
|
| 118 |
if tlen > MAX_TOK:
|
| 119 |
-
raise ValueError(f"Text too long: {tlen} tokens > {MAX_TOK}
|
| 120 |
x_pad = np.zeros((1, self.hidden_channels, MAX_TOK), dtype=np.float32)
|
| 121 |
x_pad[:, :, :tlen] = x_emb
|
| 122 |
|
| 123 |
enc_out = self.enc_session.run(None, {
|
| 124 |
"x_emb": x_pad,
|
| 125 |
-
"lengths": np.array([tlen], dtype=np.
|
| 126 |
})
|
| 127 |
-
m_p = np.asarray(enc_out[0])[:, :, :tlen]
|
| 128 |
logs_p = np.asarray(enc_out[1])[:, :, :tlen]
|
| 129 |
x_enc = np.asarray(enc_out[2])[:, :, :tlen]
|
| 130 |
x_mask = np.asarray(enc_out[3])[:, :, :tlen]
|
| 131 |
|
| 132 |
-
# โโ
|
| 133 |
logw = self._dp_sess.run(None, {
|
| 134 |
"x": x_enc.astype(np.float32),
|
| 135 |
"x_mask": x_mask.astype(np.float32),
|
| 136 |
-
})[0]
|
| 137 |
|
| 138 |
-
# โโ
|
| 139 |
rng = np.random.RandomState(seed + idx)
|
| 140 |
w = np.exp(logw) * x_mask * (1.0 / speed)
|
| 141 |
w_ceil = np.ceil(w)
|
| 142 |
y_lengths = np.clip(w_ceil.sum(axis=(1, 2)), 1, None).astype(np.int64)
|
| 143 |
y_mask = _sequence_mask(y_lengths, None)[:, np.newaxis, :].astype(x_mask.dtype)
|
| 144 |
attn_mask = x_mask[:, :, np.newaxis, :] * y_mask[:, :, :, np.newaxis]
|
| 145 |
-
|
| 146 |
-
attn = _generate_path(w_ceil, attn_mask) # [1, 1, Ty, T]
|
| 147 |
|
| 148 |
m_p = (attn.squeeze(1) @ m_p.transpose(0, 2, 1)).transpose(0, 2, 1)
|
| 149 |
logs_p = (attn.squeeze(1) @ logs_p.transpose(0, 2, 1)).transpose(0, 2, 1)
|
| 150 |
z_p = m_p + rng.randn(*m_p.shape).astype(np.float32) * np.exp(logs_p) * variation
|
| 151 |
|
| 152 |
-
# โโ
|
| 153 |
mel_len = z_p.shape[2]
|
| 154 |
MAX_MEL = 500
|
| 155 |
if mel_len > MAX_MEL:
|
| 156 |
-
raise ValueError(f"Audio too long: {mel_len} frames > {MAX_MEL}
|
| 157 |
-
|
| 158 |
-
|
| 159 |
-
|
| 160 |
-
|
| 161 |
-
|
| 162 |
-
|
| 163 |
-
"z_p": zp_np,
|
| 164 |
-
|
| 165 |
-
|
| 166 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 167 |
pieces.append(waveform)
|
| 168 |
|
| 169 |
waveform = np.clip(np.concatenate(pieces), -1.0, 1.0)
|
|
|
|
| 1 |
"""
|
| 2 |
+
Inflect-Micro-v2 TTS Engine โ multi-chip, zero PyTorch.
|
| 3 |
+
|
| 4 |
+
Supports: AX650 (full NPU), AX620E (encoder NPU + decoder ONNX), AX637 (encoder NPU + decoder ONNX).
|
| 5 |
"""
|
| 6 |
from __future__ import annotations
|
| 7 |
+
import sys, os, io, math, re, json
|
| 8 |
from pathlib import Path
|
| 9 |
import numpy as np
|
| 10 |
import onnxruntime as ort
|
|
|
|
| 11 |
|
| 12 |
PKG = Path(__file__).resolve().parent
|
| 13 |
sys.path.insert(0, str(PKG))
|
|
|
|
| 18 |
from text import cleaned_text_to_sequence
|
| 19 |
|
| 20 |
|
| 21 |
+
# โโ numpy ๅทฅๅ
ทๅฝๆฐ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
|
| 22 |
|
| 23 |
def _intersperse(lst, item):
|
| 24 |
result = [item] * (len(lst) * 2 + 1)
|
| 25 |
result[1::2] = lst
|
| 26 |
return result
|
| 27 |
|
|
|
|
| 28 |
def _sequence_mask(length, max_length=None):
|
| 29 |
if max_length is None:
|
| 30 |
max_length = int(length.max())
|
| 31 |
return np.arange(max_length, dtype=length.dtype)[None, :] < length[:, None]
|
| 32 |
|
|
|
|
| 33 |
def _generate_path(duration, mask):
|
|
|
|
|
|
|
|
|
|
|
|
|
| 34 |
b, _, t_y, t_x = mask.shape
|
| 35 |
+
cum_duration = np.cumsum(duration, axis=-1)
|
| 36 |
+
cum_duration_flat = cum_duration.reshape(b * t_x)
|
| 37 |
+
path = _sequence_mask(cum_duration_flat, t_y).astype(mask.dtype)
|
| 38 |
path = path.reshape(b, t_x, t_y)
|
| 39 |
+
padded = np.pad(path[:, :-1], ((0, 0), (1, 0), (0, 0)))
|
| 40 |
path = path - padded
|
| 41 |
+
path = path[:, np.newaxis, :, :].transpose(0, 1, 3, 2) * mask
|
| 42 |
return path
|
| 43 |
|
| 44 |
|
| 45 |
# โโ ๅผๆ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
|
| 46 |
|
| 47 |
class InflectTTSEngine:
|
| 48 |
+
"""Inflect-Micro-v2 TTS โ ๅค่ฏ็๏ผ้ถ PyTorchใ"""
|
| 49 |
|
| 50 |
+
def __init__(self, model_dir: str | Path | None = None, chip: str = "ax650"):
|
| 51 |
if model_dir:
|
| 52 |
self.root = Path(model_dir)
|
| 53 |
else:
|
| 54 |
+
# Default: models/{chip}/
|
| 55 |
+
self.root = PKG.parent.parent / "models" / chip
|
| 56 |
+
|
| 57 |
+
self.chip = chip
|
| 58 |
|
| 59 |
# ๅ ่ฝฝ้
็ฝฎ
|
| 60 |
+
with open(PKG / "config.json") as f:
|
|
|
|
| 61 |
self.hps = json.load(f)
|
| 62 |
self.sample_rate = int(self.hps["data"]["sampling_rate"])
|
| 63 |
self.hop_length = int(self.hps["data"]["hop_length"])
|
| 64 |
self.hidden_channels = int(self.hps["model"]["hidden_channels"])
|
| 65 |
+
self.inter_channels = int(self.hps["model"]["inter_channels"])
|
| 66 |
+
|
| 67 |
+
# ๅ ่ฝฝ chip-specific model_meta
|
| 68 |
+
meta_path = self.root / "model_meta.json"
|
| 69 |
+
if meta_path.exists():
|
| 70 |
+
with open(meta_path) as f:
|
| 71 |
+
self.meta = json.load(f)
|
| 72 |
+
else:
|
| 73 |
+
self.meta = {}
|
| 74 |
|
| 75 |
+
# Embedding (n_vocab, hidden_channels)
|
| 76 |
+
self._emb = np.load(str(self.root / "emb.npy"))
|
|
|
|
| 77 |
self._emb_scale = math.sqrt(self.hidden_channels)
|
| 78 |
|
| 79 |
# Duration Predictor ONNX
|
| 80 |
+
self._dp_sess = ort.InferenceSession(
|
| 81 |
+
str(self.root / "dp.onnx"), providers=["CPUExecutionProvider"]
|
| 82 |
+
)
|
| 83 |
+
|
| 84 |
+
# Encoder โ always AXMODEL
|
| 85 |
+
try:
|
| 86 |
+
import axengine as axe
|
| 87 |
+
enc_path = str(self.root / "inflect_encoder.axmodel")
|
| 88 |
+
self.enc_session = axe.InferenceSession(enc_path)
|
| 89 |
+
self._has_npu = True
|
| 90 |
+
except ImportError:
|
| 91 |
+
self._has_npu = False
|
| 92 |
+
print("[WARN] axengine not available, encoder will use CPU ONNX")
|
| 93 |
+
self.enc_session = ort.InferenceSession(
|
| 94 |
+
str(self.root / "inflect_encoder.onnx"),
|
| 95 |
+
providers=["CPUExecutionProvider"],
|
| 96 |
+
)
|
| 97 |
+
|
| 98 |
+
# Decoder โ AXMODEL or ONNX fallback
|
| 99 |
+
dec_axmodel = self.root / "inflect_decoder.axmodel"
|
| 100 |
+
if dec_axmodel.exists() and self._has_npu:
|
| 101 |
+
self.dec_session = axe.InferenceSession(str(dec_axmodel))
|
| 102 |
+
self._dec_is_npu = True
|
| 103 |
+
else:
|
| 104 |
+
dec_onnx = self.root / "inflect_decoder.onnx"
|
| 105 |
+
self.dec_session = ort.InferenceSession(
|
| 106 |
+
str(dec_onnx), providers=["CPUExecutionProvider"]
|
| 107 |
+
)
|
| 108 |
+
self._dec_is_npu = False
|
| 109 |
+
if self._has_npu:
|
| 110 |
+
print(f"[INFO] {chip}: decoder running on CPU (ONNX)")
|
| 111 |
|
| 112 |
def synthesize(self, text: str, speed: float = 1.0, variation: float = 0.667,
|
| 113 |
seed: int = 0):
|
|
|
|
| 122 |
pieces = []
|
| 123 |
for idx, chunk in enumerate(sentences):
|
| 124 |
if idx > 0:
|
| 125 |
+
pieces.append(np.zeros(round(self.sample_rate * 0.08), dtype=np.float32))
|
|
|
|
| 126 |
|
| 127 |
phonemes = run_vits_frontend(chunk).phoneme_text
|
| 128 |
seq = cleaned_text_to_sequence(phonemes)
|
|
|
|
| 131 |
if not seq:
|
| 132 |
continue
|
| 133 |
|
| 134 |
+
tokens = np.array(seq, dtype=np.int64)
|
| 135 |
tlen = len(tokens)
|
| 136 |
|
| 137 |
+
# โโ Embedding โโ
|
| 138 |
+
x_emb = self._emb[tokens] * self._emb_scale
|
| 139 |
+
x_emb = x_emb.T[np.newaxis, :, :]
|
| 140 |
|
| 141 |
+
# โโ Encoder โโ
|
| 142 |
MAX_TOK = 200
|
| 143 |
if tlen > MAX_TOK:
|
| 144 |
+
raise ValueError(f"Text too long: {tlen} tokens > {MAX_TOK}")
|
| 145 |
x_pad = np.zeros((1, self.hidden_channels, MAX_TOK), dtype=np.float32)
|
| 146 |
x_pad[:, :, :tlen] = x_emb
|
| 147 |
|
| 148 |
enc_out = self.enc_session.run(None, {
|
| 149 |
"x_emb": x_pad,
|
| 150 |
+
"lengths": np.array([tlen], dtype=np.int32),
|
| 151 |
})
|
| 152 |
+
m_p = np.asarray(enc_out[0])[:, :, :tlen]
|
| 153 |
logs_p = np.asarray(enc_out[1])[:, :, :tlen]
|
| 154 |
x_enc = np.asarray(enc_out[2])[:, :, :tlen]
|
| 155 |
x_mask = np.asarray(enc_out[3])[:, :, :tlen]
|
| 156 |
|
| 157 |
+
# โโ Duration Predictor โโ
|
| 158 |
logw = self._dp_sess.run(None, {
|
| 159 |
"x": x_enc.astype(np.float32),
|
| 160 |
"x_mask": x_mask.astype(np.float32),
|
| 161 |
+
})[0]
|
| 162 |
|
| 163 |
+
# โโ Duration + Alignment โโ
|
| 164 |
rng = np.random.RandomState(seed + idx)
|
| 165 |
w = np.exp(logw) * x_mask * (1.0 / speed)
|
| 166 |
w_ceil = np.ceil(w)
|
| 167 |
y_lengths = np.clip(w_ceil.sum(axis=(1, 2)), 1, None).astype(np.int64)
|
| 168 |
y_mask = _sequence_mask(y_lengths, None)[:, np.newaxis, :].astype(x_mask.dtype)
|
| 169 |
attn_mask = x_mask[:, :, np.newaxis, :] * y_mask[:, :, :, np.newaxis]
|
| 170 |
+
attn = _generate_path(w_ceil, attn_mask)
|
|
|
|
| 171 |
|
| 172 |
m_p = (attn.squeeze(1) @ m_p.transpose(0, 2, 1)).transpose(0, 2, 1)
|
| 173 |
logs_p = (attn.squeeze(1) @ logs_p.transpose(0, 2, 1)).transpose(0, 2, 1)
|
| 174 |
z_p = m_p + rng.randn(*m_p.shape).astype(np.float32) * np.exp(logs_p) * variation
|
| 175 |
|
| 176 |
+
# โโ Decoder โโ
|
| 177 |
mel_len = z_p.shape[2]
|
| 178 |
MAX_MEL = 500
|
| 179 |
if mel_len > MAX_MEL:
|
| 180 |
+
raise ValueError(f"Audio too long: {mel_len} frames > {MAX_MEL}")
|
| 181 |
+
|
| 182 |
+
if self._dec_is_npu:
|
| 183 |
+
zp_np = np.zeros((1, self.inter_channels, MAX_MEL), dtype=np.float32)
|
| 184 |
+
ym_np = np.zeros((1, 1, MAX_MEL), dtype=np.float32)
|
| 185 |
+
zp_np[:, :, :mel_len] = z_p
|
| 186 |
+
ym_np[:, :, :mel_len] = y_mask
|
| 187 |
+
dec_out = self.dec_session.run(None, {"z_p": zp_np, "y_mask": ym_np})
|
| 188 |
+
waveform = dec_out[0][0, 0, :mel_len * self.hop_length]
|
| 189 |
+
else:
|
| 190 |
+
dec_out = self.dec_session.run(None, {
|
| 191 |
+
"z_p": z_p.astype(np.float32),
|
| 192 |
+
"y_mask": y_mask.astype(np.float32),
|
| 193 |
+
})
|
| 194 |
+
waveform = dec_out[0][0, 0, :mel_len * self.hop_length]
|
| 195 |
+
|
| 196 |
pieces.append(waveform)
|
| 197 |
|
| 198 |
waveform = np.clip(np.concatenate(pieces), -1.0, 1.0)
|
run.sh
CHANGED
|
@@ -3,15 +3,16 @@ set -euo pipefail
|
|
| 3 |
TEXT="${1:-Hello world, this is Inflect Micro V2.}"
|
| 4 |
OUTPUT="${2:-output.wav}"
|
| 5 |
MODE="${3:-cli}"
|
|
|
|
| 6 |
|
| 7 |
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
| 8 |
cd "$SCRIPT_DIR/python"
|
| 9 |
|
| 10 |
if [ "$MODE" = "server" ]; then
|
| 11 |
-
echo "๐ Starting OpenAI TTS Server on port ${
|
| 12 |
-
python3 -m inflect_tts_sdk.server --port "${
|
| 13 |
else
|
| 14 |
echo "๐ค ${TEXT}"
|
| 15 |
-
python3 -m inflect_tts_sdk --text "${TEXT}" --output "${OUTPUT}"
|
| 16 |
echo "๐ ${OUTPUT}"
|
| 17 |
fi
|
|
|
|
| 3 |
TEXT="${1:-Hello world, this is Inflect Micro V2.}"
|
| 4 |
OUTPUT="${2:-output.wav}"
|
| 5 |
MODE="${3:-cli}"
|
| 6 |
+
CHIP="${4:-ax650}"
|
| 7 |
|
| 8 |
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
| 9 |
cd "$SCRIPT_DIR/python"
|
| 10 |
|
| 11 |
if [ "$MODE" = "server" ]; then
|
| 12 |
+
echo "๐ Starting OpenAI TTS Server on port ${5:-8000}..."
|
| 13 |
+
python3 -m inflect_tts_sdk.server --port "${5:-8000}" --chip "$CHIP"
|
| 14 |
else
|
| 15 |
echo "๐ค ${TEXT}"
|
| 16 |
+
python3 -m inflect_tts_sdk --text "${TEXT}" --output "${OUTPUT}" --chip "$CHIP"
|
| 17 |
echo "๐ ${OUTPUT}"
|
| 18 |
fi
|