FastWave.AXERA: AX650 NPU audio super-resolution (U16)
Browse files- .gitattributes +14 -0
- README.md +61 -36
- bin/fastwave_ax650 +0 -0
- models/assets/emb_p1_b.npy +3 -0
- models/assets/emb_p1_w.npy +3 -0
- models/assets/emb_p2_b.npy +3 -0
- models/assets/emb_p2_w.npy +3 -0
- python/benchmark.py +2 -1
- python/demo.py +3 -1
- python/fastwave_sdk/audio.py +13 -12
- python/fastwave_sdk/pipeline.py +12 -4
- python/fastwave_sdk/postfilter.py +41 -0
- reports/performance.md +18 -11
- run_ax650.sh +29 -6
- samples/listen/48k_from16k/speech_2_output_48k.wav +1 -1
- samples/listen/48k_from16k/speech_3_output_48k.wav +1 -1
- samples/listen/48k_from8k/OSR_cn_000_0072_output_48k.wav +1 -1
- samples/listen/48k_from8k/OSR_cn_000_0074_output_48k.wav +1 -1
- samples/listen/48k_from8k/OSR_us_000_0010_output_48k.wav +1 -1
- samples/listen/48k_from8k/OSR_us_000_0013_output_48k.wav +1 -1
- samples/listen/spectrograms/OSR_cn_000_0072.png +2 -2
- samples/listen/spectrograms/OSR_cn_000_0072_input_spec.png +3 -0
- samples/listen/spectrograms/OSR_cn_000_0072_output_spec.png +3 -0
- samples/listen/spectrograms/OSR_cn_000_0074.png +3 -0
- samples/listen/spectrograms/OSR_cn_000_0074_input_spec.png +3 -0
- samples/listen/spectrograms/OSR_cn_000_0074_output_spec.png +3 -0
- samples/listen/spectrograms/OSR_us_000_0010.png +2 -2
- samples/listen/spectrograms/OSR_us_000_0010_input_spec.png +3 -0
- samples/listen/spectrograms/OSR_us_000_0010_output_spec.png +3 -0
- samples/listen/spectrograms/OSR_us_000_0013.png +3 -0
- samples/listen/spectrograms/OSR_us_000_0013_input_spec.png +3 -0
- samples/listen/spectrograms/OSR_us_000_0013_output_spec.png +3 -0
- samples/listen/spectrograms/speech_2.png +2 -2
- samples/listen/spectrograms/speech_2_input_spec.png +3 -0
- samples/listen/spectrograms/speech_2_output_spec.png +3 -0
- samples/listen/spectrograms/speech_3.png +2 -2
- samples/listen/spectrograms/speech_3_input_spec.png +3 -0
- samples/listen/spectrograms/speech_3_output_spec.png +3 -0
.gitattributes
CHANGED
|
@@ -50,3 +50,17 @@ samples/listen/spectrograms/OSR_cn_000_0072.png filter=lfs diff=lfs merge=lfs -t
|
|
| 50 |
samples/listen/spectrograms/OSR_us_000_0010.png filter=lfs diff=lfs merge=lfs -text
|
| 51 |
samples/listen/spectrograms/speech_2.png filter=lfs diff=lfs merge=lfs -text
|
| 52 |
samples/listen/spectrograms/speech_3.png filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 50 |
samples/listen/spectrograms/OSR_us_000_0010.png filter=lfs diff=lfs merge=lfs -text
|
| 51 |
samples/listen/spectrograms/speech_2.png filter=lfs diff=lfs merge=lfs -text
|
| 52 |
samples/listen/spectrograms/speech_3.png filter=lfs diff=lfs merge=lfs -text
|
| 53 |
+
samples/listen/spectrograms/OSR_cn_000_0072_input_spec.png filter=lfs diff=lfs merge=lfs -text
|
| 54 |
+
samples/listen/spectrograms/OSR_cn_000_0072_output_spec.png filter=lfs diff=lfs merge=lfs -text
|
| 55 |
+
samples/listen/spectrograms/OSR_cn_000_0074.png filter=lfs diff=lfs merge=lfs -text
|
| 56 |
+
samples/listen/spectrograms/OSR_cn_000_0074_input_spec.png filter=lfs diff=lfs merge=lfs -text
|
| 57 |
+
samples/listen/spectrograms/OSR_cn_000_0074_output_spec.png filter=lfs diff=lfs merge=lfs -text
|
| 58 |
+
samples/listen/spectrograms/OSR_us_000_0010_input_spec.png filter=lfs diff=lfs merge=lfs -text
|
| 59 |
+
samples/listen/spectrograms/OSR_us_000_0010_output_spec.png filter=lfs diff=lfs merge=lfs -text
|
| 60 |
+
samples/listen/spectrograms/OSR_us_000_0013.png filter=lfs diff=lfs merge=lfs -text
|
| 61 |
+
samples/listen/spectrograms/OSR_us_000_0013_input_spec.png filter=lfs diff=lfs merge=lfs -text
|
| 62 |
+
samples/listen/spectrograms/OSR_us_000_0013_output_spec.png filter=lfs diff=lfs merge=lfs -text
|
| 63 |
+
samples/listen/spectrograms/speech_2_input_spec.png filter=lfs diff=lfs merge=lfs -text
|
| 64 |
+
samples/listen/spectrograms/speech_2_output_spec.png filter=lfs diff=lfs merge=lfs -text
|
| 65 |
+
samples/listen/spectrograms/speech_3_input_spec.png filter=lfs diff=lfs merge=lfs -text
|
| 66 |
+
samples/listen/spectrograms/speech_3_output_spec.png filter=lfs diff=lfs merge=lfs -text
|
README.md
CHANGED
|
@@ -10,7 +10,6 @@ tags:
|
|
| 10 |
- audio-super-resolution
|
| 11 |
- speech-super-resolution
|
| 12 |
- bandwidth-extension
|
| 13 |
-
- diffuse
|
| 14 |
- audio-to-audio
|
| 15 |
library_name: axera
|
| 16 |
pipeline_tag: audio-to-audio
|
|
@@ -19,63 +18,89 @@ base_model: Nikait/FastWave
|
|
| 19 |
|
| 20 |
# FastWave.AXERA
|
| 21 |
|
| 22 |
-
**AX650 上的通用音频超分
|
| 23 |
|
| 24 |
-
|
| 25 |
-
[论文](https://arxiv.org/abs/2603.04122))。本仓库提供 AX650 NPU 版:
|
| 26 |
|
| 27 |
-
-
|
| 28 |
-
-
|
| 29 |
-
-
|
| 30 |
-
- `reports/`:精度与性能报告
|
| 31 |
|
| 32 |
-
|
| 33 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 34 |
|
| 35 |
## 快速开始(AX650 板端)
|
| 36 |
|
|
|
|
|
|
|
| 37 |
```bash
|
| 38 |
-
#
|
| 39 |
bash run_ax650.sh samples/listen/8k/OSR_us_000_0010_input_8k.wav output.wav
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 40 |
bash run_ax650.sh samples/listen/16k/speech_2_input_16k.wav output.wav --steps 4
|
|
|
|
| 41 |
```
|
| 42 |
|
| 43 |
-
输出固定 48 kHz。
|
| 44 |
|
| 45 |
## 结果
|
| 46 |
|
| 47 |
-
|
| 48 |
-
|---|---|
|
| 49 |
-
| [▶ 8k OSR_us_000_0010](samples/listen/8k/OSR_us_000_0010_input_8k.wav) | [▶ 48k](samples/listen/48k_from8k/OSR_us_000_0010_output_48k.wav) |
|
| 50 |
-
| [▶ 8k OSR_cn_000_0072](samples/listen/8k/OSR_cn_000_0072_input_8k.wav) | [▶ 48k](samples/listen/48k_from8k/OSR_cn_000_0072_output_48k.wav) |
|
| 51 |
-
| [▶ 16k speech_2](samples/listen/16k/speech_2_input_16k.wav) | [▶ 48k](samples/listen/48k_from16k/speech_2_output_48k.wav) |
|
| 52 |
|
| 53 |
-
|
| 54 |
-
|
| 55 |
-
[
|
|
|
|
|
|
|
|
|
|
| 56 |
|
| 57 |
-
##
|
| 58 |
|
| 59 |
-
|
|
| 60 |
|---|---|---|
|
| 61 |
-
|
|
| 62 |
-
|
|
| 63 |
|
| 64 |
-
##
|
| 65 |
|
| 66 |
-
|
| 67 |
-
|
| 68 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 69 |
|
| 70 |
-
##
|
| 71 |
|
| 72 |
-
|
| 73 |
-
|
| 74 |
-
|
| 75 |
-
2. EDM 的 `c_skip/c_in/c_out` 与正弦扩散 embedding 只依赖 sigma,host 端预计算后作为图输入,
|
| 76 |
-
图上没有 log/sqrt/sin/cos/OneHot。
|
| 77 |
-
3. 输入采样率通过 band one-hot 输入,一个模型覆盖 6–48 kHz。
|
| 78 |
|
| 79 |
## License
|
| 80 |
|
| 81 |
-
Apache-2.0
|
|
|
|
| 10 |
- audio-super-resolution
|
| 11 |
- speech-super-resolution
|
| 12 |
- bandwidth-extension
|
|
|
|
| 13 |
- audio-to-audio
|
| 14 |
library_name: axera
|
| 15 |
pipeline_tag: audio-to-audio
|
|
|
|
| 18 |
|
| 19 |
# FastWave.AXERA
|
| 20 |
|
| 21 |
+
**AX650 上的通用音频超分:6–48 kHz → 48 kHz。C++ 全链路 RTF 10.97(8 步)/ 5.51(4 步),Python SDK RTF 10.96 / 5.49;单 axmodel 11.9 MB,Python/C++ SDK 即取即用。**
|
| 22 |
|
| 23 |
+
  
|
|
|
|
| 24 |
|
| 25 |
+
- HuggingFace: https://huggingface.co/HY-2012/FastWave.AXERA
|
| 26 |
+
- GitHub: https://github.com/ZY-2012/FastWave.AXERA
|
| 27 |
+
- 上游: https://github.com/Nikait/FastWave
|
|
|
|
| 28 |
|
| 29 |
+
## 目录
|
| 30 |
+
|
| 31 |
+
- 快速开始
|
| 32 |
+
- 结果
|
| 33 |
+
- 推理速度
|
| 34 |
+
- 文件说明
|
| 35 |
+
- 数据来源
|
| 36 |
+
- License
|
| 37 |
|
| 38 |
## 快速开始(AX650 板端)
|
| 39 |
|
| 40 |
+
本仓库即完整包(`models/` + `bin/` + `python/` + `samples/`),下载后在包根目录运行:
|
| 41 |
+
|
| 42 |
```bash
|
| 43 |
+
# C++(默认)
|
| 44 |
bash run_ax650.sh samples/listen/8k/OSR_us_000_0010_input_8k.wav output.wav
|
| 45 |
+
|
| 46 |
+
# Python SDK
|
| 47 |
+
bash run_ax650.sh samples/listen/8k/OSR_us_000_0010_input_8k.wav output.wav --python
|
| 48 |
+
|
| 49 |
+
# 4 步采样(更快)/ 关闭后置降噪
|
| 50 |
bash run_ax650.sh samples/listen/16k/speech_2_input_16k.wav output.wav --steps 4
|
| 51 |
+
bash run_ax650.sh samples/listen/8k/OSR_cn_000_0072_input_8k.wav output.wav --no-denoise
|
| 52 |
```
|
| 53 |
|
| 54 |
+
输入支持 6–48 kHz,输出固定 48 kHz。
|
| 55 |
|
| 56 |
## 结果
|
| 57 |
|
| 58 |
+
### 8 kHz 输入(Open Speech Repository)
|
|
|
|
|
|
|
|
|
|
|
|
|
| 59 |
|
| 60 |
+
| 版本 | 输入 | FastWave.AXERA 输出 |
|
| 61 |
+
|---|---|---|
|
| 62 |
+
| **OSR_us_000_0010** | [▶](samples/listen/8k/OSR_us_000_0010_input_8k.wav)<br><img src="samples/listen/spectrograms/OSR_us_000_0010_input_spec.png" width="360"> | [▶](samples/listen/48k_from8k/OSR_us_000_0010_output_48k.wav)<br><img src="samples/listen/spectrograms/OSR_us_000_0010_output_spec.png" width="360"> |
|
| 63 |
+
| **OSR_us_000_0013** | [▶](samples/listen/8k/OSR_us_000_0013_input_8k.wav)<br><img src="samples/listen/spectrograms/OSR_us_000_0013_input_spec.png" width="360"> | [▶](samples/listen/48k_from8k/OSR_us_000_0013_output_48k.wav)<br><img src="samples/listen/spectrograms/OSR_us_000_0013_output_spec.png" width="360"> |
|
| 64 |
+
| **OSR_cn_000_0072** | [▶](samples/listen/8k/OSR_cn_000_0072_input_8k.wav)<br><img src="samples/listen/spectrograms/OSR_cn_000_0072_input_spec.png" width="360"> | [▶](samples/listen/48k_from8k/OSR_cn_000_0072_output_48k.wav)<br><img src="samples/listen/spectrograms/OSR_cn_000_0072_output_spec.png" width="360"> |
|
| 65 |
+
| **OSR_cn_000_0074** | [▶](samples/listen/8k/OSR_cn_000_0074_input_8k.wav)<br><img src="samples/listen/spectrograms/OSR_cn_000_0074_input_spec.png" width="360"> | [▶](samples/listen/48k_from8k/OSR_cn_000_0074_output_48k.wav)<br><img src="samples/listen/spectrograms/OSR_cn_000_0074_output_spec.png" width="360"> |
|
| 66 |
|
| 67 |
+
### 16 kHz 输入(人声)
|
| 68 |
|
| 69 |
+
| 版本 | 输入 | FastWave.AXERA 输出 |
|
| 70 |
|---|---|---|
|
| 71 |
+
| **speech_2** | [▶](samples/listen/16k/speech_2_input_16k.wav)<br><img src="samples/listen/spectrograms/speech_2_input_spec.png" width="360"> | [▶](samples/listen/48k_from16k/speech_2_output_48k.wav)<br><img src="samples/listen/spectrograms/speech_2_output_spec.png" width="360"> |
|
| 72 |
+
| **speech_3** | [▶](samples/listen/16k/speech_3_input_16k.wav)<br><img src="samples/listen/spectrograms/speech_3_input_spec.png" width="360"> | [▶](samples/listen/48k_from16k/speech_3_output_48k.wav)<br><img src="samples/listen/spectrograms/speech_3_output_spec.png" width="360"> |
|
| 73 |
|
| 74 |
+
### 8k → 48k 频谱对比(上:输入,下:输出)
|
| 75 |
|
| 76 |
+
<img src="samples/listen/spectrograms/OSR_us_000_0010.png" width="720">
|
| 77 |
+
|
| 78 |
+
## 推理速度:RTF
|
| 79 |
+
|
| 80 |
+
AX650(NPU3,BSP V3.10.2),chunk 32768 点(0.68 s):
|
| 81 |
+
|
| 82 |
+
| 链路 | 8 步 | 4 步 | 无重叠(8 步) |
|
| 83 |
+
|---|---:|---:|---:|
|
| 84 |
+
| C++ | 10.97 | 5.51 | 6.29 |
|
| 85 |
+
| Python SDK | 10.96 | 5.49 | — |
|
| 86 |
+
|
| 87 |
+
## 文件说明
|
| 88 |
+
|
| 89 |
+
| 路径 | 内容 |
|
| 90 |
+
|---|---|
|
| 91 |
+
| `models/fastwave_nfe.axmodel` | 单次 NFE 图(11.9 MB,U16 激活 / S8 权重 / U16 输出) |
|
| 92 |
+
| `models/assets/*.npy` | 扩散 embedding 权重(host 端计算) |
|
| 93 |
+
| `bin/fastwave_ax650` | C++ 推理程序(aarch64,AX650) |
|
| 94 |
+
| `python/` | Python SDK(numpy + axengine) |
|
| 95 |
+
| `samples/` | 8k / 16k 输入与 48k 输出、频谱图 |
|
| 96 |
+
| `reports/` | 精度与性能报告 |
|
| 97 |
|
| 98 |
+
## 数据来源
|
| 99 |
|
| 100 |
+
- Open Speech Repository(English / Chinese 8k):http://voiptroubleshooter.com/open_speech/index.html
|
| 101 |
+
- 16 kHz 参照:公开人声样例经 48k → 16k 降采样
|
| 102 |
+
- 上游模型权重:Nikait/FastWave(MIT)
|
|
|
|
|
|
|
|
|
|
| 103 |
|
| 104 |
## License
|
| 105 |
|
| 106 |
+
Apache-2.0。
|
bin/fastwave_ax650
ADDED
|
Binary file (76.6 kB). View file
|
|
|
models/assets/emb_p1_b.npy
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:56eb0ba80672dd4b474a72cf665645481fca907e35b89defdf48c9aa6623e9cb
|
| 3 |
+
size 2176
|
models/assets/emb_p1_w.npy
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:472e7ed0456762e5bcf2f4e47a609b9cacdd7c895e17a6a90dbc6f7a5b60eb47
|
| 3 |
+
size 262272
|
models/assets/emb_p2_b.npy
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:197f365eb799bfc44aa437800f443f7d4796c832cd120db7a39291e909f42cdc
|
| 3 |
+
size 2176
|
models/assets/emb_p2_w.npy
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:a5db3ac68049e055d8b17e56691fb0e8e822543673f4005b1d0aa7066aa39d2e
|
| 3 |
+
size 1048704
|
python/benchmark.py
CHANGED
|
@@ -22,9 +22,10 @@ def main():
|
|
| 22 |
ap.add_argument("--seconds", type=float, default=3.0)
|
| 23 |
ap.add_argument("--steps", type=int, default=8)
|
| 24 |
ap.add_argument("--iters", type=int, default=20)
|
|
|
|
| 25 |
a = ap.parse_args()
|
| 26 |
|
| 27 |
-
m = FastWaveSR(a.model, num_steps=a.steps)
|
| 28 |
wav, sr = A.read_wav(a.wav)
|
| 29 |
wav = wav[: int(a.seconds * sr)]
|
| 30 |
low = A.low_quality(wav / (np.abs(wav).max() + 1e-8), sr)
|
|
|
|
| 22 |
ap.add_argument("--seconds", type=float, default=3.0)
|
| 23 |
ap.add_argument("--steps", type=int, default=8)
|
| 24 |
ap.add_argument("--iters", type=int, default=20)
|
| 25 |
+
ap.add_argument("--overlap", type=float, default=0.5)
|
| 26 |
a = ap.parse_args()
|
| 27 |
|
| 28 |
+
m = FastWaveSR(a.model, num_steps=a.steps, overlap=a.overlap)
|
| 29 |
wav, sr = A.read_wav(a.wav)
|
| 30 |
wav = wav[: int(a.seconds * sr)]
|
| 31 |
low = A.low_quality(wav / (np.abs(wav).max() + 1e-8), sr)
|
python/demo.py
CHANGED
|
@@ -21,6 +21,7 @@ def main():
|
|
| 21 |
ap.add_argument("--overlap", type=float, default=0.5, help="chunk overlap ratio")
|
| 22 |
ap.add_argument("--steps", type=int, default=8, help="EDM sampling steps (4 or 8)")
|
| 23 |
ap.add_argument("--seed", type=int, default=1234)
|
|
|
|
| 24 |
args = ap.parse_args()
|
| 25 |
|
| 26 |
wav, sr = A.read_wav(args.input)
|
|
@@ -28,7 +29,8 @@ def main():
|
|
| 28 |
model = FastWaveSR(args.model, num_steps=args.steps, chunk=args.chunk,
|
| 29 |
overlap=args.overlap, seed=args.seed)
|
| 30 |
t0 = time.time()
|
| 31 |
-
out = model.enhance(wav, sr,
|
|
|
|
| 32 |
dt = time.time() - t0
|
| 33 |
print()
|
| 34 |
A.write_wav(args.output, out, 48000)
|
|
|
|
| 21 |
ap.add_argument("--overlap", type=float, default=0.5, help="chunk overlap ratio")
|
| 22 |
ap.add_argument("--steps", type=int, default=8, help="EDM sampling steps (4 or 8)")
|
| 23 |
ap.add_argument("--seed", type=int, default=1234)
|
| 24 |
+
ap.add_argument("--no-denoise", action="store_true", help="disable the output noise post filter")
|
| 25 |
args = ap.parse_args()
|
| 26 |
|
| 27 |
wav, sr = A.read_wav(args.input)
|
|
|
|
| 29 |
model = FastWaveSR(args.model, num_steps=args.steps, chunk=args.chunk,
|
| 30 |
overlap=args.overlap, seed=args.seed)
|
| 31 |
t0 = time.time()
|
| 32 |
+
out = model.enhance(wav, sr, denoise=not args.no_denoise,
|
| 33 |
+
progress=lambda i, n: print(f"\r[chunk ] {i}/{n}", end="", flush=True))
|
| 34 |
dt = time.time() - t0
|
| 35 |
print()
|
| 36 |
A.write_wav(args.output, out, 48000)
|
python/fastwave_sdk/audio.py
CHANGED
|
@@ -78,19 +78,20 @@ def resample_poly(x, up, down):
|
|
| 78 |
return x.astype(np.float32)
|
| 79 |
max_rate = max(up, down)
|
| 80 |
half = 10 * max_rate
|
| 81 |
-
h = _firwin(1.0 / max_rate, 2 * half + 1) * up
|
| 82 |
-
|
| 83 |
-
y = np.zeros((len(x) - 1) * up + 1, dtype=np.float64)
|
| 84 |
-
y[::up] = x
|
| 85 |
-
y = np.convolve(y, h)
|
| 86 |
-
# compensate the FIR group delay, then decimate
|
| 87 |
-
y = y[half::down]
|
| 88 |
n_out = int(math.ceil(len(x) * up / down))
|
| 89 |
-
|
| 90 |
-
|
| 91 |
-
|
| 92 |
-
|
| 93 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 94 |
|
| 95 |
|
| 96 |
def resample_to(x, sr_in, sr_out):
|
|
|
|
| 78 |
return x.astype(np.float32)
|
| 79 |
max_rate = max(up, down)
|
| 80 |
half = 10 * max_rate
|
| 81 |
+
h = (_firwin(1.0 / max_rate, 2 * half + 1) * up).astype(np.float64)
|
| 82 |
+
hl = len(h)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 83 |
n_out = int(math.ceil(len(x) * up / down))
|
| 84 |
+
# polyphase evaluation: out[k] = sum_j h[j] * x[(half + k*down - j)/up]
|
| 85 |
+
base = half + np.arange(n_out, dtype=np.int64) * down
|
| 86 |
+
m0 = (base - hl + 1 + up - 1) // up
|
| 87 |
+
m0 = np.maximum(m0, 0)
|
| 88 |
+
acc = np.zeros(n_out, dtype=np.float64)
|
| 89 |
+
for t in range(hl // up + 2):
|
| 90 |
+
m = m0 + t
|
| 91 |
+
j = base - m * up
|
| 92 |
+
valid = (j >= 0) & (j < hl) & (m < len(x))
|
| 93 |
+
acc[valid] += h[j[valid]] * x[m[valid]]
|
| 94 |
+
return acc.astype(np.float32)
|
| 95 |
|
| 96 |
|
| 97 |
def resample_to(x, sr_in, sr_out):
|
python/fastwave_sdk/pipeline.py
CHANGED
|
@@ -12,6 +12,7 @@ import os
|
|
| 12 |
import numpy as np
|
| 13 |
|
| 14 |
from . import audio as A
|
|
|
|
| 15 |
|
| 16 |
ASSETS = os.path.join(os.path.dirname(os.path.abspath(__file__)), "assets")
|
| 17 |
|
|
@@ -96,7 +97,7 @@ class FastWaveSR:
|
|
| 96 |
return np.clip(x[0], -1.0, 1.0 - 1e-7)
|
| 97 |
|
| 98 |
# --------------------------------------------------------------- pipeline
|
| 99 |
-
def enhance(self, wav, sr_in, progress=None, normalize=True):
|
| 100 |
"""wav: 1-D float32 at sr_in. Returns 48 kHz float32."""
|
| 101 |
wav = np.asarray(wav, dtype=np.float32)
|
| 102 |
peak = float(np.max(np.abs(wav))) if len(wav) else 0.0
|
|
@@ -123,7 +124,10 @@ class FastWaveSR:
|
|
| 123 |
np.ascontiguousarray(seg.reshape(1, -1)), band3))
|
| 124 |
if progress:
|
| 125 |
progress(i + 1, len(starts))
|
| 126 |
-
|
|
|
|
|
|
|
|
|
|
| 127 |
starts = A.chunk_bounds(n, self.chunk, self.hop)
|
| 128 |
outs = []
|
| 129 |
for i, s in enumerate(starts):
|
|
@@ -134,5 +138,9 @@ class FastWaveSR:
|
|
| 134 |
if progress:
|
| 135 |
progress(i + 1, len(starts))
|
| 136 |
if len(starts) == 1:
|
| 137 |
-
|
| 138 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 12 |
import numpy as np
|
| 13 |
|
| 14 |
from . import audio as A
|
| 15 |
+
from .postfilter import denoise as _denoise
|
| 16 |
|
| 17 |
ASSETS = os.path.join(os.path.dirname(os.path.abspath(__file__)), "assets")
|
| 18 |
|
|
|
|
| 97 |
return np.clip(x[0], -1.0, 1.0 - 1e-7)
|
| 98 |
|
| 99 |
# --------------------------------------------------------------- pipeline
|
| 100 |
+
def enhance(self, wav, sr_in, progress=None, normalize=True, denoise=True, denoise_alpha=1.0):
|
| 101 |
"""wav: 1-D float32 at sr_in. Returns 48 kHz float32."""
|
| 102 |
wav = np.asarray(wav, dtype=np.float32)
|
| 103 |
peak = float(np.max(np.abs(wav))) if len(wav) else 0.0
|
|
|
|
| 124 |
np.ascontiguousarray(seg.reshape(1, -1)), band3))
|
| 125 |
if progress:
|
| 126 |
progress(i + 1, len(starts))
|
| 127 |
+
y = np.concatenate(outs)[:n].astype(np.float32)
|
| 128 |
+
if denoise:
|
| 129 |
+
y = _denoise(y, self.sr_out, alpha=denoise_alpha)
|
| 130 |
+
return y * peak
|
| 131 |
starts = A.chunk_bounds(n, self.chunk, self.hop)
|
| 132 |
outs = []
|
| 133 |
for i, s in enumerate(starts):
|
|
|
|
| 138 |
if progress:
|
| 139 |
progress(i + 1, len(starts))
|
| 140 |
if len(starts) == 1:
|
| 141 |
+
y = outs[0][:n].astype(np.float32)
|
| 142 |
+
else:
|
| 143 |
+
y = A.overlap_add(outs, starts, n, self.chunk)
|
| 144 |
+
if denoise:
|
| 145 |
+
y = _denoise(y, self.sr_out, alpha=denoise_alpha)
|
| 146 |
+
return y * peak
|
python/fastwave_sdk/postfilter.py
ADDED
|
@@ -0,0 +1,41 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Light spectral-subtraction post filter for the FastWave output hiss (numpy).
|
| 2 |
+
|
| 3 |
+
The diffusion model leaves a broadband noise floor (mostly above the input band).
|
| 4 |
+
We estimate the noise magnitude per frequency from the quietest frames of the
|
| 5 |
+
output itself and apply a smoothed Wiener-like gain.
|
| 6 |
+
"""
|
| 7 |
+
import numpy as np
|
| 8 |
+
|
| 9 |
+
from .audio import hann
|
| 10 |
+
|
| 11 |
+
|
| 12 |
+
def denoise(x, sr=48000, n_fft=1024, hop=256, alpha=1.2, floor=0.10, quiet_percentile=20,
|
| 13 |
+
smooth=3):
|
| 14 |
+
x = np.asarray(x, dtype=np.float32)
|
| 15 |
+
if len(x) < n_fft:
|
| 16 |
+
return x
|
| 17 |
+
win = hann(n_fft)
|
| 18 |
+
m = 1 + (len(x) - n_fft) // hop
|
| 19 |
+
idx = np.arange(n_fft)[None, :] + hop * np.arange(m)[:, None]
|
| 20 |
+
fr = x[idx] * win
|
| 21 |
+
X = np.fft.rfft(fr, axis=1)
|
| 22 |
+
mag = np.abs(X)
|
| 23 |
+
|
| 24 |
+
noise = np.percentile(mag, quiet_percentile, axis=0)
|
| 25 |
+
gain = 1.0 - alpha * noise[None, :] / np.maximum(mag, 1e-8)
|
| 26 |
+
gain = np.clip(gain, floor, 1.0)
|
| 27 |
+
if smooth > 1 and m >= smooth:
|
| 28 |
+
k = np.ones(smooth) / smooth
|
| 29 |
+
gain = np.apply_along_axis(lambda c: np.convolve(c, k, mode="same"), 0, gain)
|
| 30 |
+
|
| 31 |
+
Y = X * gain
|
| 32 |
+
frames = np.fft.irfft(Y, n=n_fft, axis=1) * win
|
| 33 |
+
|
| 34 |
+
out_len = (m - 1) * hop + n_fft
|
| 35 |
+
acc = np.zeros(out_len, dtype=np.float64)
|
| 36 |
+
wsum = np.zeros(out_len, dtype=np.float64)
|
| 37 |
+
for i in range(m):
|
| 38 |
+
acc[i * hop: i * hop + n_fft] += frames[i]
|
| 39 |
+
wsum[i * hop: i * hop + n_fft] += win ** 2
|
| 40 |
+
acc /= np.maximum(wsum, 1e-8)
|
| 41 |
+
return acc[: len(x)].astype(np.float32)
|
reports/performance.md
CHANGED
|
@@ -7,9 +7,9 @@
|
|
| 7 |
| 量化 | U16 激活 / S8 权重 / U16 输出,MinMax 标定,Pulsar2 8.0,`compiler.check=2` |
|
| 8 |
| 单次 NFE(chunk 32768 = 0.68 s @48k) | **499.8 ms**(min 499.5 / max 500.1,n=10) |
|
| 9 |
| 8 步 / chunk | 3.99 s |
|
| 10 |
-
| 端到端 RTF(8 步,50% overlap) | **10.
|
| 11 |
-
| 端到端 RTF(4 步,50% overlap) | **5.
|
| 12 |
-
| 端到端 RTF(8 步,`--overlap 0`) | **6.
|
| 13 |
|
| 14 |
## 说明
|
| 15 |
|
|
@@ -17,18 +17,25 @@
|
|
| 17 |
- 8 步与 4 步共用同一个 axmodel(sigma/系数由 host 提供),`--steps 4` 直接可用。
|
| 18 |
- 端到端耗时 ≈ 音频时长 × 2(50% overlap 的重叠系数)× NFE 次数 × 0.5 s。
|
| 19 |
|
| 20 |
-
##
|
| 21 |
|
| 22 |
-
|
| 23 |
-
|
| 24 |
-
(
|
| 25 |
-
|
| 26 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 27 |
|
| 28 |
可选的优化方向(按收益排序):
|
| 29 |
|
| 30 |
-
1. **换成 int8 激活**(S8/U8)再测一次:同尺寸探针 U8 比 U16 快 2.8×,
|
| 31 |
-
分支的动态范围较大,需要重新看精度;
|
| 32 |
2. 增大 chunk(如 65536)减少每秒的算子启动次数;
|
| 33 |
3. 把 FFT 的 32×32 MatMul 换成一次更大的 GEMM(合并 stage1/stage2 的批维);
|
| 34 |
4. 用 `compiler.npu_perf` / `dump_simulator_cycle_trace` 做逐算子定位后再定点优化。
|
|
|
|
| 7 |
| 量化 | U16 激活 / S8 权重 / U16 输出,MinMax 标定,Pulsar2 8.0,`compiler.check=2` |
|
| 8 |
| 单次 NFE(chunk 32768 = 0.68 s @48k) | **499.8 ms**(min 499.5 / max 500.1,n=10) |
|
| 9 |
| 8 步 / chunk | 3.99 s |
|
| 10 |
+
| 端到端 RTF(8 步,50% overlap) | **10.97**(C++)/ **10.96**(Python) |
|
| 11 |
+
| 端到端 RTF(4 步,50% overlap) | **5.51**(C++)/ **5.49**(Python) |
|
| 12 |
+
| 端到端 RTF(8 步,`--overlap 0`) | **6.29**(C++,无重叠拼接) |
|
| 13 |
|
| 14 |
## 说明
|
| 15 |
|
|
|
|
| 17 |
- 8 步与 4 步共用同一个 axmodel(sigma/系数由 host 提供),`--steps 4` 直接可用。
|
| 18 |
- 端到端耗时 ≈ 音频时长 × 2(50% overlap 的重叠系数)× NFE 次数 × 0.5 s。
|
| 19 |
|
| 20 |
+
## 单 NFE 0.5 s 的构成(实测探针)
|
| 21 |
|
| 22 |
+
| 组成 | 15 份合计 |
|
| 23 |
+
|---|---|
|
| 24 |
+
| Conv(12.8 GMAC,同尺寸 1x1 Conv 探针 ≈ 570 GMAC/s) | ~25 ms |
|
| 25 |
+
| STFT(15 份) | 15 ms |
|
| 26 |
+
| iSTFT(15 份) | 37 ms |
|
| 27 |
+
| sigmoid×tanh(128×32768) | 31 ms |
|
| 28 |
+
| GRN(64×32768) | 22 ms |
|
| 29 |
+
| 其余逐元素 / reshape / gather / overlap-add(15 个 FFC 块,2600+ 个 op) | ~370 ms |
|
| 30 |
+
|
| 31 |
+
单次 NFE 的真正瓶颈不是卷积,而是 15 个 FFC 块里 STFT/iSTFT 带来的大量
|
| 32 |
+
4.2 M 元素级逐元素算子(每 NFE 约 4 GB DDR 访存)。图是严格串行的,
|
| 33 |
+
AX650 的三核无法并行(实测只有 core0 在工作,多 session 并行也无加速)。
|
| 34 |
|
| 35 |
可选的优化方向(按收益排序):
|
| 36 |
|
| 37 |
+
1. **换成 int8 激活**(S8/U8)再测一次:同尺寸探针 U8 比 U16 快 2.8×,访存减半,
|
| 38 |
+
但 FFT 分支的动态范围较大,需要重新看精度;
|
| 39 |
2. 增大 chunk(如 65536)减少每秒的算子启动次数;
|
| 40 |
3. 把 FFT 的 32×32 MatMul 换成一次更大的 GEMM(合并 stage1/stage2 的批维);
|
| 41 |
4. 用 `compiler.npu_perf` / `dump_simulator_cycle_trace` 做逐算子定位后再定点优化。
|
run_ax650.sh
CHANGED
|
@@ -1,7 +1,30 @@
|
|
| 1 |
-
#!/
|
| 2 |
-
# FastWave.AXERA
|
| 3 |
-
#
|
| 4 |
set -e
|
| 5 |
-
|
| 6 |
-
export
|
| 7 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/bin/bash
|
| 2 |
+
# FastWave.AXERA one-click demo: any sample rate -> 48 kHz.
|
| 3 |
+
# bash run_ax650.sh [input.wav] [output.wav] [--cpp|--python] [--steps N] [--overlap R] [--no-denoise]
|
| 4 |
set -e
|
| 5 |
+
cd "$(dirname "$0")"
|
| 6 |
+
export LD_LIBRARY_PATH=/soc/lib:${LD_LIBRARY_PATH:-}
|
| 7 |
+
export PYTHONDONTWRITEBYTECODE=1
|
| 8 |
+
|
| 9 |
+
INPUT="${1:-samples/listen/8k/OSR_us_000_0010_input_8k.wav}"
|
| 10 |
+
OUTPUT="${2:-output.wav}"
|
| 11 |
+
if [ $# -ge 2 ]; then shift 2; else shift $#; fi
|
| 12 |
+
|
| 13 |
+
MODE="cpp"
|
| 14 |
+
EXTRA=()
|
| 15 |
+
for a in "$@"; do
|
| 16 |
+
case "$a" in
|
| 17 |
+
--python) MODE="python" ;;
|
| 18 |
+
--cpp) MODE="cpp" ;;
|
| 19 |
+
*) EXTRA+=("$a") ;;
|
| 20 |
+
esac
|
| 21 |
+
done
|
| 22 |
+
if [ ! -x ./bin/fastwave_ax650 ]; then MODE="python"; fi
|
| 23 |
+
|
| 24 |
+
if [ "$MODE" = "cpp" ]; then
|
| 25 |
+
exec ./bin/fastwave_ax650 models/fastwave_nfe.axmodel "$INPUT" "$OUTPUT" \
|
| 26 |
+
--assets models/assets "${EXTRA[@]}"
|
| 27 |
+
else
|
| 28 |
+
exec python3 python/demo.py "$INPUT" "$OUTPUT" \
|
| 29 |
+
--model models/fastwave_nfe.axmodel "${EXTRA[@]}"
|
| 30 |
+
fi
|
samples/listen/48k_from16k/speech_2_output_48k.wav
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
size 491564
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:62f6293d207e9a5354f0c3effb3584bf60d955e3fe46e6f04c3e36ae82bf2620
|
| 3 |
size 491564
|
samples/listen/48k_from16k/speech_3_output_48k.wav
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
size 491564
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:f0f9fb0b54872aafeb90600281451e6b2553c0cdc166d8bb48c0fdaa4b4e45a5
|
| 3 |
size 491564
|
samples/listen/48k_from8k/OSR_cn_000_0072_output_48k.wav
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
size 1916936
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:de924b54d7dea511a501b02217da8317584c99d2826552d5f0db701adca27ebb
|
| 3 |
size 1916936
|
samples/listen/48k_from8k/OSR_cn_000_0074_output_48k.wav
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
size 2207708
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:e75d1598afe5383d0ee9dbb95d5f90f8998efbe5741c72e042996dadf07165e3
|
| 3 |
size 2207708
|
samples/listen/48k_from8k/OSR_us_000_0010_output_48k.wav
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
size 3227864
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:3b873ccfba276b27977287cb672a62dfe5976004ece9d5fa0be2d54c0e14b1f2
|
| 3 |
size 3227864
|
samples/listen/48k_from8k/OSR_us_000_0013_output_48k.wav
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
size 3124160
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:38ff41ac246aad0a45d45e0c7ac4ad85b27383b596ebb057892ca13baca639bf
|
| 3 |
size 3124160
|
samples/listen/spectrograms/OSR_cn_000_0072.png
CHANGED
|
Git LFS Details
|
|
Git LFS Details
|
samples/listen/spectrograms/OSR_cn_000_0072_input_spec.png
ADDED
|
Git LFS Details
|
samples/listen/spectrograms/OSR_cn_000_0072_output_spec.png
ADDED
|
Git LFS Details
|
samples/listen/spectrograms/OSR_cn_000_0074.png
ADDED
|
Git LFS Details
|
samples/listen/spectrograms/OSR_cn_000_0074_input_spec.png
ADDED
|
Git LFS Details
|
samples/listen/spectrograms/OSR_cn_000_0074_output_spec.png
ADDED
|
Git LFS Details
|
samples/listen/spectrograms/OSR_us_000_0010.png
CHANGED
|
Git LFS Details
|
|
Git LFS Details
|
samples/listen/spectrograms/OSR_us_000_0010_input_spec.png
ADDED
|
Git LFS Details
|
samples/listen/spectrograms/OSR_us_000_0010_output_spec.png
ADDED
|
Git LFS Details
|
samples/listen/spectrograms/OSR_us_000_0013.png
ADDED
|
Git LFS Details
|
samples/listen/spectrograms/OSR_us_000_0013_input_spec.png
ADDED
|
Git LFS Details
|
samples/listen/spectrograms/OSR_us_000_0013_output_spec.png
ADDED
|
Git LFS Details
|
samples/listen/spectrograms/speech_2.png
CHANGED
|
Git LFS Details
|
|
Git LFS Details
|
samples/listen/spectrograms/speech_2_input_spec.png
ADDED
|
Git LFS Details
|
samples/listen/spectrograms/speech_2_output_spec.png
ADDED
|
Git LFS Details
|
samples/listen/spectrograms/speech_3.png
CHANGED
|
Git LFS Details
|
|
Git LFS Details
|
samples/listen/spectrograms/speech_3_input_spec.png
ADDED
|
Git LFS Details
|
samples/listen/spectrograms/speech_3_output_spec.png
ADDED
|
Git LFS Details
|