HY-2012 commited on
Commit
69261fe
·
verified ·
1 Parent(s): adb2846

FastWave.AXERA: AX650 NPU audio super-resolution (U16)

Browse files
Files changed (38) hide show
  1. .gitattributes +14 -0
  2. README.md +61 -36
  3. bin/fastwave_ax650 +0 -0
  4. models/assets/emb_p1_b.npy +3 -0
  5. models/assets/emb_p1_w.npy +3 -0
  6. models/assets/emb_p2_b.npy +3 -0
  7. models/assets/emb_p2_w.npy +3 -0
  8. python/benchmark.py +2 -1
  9. python/demo.py +3 -1
  10. python/fastwave_sdk/audio.py +13 -12
  11. python/fastwave_sdk/pipeline.py +12 -4
  12. python/fastwave_sdk/postfilter.py +41 -0
  13. reports/performance.md +18 -11
  14. run_ax650.sh +29 -6
  15. samples/listen/48k_from16k/speech_2_output_48k.wav +1 -1
  16. samples/listen/48k_from16k/speech_3_output_48k.wav +1 -1
  17. samples/listen/48k_from8k/OSR_cn_000_0072_output_48k.wav +1 -1
  18. samples/listen/48k_from8k/OSR_cn_000_0074_output_48k.wav +1 -1
  19. samples/listen/48k_from8k/OSR_us_000_0010_output_48k.wav +1 -1
  20. samples/listen/48k_from8k/OSR_us_000_0013_output_48k.wav +1 -1
  21. samples/listen/spectrograms/OSR_cn_000_0072.png +2 -2
  22. samples/listen/spectrograms/OSR_cn_000_0072_input_spec.png +3 -0
  23. samples/listen/spectrograms/OSR_cn_000_0072_output_spec.png +3 -0
  24. samples/listen/spectrograms/OSR_cn_000_0074.png +3 -0
  25. samples/listen/spectrograms/OSR_cn_000_0074_input_spec.png +3 -0
  26. samples/listen/spectrograms/OSR_cn_000_0074_output_spec.png +3 -0
  27. samples/listen/spectrograms/OSR_us_000_0010.png +2 -2
  28. samples/listen/spectrograms/OSR_us_000_0010_input_spec.png +3 -0
  29. samples/listen/spectrograms/OSR_us_000_0010_output_spec.png +3 -0
  30. samples/listen/spectrograms/OSR_us_000_0013.png +3 -0
  31. samples/listen/spectrograms/OSR_us_000_0013_input_spec.png +3 -0
  32. samples/listen/spectrograms/OSR_us_000_0013_output_spec.png +3 -0
  33. samples/listen/spectrograms/speech_2.png +2 -2
  34. samples/listen/spectrograms/speech_2_input_spec.png +3 -0
  35. samples/listen/spectrograms/speech_2_output_spec.png +3 -0
  36. samples/listen/spectrograms/speech_3.png +2 -2
  37. samples/listen/spectrograms/speech_3_input_spec.png +3 -0
  38. samples/listen/spectrograms/speech_3_output_spec.png +3 -0
.gitattributes CHANGED
@@ -50,3 +50,17 @@ samples/listen/spectrograms/OSR_cn_000_0072.png filter=lfs diff=lfs merge=lfs -t
50
  samples/listen/spectrograms/OSR_us_000_0010.png filter=lfs diff=lfs merge=lfs -text
51
  samples/listen/spectrograms/speech_2.png filter=lfs diff=lfs merge=lfs -text
52
  samples/listen/spectrograms/speech_3.png filter=lfs diff=lfs merge=lfs -text
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
50
  samples/listen/spectrograms/OSR_us_000_0010.png filter=lfs diff=lfs merge=lfs -text
51
  samples/listen/spectrograms/speech_2.png filter=lfs diff=lfs merge=lfs -text
52
  samples/listen/spectrograms/speech_3.png filter=lfs diff=lfs merge=lfs -text
53
+ samples/listen/spectrograms/OSR_cn_000_0072_input_spec.png filter=lfs diff=lfs merge=lfs -text
54
+ samples/listen/spectrograms/OSR_cn_000_0072_output_spec.png filter=lfs diff=lfs merge=lfs -text
55
+ samples/listen/spectrograms/OSR_cn_000_0074.png filter=lfs diff=lfs merge=lfs -text
56
+ samples/listen/spectrograms/OSR_cn_000_0074_input_spec.png filter=lfs diff=lfs merge=lfs -text
57
+ samples/listen/spectrograms/OSR_cn_000_0074_output_spec.png filter=lfs diff=lfs merge=lfs -text
58
+ samples/listen/spectrograms/OSR_us_000_0010_input_spec.png filter=lfs diff=lfs merge=lfs -text
59
+ samples/listen/spectrograms/OSR_us_000_0010_output_spec.png filter=lfs diff=lfs merge=lfs -text
60
+ samples/listen/spectrograms/OSR_us_000_0013.png filter=lfs diff=lfs merge=lfs -text
61
+ samples/listen/spectrograms/OSR_us_000_0013_input_spec.png filter=lfs diff=lfs merge=lfs -text
62
+ samples/listen/spectrograms/OSR_us_000_0013_output_spec.png filter=lfs diff=lfs merge=lfs -text
63
+ samples/listen/spectrograms/speech_2_input_spec.png filter=lfs diff=lfs merge=lfs -text
64
+ samples/listen/spectrograms/speech_2_output_spec.png filter=lfs diff=lfs merge=lfs -text
65
+ samples/listen/spectrograms/speech_3_input_spec.png filter=lfs diff=lfs merge=lfs -text
66
+ samples/listen/spectrograms/speech_3_output_spec.png filter=lfs diff=lfs merge=lfs -text
README.md CHANGED
@@ -10,7 +10,6 @@ tags:
10
  - audio-super-resolution
11
  - speech-super-resolution
12
  - bandwidth-extension
13
- - diffuse
14
  - audio-to-audio
15
  library_name: axera
16
  pipeline_tag: audio-to-audio
@@ -19,63 +18,89 @@ base_model: Nikait/FastWave
19
 
20
  # FastWave.AXERA
21
 
22
- **AX650 上的通用音频超分(6–48 kHz → 48 kHz),单模型 3.8 MB,纯 NPU 推理,即取即用。**
23
 
24
- FastWave 是 1.3 M 参数的扩散式音频超分模型(EDM preconditioning + NU-Wave 2 骨干,
25
- [论文](https://arxiv.org/abs/2603.04122))。本仓库提供 AX650 NPU 版:
26
 
27
- - `models/fastwave_nfe.axmodel`:量化后的单次 NFE 图(11.9 MB,U16 激活 / S8 权重),板端 8 步采样
28
- - `python/`:完整流水线 SDK(分块 + overlap-add + EDM 采样),只依赖 `numpy` + `axengine`
29
- - `samples/`:8 k / 16 k 输入与 48 k 输出示例
30
- - `reports/`:精度与性能报告
31
 
32
- - GitHub(源码 + 复现流程):https://github.com/ZY-2012/FastWave.AXERA
33
- - 上游:https://github.com/Nikait/FastWave
 
 
 
 
 
 
34
 
35
  ## 快速开始(AX650 板端)
36
 
 
 
37
  ```bash
38
- # 需要:python3 + numpy + axengine(AX650 BSP 镜像自带)
39
  bash run_ax650.sh samples/listen/8k/OSR_us_000_0010_input_8k.wav output.wav
 
 
 
 
 
40
  bash run_ax650.sh samples/listen/16k/speech_2_input_16k.wav output.wav --steps 4
 
41
  ```
42
 
43
- 输出固定 48 kHz。`--steps 4` 速度更快、质量略降;`--overlap 0.5`(默认)可改拼接重叠率。
44
 
45
  ## 结果
46
 
47
- | 输入 | 输出 |
48
- |---|---|
49
- | [▶ 8k OSR_us_000_0010](samples/listen/8k/OSR_us_000_0010_input_8k.wav) | [▶ 48k](samples/listen/48k_from8k/OSR_us_000_0010_output_48k.wav) |
50
- | [▶ 8k OSR_cn_000_0072](samples/listen/8k/OSR_cn_000_0072_input_8k.wav) | [▶ 48k](samples/listen/48k_from8k/OSR_cn_000_0072_output_48k.wav) |
51
- | [▶ 16k speech_2](samples/listen/16k/speech_2_input_16k.wav) | [▶ 48k](samples/listen/48k_from16k/speech_2_output_48k.wav) |
52
 
53
- 频谱对比:[OSR_us_000_0010](samples/listen/spectrograms/OSR_us_000_0010.png) ·
54
- [OSR_cn_000_0072](samples/listen/spectrograms/OSR_cn_000_0072.png) ·
55
- [speech_2](samples/listen/spectrograms/speech_2.png)
 
 
 
56
 
57
- ## 性能(AX650 NPU3,chunk 0.68 s)
58
 
59
- | 配置 | 单次 NFE | 端到端 RTF |
60
  |---|---|---|
61
- | 8 步 + 50% overlap(默认) | 499.8 ms | **10.0** |
62
- | 4 步 + 50% overlap(`--steps 4`) | 499.8 ms | **5.0** |
63
 
64
- ## 精度
65
 
66
- 量化配置:Pulsar2 8.0,**U16 激活 / S8 权重 / U16 输出**(`DEFAULT` 全图),MinMax 标定
67
- (真实 48 kHz 语音,8/12/16/24 kHz 退化输入,8 个采样步全覆盖,64 个样本),`compiler.check=2`。
68
- AXMODEL 与浮点 ONNX 的单步 NFE 余弦相似度见 `reports/accuracy.md`(最低 TBD)。
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
69
 
70
- ## 实现要点
71
 
72
- 1. 模型内部每层都有 STFT/iSTFT(FFC 全局分支)。AX NPU 不支持 FFT/复数算子,这里把
73
- `torch.stft/istft` 等价改写为 32×32 Cooley-Tukey 矩阵乘 FFT(reflect padding、hann 窗、
74
- overlap-add、窗包络归一化全部保留),与 torch 误差 < 1e-7,图上只有 MatMul/Add/Mul/Reshape。
75
- 2. EDM 的 `c_skip/c_in/c_out` 与正弦扩散 embedding 只依赖 sigma,host 端预计算后作为图输入,
76
- 图上没有 log/sqrt/sin/cos/OneHot。
77
- 3. 输入采样率通过 band one-hot 输入,一个模型覆盖 6–48 kHz。
78
 
79
  ## License
80
 
81
- Apache-2.0(本仓库代码与量化模型)。上游 FastWave 模型为 MIT License。
 
10
  - audio-super-resolution
11
  - speech-super-resolution
12
  - bandwidth-extension
 
13
  - audio-to-audio
14
  library_name: axera
15
  pipeline_tag: audio-to-audio
 
18
 
19
  # FastWave.AXERA
20
 
21
+ **AX650 上的通用音频超分:6–48 kHz → 48 kHz。C++ 全链路 RTF 10.97(8 步)/ 5.51(4 步),Python SDK RTF 10.96 / 5.49;单 axmodel 11.9 MB,Python/C++ SDK 即取即用。**
22
 
23
+ ![RTF](https://img.shields.io/badge/RTF%20%28C%2B%2B%29-10.97-brightgreen) ![Size](https://img.shields.io/badge/axmodel-11.9%20MB-green) ![License](https://img.shields.io/badge/license-Apache--2.0-blue)
 
24
 
25
+ - HuggingFace: https://huggingface.co/HY-2012/FastWave.AXERA
26
+ - GitHub: https://github.com/ZY-2012/FastWave.AXERA
27
+ - 上游: https://github.com/Nikait/FastWave
 
28
 
29
+ ## 目录
30
+
31
+ - 快速开始
32
+ - 结果
33
+ - 推理速度
34
+ - 文件说明
35
+ - 数据来源
36
+ - License
37
 
38
  ## 快速开始(AX650 板端)
39
 
40
+ 本仓库即完整包(`models/` + `bin/` + `python/` + `samples/`),下载后在包根目录运行:
41
+
42
  ```bash
43
+ # C++(默认)
44
  bash run_ax650.sh samples/listen/8k/OSR_us_000_0010_input_8k.wav output.wav
45
+
46
+ # Python SDK
47
+ bash run_ax650.sh samples/listen/8k/OSR_us_000_0010_input_8k.wav output.wav --python
48
+
49
+ # 4 步采样(更快)/ 关闭后置降噪
50
  bash run_ax650.sh samples/listen/16k/speech_2_input_16k.wav output.wav --steps 4
51
+ bash run_ax650.sh samples/listen/8k/OSR_cn_000_0072_input_8k.wav output.wav --no-denoise
52
  ```
53
 
54
+ 输入支持 6–48 kHz,输出固定 48 kHz。
55
 
56
  ## 结果
57
 
58
+ ### 8 kHz 输入(Open Speech Repository)
 
 
 
 
59
 
60
+ | 版本 | 输入 | FastWave.AXERA 输出 |
61
+ |---|---|---|
62
+ | **OSR_us_000_0010** | [▶](samples/listen/8k/OSR_us_000_0010_input_8k.wav)<br><img src="samples/listen/spectrograms/OSR_us_000_0010_input_spec.png" width="360"> | [▶](samples/listen/48k_from8k/OSR_us_000_0010_output_48k.wav)<br><img src="samples/listen/spectrograms/OSR_us_000_0010_output_spec.png" width="360"> |
63
+ | **OSR_us_000_0013** | [▶](samples/listen/8k/OSR_us_000_0013_input_8k.wav)<br><img src="samples/listen/spectrograms/OSR_us_000_0013_input_spec.png" width="360"> | [▶](samples/listen/48k_from8k/OSR_us_000_0013_output_48k.wav)<br><img src="samples/listen/spectrograms/OSR_us_000_0013_output_spec.png" width="360"> |
64
+ | **OSR_cn_000_0072** | [▶](samples/listen/8k/OSR_cn_000_0072_input_8k.wav)<br><img src="samples/listen/spectrograms/OSR_cn_000_0072_input_spec.png" width="360"> | [▶](samples/listen/48k_from8k/OSR_cn_000_0072_output_48k.wav)<br><img src="samples/listen/spectrograms/OSR_cn_000_0072_output_spec.png" width="360"> |
65
+ | **OSR_cn_000_0074** | [▶](samples/listen/8k/OSR_cn_000_0074_input_8k.wav)<br><img src="samples/listen/spectrograms/OSR_cn_000_0074_input_spec.png" width="360"> | [▶](samples/listen/48k_from8k/OSR_cn_000_0074_output_48k.wav)<br><img src="samples/listen/spectrograms/OSR_cn_000_0074_output_spec.png" width="360"> |
66
 
67
+ ### 16 kHz 输入(人声)
68
 
69
+ | 版本 | 输入 | FastWave.AXERA 输出 |
70
  |---|---|---|
71
+ | **speech_2** | [▶](samples/listen/16k/speech_2_input_16k.wav)<br><img src="samples/listen/spectrograms/speech_2_input_spec.png" width="360"> | [▶](samples/listen/48k_from16k/speech_2_output_48k.wav)<br><img src="samples/listen/spectrograms/speech_2_output_spec.png" width="360"> |
72
+ | **speech_3** | [▶](samples/listen/16k/speech_3_input_16k.wav)<br><img src="samples/listen/spectrograms/speech_3_input_spec.png" width="360"> | [▶](samples/listen/48k_from16k/speech_3_output_48k.wav)<br><img src="samples/listen/spectrograms/speech_3_output_spec.png" width="360"> |
73
 
74
+ ### 8k → 48k 频谱对比(上:输入,下:输出)
75
 
76
+ <img src="samples/listen/spectrograms/OSR_us_000_0010.png" width="720">
77
+
78
+ ## 推理速度:RTF
79
+
80
+ AX650(NPU3,BSP V3.10.2),chunk 32768 点(0.68 s):
81
+
82
+ | 链路 | 8 步 | 4 步 | 无重叠(8 步) |
83
+ |---|---:|---:|---:|
84
+ | C++ | 10.97 | 5.51 | 6.29 |
85
+ | Python SDK | 10.96 | 5.49 | — |
86
+
87
+ ## 文件说明
88
+
89
+ | 路径 | 内容 |
90
+ |---|---|
91
+ | `models/fastwave_nfe.axmodel` | 单次 NFE 图(11.9 MB,U16 激活 / S8 权重 / U16 输出) |
92
+ | `models/assets/*.npy` | 扩散 embedding 权重(host 端计算) |
93
+ | `bin/fastwave_ax650` | C++ 推理程序(aarch64,AX650) |
94
+ | `python/` | Python SDK(numpy + axengine) |
95
+ | `samples/` | 8k / 16k 输入与 48k 输出、频谱图 |
96
+ | `reports/` | 精度与性能报告 |
97
 
98
+ ## 数据来源
99
 
100
+ - Open Speech Repository(English / Chinese 8k):http://voiptroubleshooter.com/open_speech/index.html
101
+ - 16 kHz 参照:公开人声样例经 48k → 16k 降采样
102
+ - 上游模型权重:Nikait/FastWave(MIT)
 
 
 
103
 
104
  ## License
105
 
106
+ Apache-2.0。
bin/fastwave_ax650 ADDED
Binary file (76.6 kB). View file
 
models/assets/emb_p1_b.npy ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:56eb0ba80672dd4b474a72cf665645481fca907e35b89defdf48c9aa6623e9cb
3
+ size 2176
models/assets/emb_p1_w.npy ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:472e7ed0456762e5bcf2f4e47a609b9cacdd7c895e17a6a90dbc6f7a5b60eb47
3
+ size 262272
models/assets/emb_p2_b.npy ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:197f365eb799bfc44aa437800f443f7d4796c832cd120db7a39291e909f42cdc
3
+ size 2176
models/assets/emb_p2_w.npy ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a5db3ac68049e055d8b17e56691fb0e8e822543673f4005b1d0aa7066aa39d2e
3
+ size 1048704
python/benchmark.py CHANGED
@@ -22,9 +22,10 @@ def main():
22
  ap.add_argument("--seconds", type=float, default=3.0)
23
  ap.add_argument("--steps", type=int, default=8)
24
  ap.add_argument("--iters", type=int, default=20)
 
25
  a = ap.parse_args()
26
 
27
- m = FastWaveSR(a.model, num_steps=a.steps)
28
  wav, sr = A.read_wav(a.wav)
29
  wav = wav[: int(a.seconds * sr)]
30
  low = A.low_quality(wav / (np.abs(wav).max() + 1e-8), sr)
 
22
  ap.add_argument("--seconds", type=float, default=3.0)
23
  ap.add_argument("--steps", type=int, default=8)
24
  ap.add_argument("--iters", type=int, default=20)
25
+ ap.add_argument("--overlap", type=float, default=0.5)
26
  a = ap.parse_args()
27
 
28
+ m = FastWaveSR(a.model, num_steps=a.steps, overlap=a.overlap)
29
  wav, sr = A.read_wav(a.wav)
30
  wav = wav[: int(a.seconds * sr)]
31
  low = A.low_quality(wav / (np.abs(wav).max() + 1e-8), sr)
python/demo.py CHANGED
@@ -21,6 +21,7 @@ def main():
21
  ap.add_argument("--overlap", type=float, default=0.5, help="chunk overlap ratio")
22
  ap.add_argument("--steps", type=int, default=8, help="EDM sampling steps (4 or 8)")
23
  ap.add_argument("--seed", type=int, default=1234)
 
24
  args = ap.parse_args()
25
 
26
  wav, sr = A.read_wav(args.input)
@@ -28,7 +29,8 @@ def main():
28
  model = FastWaveSR(args.model, num_steps=args.steps, chunk=args.chunk,
29
  overlap=args.overlap, seed=args.seed)
30
  t0 = time.time()
31
- out = model.enhance(wav, sr, progress=lambda i, n: print(f"\r[chunk ] {i}/{n}", end="", flush=True))
 
32
  dt = time.time() - t0
33
  print()
34
  A.write_wav(args.output, out, 48000)
 
21
  ap.add_argument("--overlap", type=float, default=0.5, help="chunk overlap ratio")
22
  ap.add_argument("--steps", type=int, default=8, help="EDM sampling steps (4 or 8)")
23
  ap.add_argument("--seed", type=int, default=1234)
24
+ ap.add_argument("--no-denoise", action="store_true", help="disable the output noise post filter")
25
  args = ap.parse_args()
26
 
27
  wav, sr = A.read_wav(args.input)
 
29
  model = FastWaveSR(args.model, num_steps=args.steps, chunk=args.chunk,
30
  overlap=args.overlap, seed=args.seed)
31
  t0 = time.time()
32
+ out = model.enhance(wav, sr, denoise=not args.no_denoise,
33
+ progress=lambda i, n: print(f"\r[chunk ] {i}/{n}", end="", flush=True))
34
  dt = time.time() - t0
35
  print()
36
  A.write_wav(args.output, out, 48000)
python/fastwave_sdk/audio.py CHANGED
@@ -78,19 +78,20 @@ def resample_poly(x, up, down):
78
  return x.astype(np.float32)
79
  max_rate = max(up, down)
80
  half = 10 * max_rate
81
- h = _firwin(1.0 / max_rate, 2 * half + 1) * up
82
- # upsample + full convolution
83
- y = np.zeros((len(x) - 1) * up + 1, dtype=np.float64)
84
- y[::up] = x
85
- y = np.convolve(y, h)
86
- # compensate the FIR group delay, then decimate
87
- y = y[half::down]
88
  n_out = int(math.ceil(len(x) * up / down))
89
- if len(y) >= n_out:
90
- y = y[:n_out]
91
- else:
92
- y = np.pad(y, (0, n_out - len(y)))
93
- return y.astype(np.float32)
 
 
 
 
 
 
94
 
95
 
96
  def resample_to(x, sr_in, sr_out):
 
78
  return x.astype(np.float32)
79
  max_rate = max(up, down)
80
  half = 10 * max_rate
81
+ h = (_firwin(1.0 / max_rate, 2 * half + 1) * up).astype(np.float64)
82
+ hl = len(h)
 
 
 
 
 
83
  n_out = int(math.ceil(len(x) * up / down))
84
+ # polyphase evaluation: out[k] = sum_j h[j] * x[(half + k*down - j)/up]
85
+ base = half + np.arange(n_out, dtype=np.int64) * down
86
+ m0 = (base - hl + 1 + up - 1) // up
87
+ m0 = np.maximum(m0, 0)
88
+ acc = np.zeros(n_out, dtype=np.float64)
89
+ for t in range(hl // up + 2):
90
+ m = m0 + t
91
+ j = base - m * up
92
+ valid = (j >= 0) & (j < hl) & (m < len(x))
93
+ acc[valid] += h[j[valid]] * x[m[valid]]
94
+ return acc.astype(np.float32)
95
 
96
 
97
  def resample_to(x, sr_in, sr_out):
python/fastwave_sdk/pipeline.py CHANGED
@@ -12,6 +12,7 @@ import os
12
  import numpy as np
13
 
14
  from . import audio as A
 
15
 
16
  ASSETS = os.path.join(os.path.dirname(os.path.abspath(__file__)), "assets")
17
 
@@ -96,7 +97,7 @@ class FastWaveSR:
96
  return np.clip(x[0], -1.0, 1.0 - 1e-7)
97
 
98
  # --------------------------------------------------------------- pipeline
99
- def enhance(self, wav, sr_in, progress=None, normalize=True):
100
  """wav: 1-D float32 at sr_in. Returns 48 kHz float32."""
101
  wav = np.asarray(wav, dtype=np.float32)
102
  peak = float(np.max(np.abs(wav))) if len(wav) else 0.0
@@ -123,7 +124,10 @@ class FastWaveSR:
123
  np.ascontiguousarray(seg.reshape(1, -1)), band3))
124
  if progress:
125
  progress(i + 1, len(starts))
126
- return np.concatenate(outs)[:n].astype(np.float32) * peak
 
 
 
127
  starts = A.chunk_bounds(n, self.chunk, self.hop)
128
  outs = []
129
  for i, s in enumerate(starts):
@@ -134,5 +138,9 @@ class FastWaveSR:
134
  if progress:
135
  progress(i + 1, len(starts))
136
  if len(starts) == 1:
137
- return outs[0][:n].astype(np.float32) * peak
138
- return A.overlap_add(outs, starts, n, self.chunk) * peak
 
 
 
 
 
12
  import numpy as np
13
 
14
  from . import audio as A
15
+ from .postfilter import denoise as _denoise
16
 
17
  ASSETS = os.path.join(os.path.dirname(os.path.abspath(__file__)), "assets")
18
 
 
97
  return np.clip(x[0], -1.0, 1.0 - 1e-7)
98
 
99
  # --------------------------------------------------------------- pipeline
100
+ def enhance(self, wav, sr_in, progress=None, normalize=True, denoise=True, denoise_alpha=1.0):
101
  """wav: 1-D float32 at sr_in. Returns 48 kHz float32."""
102
  wav = np.asarray(wav, dtype=np.float32)
103
  peak = float(np.max(np.abs(wav))) if len(wav) else 0.0
 
124
  np.ascontiguousarray(seg.reshape(1, -1)), band3))
125
  if progress:
126
  progress(i + 1, len(starts))
127
+ y = np.concatenate(outs)[:n].astype(np.float32)
128
+ if denoise:
129
+ y = _denoise(y, self.sr_out, alpha=denoise_alpha)
130
+ return y * peak
131
  starts = A.chunk_bounds(n, self.chunk, self.hop)
132
  outs = []
133
  for i, s in enumerate(starts):
 
138
  if progress:
139
  progress(i + 1, len(starts))
140
  if len(starts) == 1:
141
+ y = outs[0][:n].astype(np.float32)
142
+ else:
143
+ y = A.overlap_add(outs, starts, n, self.chunk)
144
+ if denoise:
145
+ y = _denoise(y, self.sr_out, alpha=denoise_alpha)
146
+ return y * peak
python/fastwave_sdk/postfilter.py ADDED
@@ -0,0 +1,41 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Light spectral-subtraction post filter for the FastWave output hiss (numpy).
2
+
3
+ The diffusion model leaves a broadband noise floor (mostly above the input band).
4
+ We estimate the noise magnitude per frequency from the quietest frames of the
5
+ output itself and apply a smoothed Wiener-like gain.
6
+ """
7
+ import numpy as np
8
+
9
+ from .audio import hann
10
+
11
+
12
+ def denoise(x, sr=48000, n_fft=1024, hop=256, alpha=1.2, floor=0.10, quiet_percentile=20,
13
+ smooth=3):
14
+ x = np.asarray(x, dtype=np.float32)
15
+ if len(x) < n_fft:
16
+ return x
17
+ win = hann(n_fft)
18
+ m = 1 + (len(x) - n_fft) // hop
19
+ idx = np.arange(n_fft)[None, :] + hop * np.arange(m)[:, None]
20
+ fr = x[idx] * win
21
+ X = np.fft.rfft(fr, axis=1)
22
+ mag = np.abs(X)
23
+
24
+ noise = np.percentile(mag, quiet_percentile, axis=0)
25
+ gain = 1.0 - alpha * noise[None, :] / np.maximum(mag, 1e-8)
26
+ gain = np.clip(gain, floor, 1.0)
27
+ if smooth > 1 and m >= smooth:
28
+ k = np.ones(smooth) / smooth
29
+ gain = np.apply_along_axis(lambda c: np.convolve(c, k, mode="same"), 0, gain)
30
+
31
+ Y = X * gain
32
+ frames = np.fft.irfft(Y, n=n_fft, axis=1) * win
33
+
34
+ out_len = (m - 1) * hop + n_fft
35
+ acc = np.zeros(out_len, dtype=np.float64)
36
+ wsum = np.zeros(out_len, dtype=np.float64)
37
+ for i in range(m):
38
+ acc[i * hop: i * hop + n_fft] += frames[i]
39
+ wsum[i * hop: i * hop + n_fft] += win ** 2
40
+ acc /= np.maximum(wsum, 1e-8)
41
+ return acc[: len(x)].astype(np.float32)
reports/performance.md CHANGED
@@ -7,9 +7,9 @@
7
  | 量化 | U16 激活 / S8 权重 / U16 输出,MinMax 标定,Pulsar2 8.0,`compiler.check=2` |
8
  | 单次 NFE(chunk 32768 = 0.68 s @48k) | **499.8 ms**(min 499.5 / max 500.1,n=10) |
9
  | 8 步 / chunk | 3.99 s |
10
- | 端到端 RTF(8 步,50% overlap) | **10.0**(2.00 s 音频 20.07 s) |
11
- | 端到端 RTF(4 步,50% overlap) | **5.0**(2.00 s 音频 10.06 s) |
12
- | 端到端 RTF(8 步,`--overlap 0`) | **6.0**(2.00 s 音频 12.04 s,无重叠拼接) |
13
 
14
  ## 说明
15
 
@@ -17,18 +17,25 @@
17
  - 8 步与 4 步共用同一个 axmodel(sigma/系数由 host 提供),`--steps 4` 直接可用。
18
  - 端到端耗时 ≈ 音频时长 × 2(50% overlap 的重叠系数)× NFE 次数 × 0.5 s。
19
 
20
- ## 为什么是 0.5 s / NFE
21
 
22
- 单次 NFE 的卷积算力约 12.8 GMAC,按 AX650 U16 实测(同尺寸 1x1 Conv 探针
23
- ≈ 570 GMAC/s)只需 ~22 ms;其余 ~480 ms 花在 15 个 FFC 块的 STFT/iSTFT
24
- (约 180 个 32×32 小 MatMul + 大量 32 通道逐元素算子,共 2600+ 个 op)上。
25
- 该图是典型的"小张量、多算子"结构,受 DDR 带宽与算子启动开销限制,
26
- NPU 只跑在 core0(图是严格串行的,三核无法并行)。
 
 
 
 
 
 
 
27
 
28
  可选的优化方向(按收益排序):
29
 
30
- 1. **换成 int8 激活**(S8/U8)再测一次:同尺寸探针 U8 比 U16 快 2.8×,但 FFT
31
- 分支的动态范围较大,需要重新看精度;
32
  2. 增大 chunk(如 65536)减少每秒的算子启动次数;
33
  3. 把 FFT 的 32×32 MatMul 换成一次更大的 GEMM(合并 stage1/stage2 的批维);
34
  4. 用 `compiler.npu_perf` / `dump_simulator_cycle_trace` 做逐算子定位后再定点优化。
 
7
  | 量化 | U16 激活 / S8 权重 / U16 输出,MinMax 标定,Pulsar2 8.0,`compiler.check=2` |
8
  | 单次 NFE(chunk 32768 = 0.68 s @48k) | **499.8 ms**(min 499.5 / max 500.1,n=10) |
9
  | 8 步 / chunk | 3.99 s |
10
+ | 端到端 RTF(8 步,50% overlap) | **10.97**(C++)/ **10.96**(Python) |
11
+ | 端到端 RTF(4 步,50% overlap) | **5.51**(C++)/ **5.49**(Python) |
12
+ | 端到端 RTF(8 步,`--overlap 0`) | **6.29**(C++,无重叠拼接) |
13
 
14
  ## 说明
15
 
 
17
  - 8 步与 4 步共用同一个 axmodel(sigma/系数由 host 提供),`--steps 4` 直接可用。
18
  - 端到端耗时 ≈ 音频时长 × 2(50% overlap 的重叠系数)× NFE 次数 × 0.5 s。
19
 
20
+ ## 单 NFE 0.5 s 的构成(实测探针)
21
 
22
+ | 组成 | 15 份合计 |
23
+ |---|---|
24
+ | Conv(12.8 GMAC,同尺寸 1x1 Conv 探针 ≈ 570 GMAC/s) | ~25 ms |
25
+ | STFT(15 份) | 15 ms |
26
+ | iSTFT(15 份) | 37 ms |
27
+ | sigmoid×tanh(128×32768) | 31 ms |
28
+ | GRN(64×32768) | 22 ms |
29
+ | 其余逐元素 / reshape / gather / overlap-add(15 个 FFC 块,2600+ 个 op) | ~370 ms |
30
+
31
+ 单次 NFE 的真正瓶颈不是卷积,而是 15 个 FFC 块里 STFT/iSTFT 带来的大量
32
+ 4.2 M 元素级逐元素算子(每 NFE 约 4 GB DDR 访存)。图是严格串行的,
33
+ AX650 的三核无法并行(实测只有 core0 在工作,多 session 并行也无加速)。
34
 
35
  可选的优化方向(按收益排序):
36
 
37
+ 1. **换成 int8 激活**(S8/U8)再测一次:同尺寸探针 U8 比 U16 快 2.8×,访存减半,
38
+ 但 FFT 分支的动态范围较大,需要重新看精度;
39
  2. 增大 chunk(如 65536)减少每秒的算子启动次数;
40
  3. 把 FFT 的 32×32 MatMul 换成一次更大的 GEMM(合并 stage1/stage2 的批维);
41
  4. 用 `compiler.npu_perf` / `dump_simulator_cycle_trace` 做逐算子定位后再定点优化。
run_ax650.sh CHANGED
@@ -1,7 +1,30 @@
1
- #!/usr/bin/env bash
2
- # FastWave.AXERA on AX650: any sample rate -> 48 kHz
3
- # usage: bash run_ax650.sh input.wav output.wav [extra demo.py args]
4
  set -e
5
- ROOT="$(cd "$(dirname "$0")" && pwd)"
6
- export PYTHONPATH="$ROOT/python:$PYTHONPATH"
7
- exec python3 "$ROOT/python/demo.py" "$@"
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/bin/bash
2
+ # FastWave.AXERA one-click demo: any sample rate -> 48 kHz.
3
+ # bash run_ax650.sh [input.wav] [output.wav] [--cpp|--python] [--steps N] [--overlap R] [--no-denoise]
4
  set -e
5
+ cd "$(dirname "$0")"
6
+ export LD_LIBRARY_PATH=/soc/lib:${LD_LIBRARY_PATH:-}
7
+ export PYTHONDONTWRITEBYTECODE=1
8
+
9
+ INPUT="${1:-samples/listen/8k/OSR_us_000_0010_input_8k.wav}"
10
+ OUTPUT="${2:-output.wav}"
11
+ if [ $# -ge 2 ]; then shift 2; else shift $#; fi
12
+
13
+ MODE="cpp"
14
+ EXTRA=()
15
+ for a in "$@"; do
16
+ case "$a" in
17
+ --python) MODE="python" ;;
18
+ --cpp) MODE="cpp" ;;
19
+ *) EXTRA+=("$a") ;;
20
+ esac
21
+ done
22
+ if [ ! -x ./bin/fastwave_ax650 ]; then MODE="python"; fi
23
+
24
+ if [ "$MODE" = "cpp" ]; then
25
+ exec ./bin/fastwave_ax650 models/fastwave_nfe.axmodel "$INPUT" "$OUTPUT" \
26
+ --assets models/assets "${EXTRA[@]}"
27
+ else
28
+ exec python3 python/demo.py "$INPUT" "$OUTPUT" \
29
+ --model models/fastwave_nfe.axmodel "${EXTRA[@]}"
30
+ fi
samples/listen/48k_from16k/speech_2_output_48k.wav CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:c002dc9bc7a43585ccfb470e867caaab845d62dad0bfff3e272995389b561725
3
  size 491564
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:62f6293d207e9a5354f0c3effb3584bf60d955e3fe46e6f04c3e36ae82bf2620
3
  size 491564
samples/listen/48k_from16k/speech_3_output_48k.wav CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:36de030f5ac7c574273c8799faeb119ea40bae6126b18f2fc727224f89de9a25
3
  size 491564
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f0f9fb0b54872aafeb90600281451e6b2553c0cdc166d8bb48c0fdaa4b4e45a5
3
  size 491564
samples/listen/48k_from8k/OSR_cn_000_0072_output_48k.wav CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:e2b28b841347e66f82a076918d998690091e6e3b24dc86ef2cfea2a976203a55
3
  size 1916936
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:de924b54d7dea511a501b02217da8317584c99d2826552d5f0db701adca27ebb
3
  size 1916936
samples/listen/48k_from8k/OSR_cn_000_0074_output_48k.wav CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:e75ef0dc777a70732748b62288d87580bf493911e206e853337f4efe3272a8d0
3
  size 2207708
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e75d1598afe5383d0ee9dbb95d5f90f8998efbe5741c72e042996dadf07165e3
3
  size 2207708
samples/listen/48k_from8k/OSR_us_000_0010_output_48k.wav CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:689ae6a81c4944091c315bf958b2054cc5ebc830798f2d8d13b27743e03e850f
3
  size 3227864
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3b873ccfba276b27977287cb672a62dfe5976004ece9d5fa0be2d54c0e14b1f2
3
  size 3227864
samples/listen/48k_from8k/OSR_us_000_0013_output_48k.wav CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:6f5706bcb90ef65a6c7fa3c445abe9852b242c4eca722aa47273900df7a8920e
3
  size 3124160
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:38ff41ac246aad0a45d45e0c7ac4ad85b27383b596ebb057892ca13baca639bf
3
  size 3124160
samples/listen/spectrograms/OSR_cn_000_0072.png CHANGED

Git LFS Details

  • SHA256: 71fa7b0ee0093cfbe2c1a7b85f1bd119b200c3ff2cf231dd569cc5f5d5833944
  • Pointer size: 131 Bytes
  • Size of remote file: 568 kB

Git LFS Details

  • SHA256: 21703a0cf7e9e82906e6381cb73a6e703e4e2073457eede89699a53adeac7909
  • Pointer size: 131 Bytes
  • Size of remote file: 598 kB
samples/listen/spectrograms/OSR_cn_000_0072_input_spec.png ADDED

Git LFS Details

  • SHA256: c3675560e9c7ea02a2052d0d476bbd7ab95291b1180abdb2305b273793757abe
  • Pointer size: 131 Bytes
  • Size of remote file: 268 kB
samples/listen/spectrograms/OSR_cn_000_0072_output_spec.png ADDED

Git LFS Details

  • SHA256: dce8b7446e310dcef3af714fdb20a3b638a76419492ae5d76452220c387a9d40
  • Pointer size: 131 Bytes
  • Size of remote file: 285 kB
samples/listen/spectrograms/OSR_cn_000_0074.png ADDED

Git LFS Details

  • SHA256: acfbaa739c0d04496c149d6347e7a0e4181c1042411b944cd4e2fea84db67ab7
  • Pointer size: 131 Bytes
  • Size of remote file: 590 kB
samples/listen/spectrograms/OSR_cn_000_0074_input_spec.png ADDED

Git LFS Details

  • SHA256: bb344f1de2d646fec72ef1c5f37eafed0f65d8154f8fd9028dcb4117ff623cb2
  • Pointer size: 131 Bytes
  • Size of remote file: 271 kB
samples/listen/spectrograms/OSR_cn_000_0074_output_spec.png ADDED

Git LFS Details

  • SHA256: c8b2b4d04cdafc9c109672c05e8abcbdaf3db1f8c7ff0d5c7ca078d23c3ba147
  • Pointer size: 131 Bytes
  • Size of remote file: 271 kB
samples/listen/spectrograms/OSR_us_000_0010.png CHANGED

Git LFS Details

  • SHA256: 67ec7bfafddccc3a43b1b5ae1525503966b6d7321ec0295ffd620c80314e2fac
  • Pointer size: 131 Bytes
  • Size of remote file: 552 kB

Git LFS Details

  • SHA256: 7887fffc55da92549bbc7d642cdadcdce1512027e02d66f1b32c0bb60e295d90
  • Pointer size: 131 Bytes
  • Size of remote file: 561 kB
samples/listen/spectrograms/OSR_us_000_0010_input_spec.png ADDED

Git LFS Details

  • SHA256: 339836bbf12e865764feb861bf136460023a7b90392e993208afc1fa9e35a850
  • Pointer size: 131 Bytes
  • Size of remote file: 265 kB
samples/listen/spectrograms/OSR_us_000_0010_output_spec.png ADDED

Git LFS Details

  • SHA256: 0926708754e69cb1cf858f4c0c1a61c51d99997e5bcad27b7eec1e1575af081b
  • Pointer size: 131 Bytes
  • Size of remote file: 242 kB
samples/listen/spectrograms/OSR_us_000_0013.png ADDED

Git LFS Details

  • SHA256: 2d9a60e768ecc94dba007e638f0ff52e8e215a9d93aeee4e815a0bde74820f6e
  • Pointer size: 131 Bytes
  • Size of remote file: 573 kB
samples/listen/spectrograms/OSR_us_000_0013_input_spec.png ADDED

Git LFS Details

  • SHA256: b29ee75033169c6590711d0c48b5d8ac5afac44782d3f599965b067d4b99d731
  • Pointer size: 131 Bytes
  • Size of remote file: 261 kB
samples/listen/spectrograms/OSR_us_000_0013_output_spec.png ADDED

Git LFS Details

  • SHA256: f23b37910e056ceb39017d098a9bc4d08b75ccb31f4d440ebfbe37eb5d844618
  • Pointer size: 131 Bytes
  • Size of remote file: 256 kB
samples/listen/spectrograms/speech_2.png CHANGED

Git LFS Details

  • SHA256: 925678e5b235e050e4d24a2f3339c424c308fb820af6a1dc5e3483fdc07e36d3
  • Pointer size: 131 Bytes
  • Size of remote file: 503 kB

Git LFS Details

  • SHA256: d76a6afff41e970b09137b34124a5d0333edf8e0ed73987028deece7885c92a0
  • Pointer size: 131 Bytes
  • Size of remote file: 498 kB
samples/listen/spectrograms/speech_2_input_spec.png ADDED

Git LFS Details

  • SHA256: 7eb19d5b60cd83b712dd329f158c42c66accde06e30ebb0628d58789ce286497
  • Pointer size: 131 Bytes
  • Size of remote file: 194 kB
samples/listen/spectrograms/speech_2_output_spec.png ADDED

Git LFS Details

  • SHA256: 64bd5323088d12be5fbb435e5374e328e93324188f997273a45cca37c8a09ace
  • Pointer size: 131 Bytes
  • Size of remote file: 284 kB
samples/listen/spectrograms/speech_3.png CHANGED

Git LFS Details

  • SHA256: 14923e8c4bccbd1a4310aa9e4d66f383bd382cd56350079837d3355b1e9e5cbf
  • Pointer size: 131 Bytes
  • Size of remote file: 461 kB

Git LFS Details

  • SHA256: 3ff3b90e5dbf83864b4399f099b989ee4d47eece21cba851095bd164d481ad0a
  • Pointer size: 131 Bytes
  • Size of remote file: 438 kB
samples/listen/spectrograms/speech_3_input_spec.png ADDED

Git LFS Details

  • SHA256: a85aeced87432d0b8f5a21f2eee56cd7a9a063dd7c15d68324038aaa45de8b97
  • Pointer size: 131 Bytes
  • Size of remote file: 154 kB
samples/listen/spectrograms/speech_3_output_spec.png ADDED

Git LFS Details

  • SHA256: e628010dfcd1e9a00cce97c48ae3dfd92951ad1c20746affcb6c9c5eea4d7a8d
  • Pointer size: 131 Bytes
  • Size of remote file: 268 kB