AudioSR.AXERA AX650 models (UNet, VAE, split vocoder)
Browse files- .gitattributes +12 -0
- README.md +127 -0
- model_convert/audiosr_ddim_10.npz +3 -0
- models/README.md +25 -0
- models/audiosr_unet.axmodel +3 -0
- models/audiosr_vae_decoder.axmodel +3 -0
- models/audiosr_vae_encoder_nhwc.axmodel +3 -0
- models/voc_pre_fp32.axmodel +3 -0
- models/voc_split_nhwc_post_fp32.axmodel +3 -0
- models/voc_split_nhwc_up2_mix.axmodel +3 -0
- models/voc_split_nhwc_up3_mix.axmodel +3 -0
- models/voc_split_nhwc_up4_mix.axmodel +3 -0
- models/voc_up0a_mix.axmodel +3 -0
- models/voc_up0b_mix.axmodel +3 -0
- models/voc_up1a_mix.axmodel +3 -0
- models/voc_up1b_mix.axmodel +3 -0
.gitattributes
CHANGED
|
@@ -33,3 +33,15 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
+
models/audiosr_unet.axmodel filter=lfs diff=lfs merge=lfs -text
|
| 37 |
+
models/audiosr_vae_decoder.axmodel filter=lfs diff=lfs merge=lfs -text
|
| 38 |
+
models/audiosr_vae_encoder_nhwc.axmodel filter=lfs diff=lfs merge=lfs -text
|
| 39 |
+
models/voc_pre_fp32.axmodel filter=lfs diff=lfs merge=lfs -text
|
| 40 |
+
models/voc_split_nhwc_post_fp32.axmodel filter=lfs diff=lfs merge=lfs -text
|
| 41 |
+
models/voc_split_nhwc_up2_mix.axmodel filter=lfs diff=lfs merge=lfs -text
|
| 42 |
+
models/voc_split_nhwc_up3_mix.axmodel filter=lfs diff=lfs merge=lfs -text
|
| 43 |
+
models/voc_split_nhwc_up4_mix.axmodel filter=lfs diff=lfs merge=lfs -text
|
| 44 |
+
models/voc_up0a_mix.axmodel filter=lfs diff=lfs merge=lfs -text
|
| 45 |
+
models/voc_up0b_mix.axmodel filter=lfs diff=lfs merge=lfs -text
|
| 46 |
+
models/voc_up1a_mix.axmodel filter=lfs diff=lfs merge=lfs -text
|
| 47 |
+
models/voc_up1b_mix.axmodel filter=lfs diff=lfs merge=lfs -text
|
README.md
ADDED
|
@@ -0,0 +1,127 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# AudioSR.AXERA
|
| 2 |
+
|
| 3 |
+
**AX650 / NPU3 语音超分:低采样率语音 → 48 kHz。完整链路 ONNX 拆分 + Pulsar2 混合量化,vocoder 板端相似度 0.9984,完整链路 vs PyTorch 0.9917。**
|
| 4 |
+
|
| 5 |
+
   [](https://huggingface.co/HY-2012/AudioSR.AXERA)
|
| 6 |
+
|
| 7 |
+
- HuggingFace(模型从这里下载):<https://huggingface.co/HY-2012/AudioSR.AXERA>
|
| 8 |
+
- GitHub:<https://github.com/ZY-2012/AudioSR_AXERA>
|
| 9 |
+
- 上游:<https://github.com/haoheliu/versatile_audio_super_resolution>
|
| 10 |
+
- 输入:8 / 16 / 24 / 48 kHz 单声道或立体声 WAV;输出:48 kHz
|
| 11 |
+
- 5.12 s 输入,AudioSR DDIM-10;vocoder 拆成 9 个小模型以适配 AX650 内存和精度
|
| 12 |
+
|
| 13 |
+
## 目录
|
| 14 |
+
|
| 15 |
+
- 快速开始
|
| 16 |
+
- 结果
|
| 17 |
+
- 推理速度
|
| 18 |
+
- 模型
|
| 19 |
+
- 模型转换
|
| 20 |
+
- 数据来源
|
| 21 |
+
- License
|
| 22 |
+
|
| 23 |
+
## 快速开始(AX650 板端)
|
| 24 |
+
|
| 25 |
+
从 HuggingFace 下载整个仓库,或只下载 `models/`,放在包根目录旁:
|
| 26 |
+
|
| 27 |
+
```bash
|
| 28 |
+
# 模型目录结构
|
| 29 |
+
# AudioSR_AXERA/
|
| 30 |
+
# └── models/
|
| 31 |
+
# ├── audiosr_unet.axmodel
|
| 32 |
+
# ├── audiosr_vae_encoder_nhwc.axmodel
|
| 33 |
+
# ├── audiosr_vae_decoder.axmodel
|
| 34 |
+
# └── voc_*.axmodel
|
| 35 |
+
|
| 36 |
+
pip install -r requirements.txt
|
| 37 |
+
|
| 38 |
+
python python/demo.py \
|
| 39 |
+
samples/listen/16k/speech_2_input_16k.wav \
|
| 40 |
+
-o output_48k.wav
|
| 41 |
+
```
|
| 42 |
+
|
| 43 |
+
输入 8 kHz / 16 kHz 均可;SDK 会重采样到 48 kHz,并按 AudioSR 的 5.12 s 窗口处理后送 NPU。
|
| 44 |
+
|
| 45 |
+
## 工程结构
|
| 46 |
+
|
| 47 |
+
```text
|
| 48 |
+
AudioSR_AXERA/
|
| 49 |
+
├── model_convert/ # ONNX 导出 → 拆分 → 校准数据 → Pulsar2 U16 量化
|
| 50 |
+
├── python/ # Python 板端推理 SDK(axengine)
|
| 51 |
+
├── cpp/ # C++ 板端推理源码 + 交叉编译脚本
|
| 52 |
+
├── models/ # 仅 README,量化模型从 HuggingFace 下载
|
| 53 |
+
├── samples/ # 中英文示例输入
|
| 54 |
+
└── requirements.txt
|
| 55 |
+
```
|
| 56 |
+
|
| 57 |
+
## 结果
|
| 58 |
+
|
| 59 |
+
### 精度
|
| 60 |
+
|
| 61 |
+
| 模块 / 链路 | 板端 vs ONNX/PyTorch |
|
| 62 |
+
|---|---:|
|
| 63 |
+
| VAE encoder(NHWC) | 0.99975 |
|
| 64 |
+
| UNet | 0.99989 |
|
| 65 |
+
| VAE decoder | 0.99999 |
|
| 66 |
+
| vocoder 9-model chain | **0.99838** |
|
| 67 |
+
| full AudioSR chain | **0.99169** |
|
| 68 |
+
|
| 69 |
+
### 示例
|
| 70 |
+
|
| 71 |
+
| 输入 | 输出 |
|
| 72 |
+
|---|---|
|
| 73 |
+
| [English 16 kHz](samples/listen/16k/speech_2_input_16k.wav) | [48 kHz](samples/listen/output/speech_2_output_48k.wav) |
|
| 74 |
+
| [English 8 kHz](samples/listen/8k/OSR_us_000_0010_input_8k.wav) | 运行 SDK 生成 |
|
| 75 |
+
| [中文 8 kHz](samples/listen/8k/OSR_cn_000_0072_input_8k.wav) | 运行 SDK 生成 |
|
| 76 |
+
|
| 77 |
+
## 推理速度
|
| 78 |
+
|
| 79 |
+
5.12 s 输入、DDIM-10、Python SDK(AX650 NPU3):
|
| 80 |
+
|
| 81 |
+
```text
|
| 82 |
+
约 20–22 s / 5.12 s 音频,RTF ≈ 4.0–4.3
|
| 83 |
+
```
|
| 84 |
+
|
| 85 |
+
短音频主要受 DDIM 10 步 UNet 和 vocoder 拆分模型调度开销影响;这是当前 Python 参考实现的实测值。
|
| 86 |
+
|
| 87 |
+
## 模型
|
| 88 |
+
|
| 89 |
+
模型不放进 GitHub,统一从 HuggingFace 下载:
|
| 90 |
+
|
| 91 |
+
- `audiosr_unet.axmodel`:273 MB
|
| 92 |
+
- `audiosr_vae_encoder_nhwc.axmodel`:102 MB
|
| 93 |
+
- `audiosr_vae_decoder.axmodel`:143 MB
|
| 94 |
+
- `voc_pre_fp32.axmodel`:11 MB
|
| 95 |
+
- `voc_up0a_mix.axmodel`:55 MB
|
| 96 |
+
- `voc_up0b_mix.axmodel`:202 MB
|
| 97 |
+
- `voc_up1a_mix.axmodel`:12 MB
|
| 98 |
+
- `voc_up1b_mix.axmodel`:52 MB
|
| 99 |
+
- `voc_split_nhwc_up2_mix.axmodel`:17 MB
|
| 100 |
+
- `voc_split_nhwc_up3_mix.axmodel`:7.7 MB
|
| 101 |
+
- `voc_split_nhwc_up4_mix.axmodel`:5.8 MB
|
| 102 |
+
- `voc_split_nhwc_post_fp32.axmodel`:99 KB
|
| 103 |
+
|
| 104 |
+
详见 [`models/README.md`](models/README.md)。
|
| 105 |
+
|
| 106 |
+
## 模型转换
|
| 107 |
+
|
| 108 |
+
```bash
|
| 109 |
+
# 1. 上游 AudioSR 代码 / checkpoint
|
| 110 |
+
# 2. 导出 VAE encoder / UNet / decoder / vocoder 子图
|
| 111 |
+
# 3. 校准数据
|
| 112 |
+
# 4. Pulsar2 8.0 量化
|
| 113 |
+
PULSAR2_IMAGE=docker-registry.aitsw.axera-tech.com/pulsar2:8.0 \
|
| 114 |
+
bash model_convert/compile.sh voc_pre_fp32 voc_up0a_mix ...
|
| 115 |
+
```
|
| 116 |
+
|
| 117 |
+
具体命令、BF16/U16/S8 配置和各阶段校准见 [`model_convert/README.md`](model_convert/README.md)。
|
| 118 |
+
|
| 119 |
+
## 数据来源
|
| 120 |
+
|
| 121 |
+
- 上游 AudioSR:<https://github.com/haoheliu/versatile_audio_super_resolution>
|
| 122 |
+
- 示例英文 / 中文语音:Open Speech Repository、公开 16 kHz 人声样例
|
| 123 |
+
- 示例输出由本项目 AX650 Python SDK 生成
|
| 124 |
+
|
| 125 |
+
## License
|
| 126 |
+
|
| 127 |
+
AudioSR 上游代码与模型 checkpoint 遵循其原始许可证;本仓库的转换脚本和 SDK 以 Apache-2.0 提供。模型权重请遵循上游许可证。
|
model_convert/audiosr_ddim_10.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:ef451a92d0b86fc9999e66fbaf99f5312a4676d54963e96a7b7387480559ee86
|
| 3 |
+
size 10074
|
models/README.md
ADDED
|
@@ -0,0 +1,25 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Models
|
| 2 |
+
|
| 3 |
+
GitHub 仓库不包含 axmodel。请从 HuggingFace 下载:
|
| 4 |
+
|
| 5 |
+
<https://huggingface.co/HY-2012/AudioSR.AXERA/tree/main/models>
|
| 6 |
+
|
| 7 |
+
需要的文件:
|
| 8 |
+
|
| 9 |
+
```text
|
| 10 |
+
audiosr_unet.axmodel
|
| 11 |
+
audiosr_vae_encoder_nhwc.axmodel
|
| 12 |
+
audiosr_vae_decoder.axmodel
|
| 13 |
+
voc_pre_fp32.axmodel
|
| 14 |
+
voc_up0a_mix.axmodel
|
| 15 |
+
voc_up0b_mix.axmodel
|
| 16 |
+
voc_up1a_mix.axmodel
|
| 17 |
+
voc_up1b_mix.axmodel
|
| 18 |
+
voc_split_nhwc_up2_mix.axmodel
|
| 19 |
+
voc_split_nhwc_up3_mix.axmodel
|
| 20 |
+
voc_split_nhwc_up4_mix.axmodel
|
| 21 |
+
voc_split_nhwc_post_fp32.axmodel
|
| 22 |
+
```
|
| 23 |
+
|
| 24 |
+
所有模型均为 Pulsar2 8.0 / AX650 NPU3 编译。vocoder 采用拆分 + 混合精度:
|
| 25 |
+
`pre/post` 为 FP32,`up0/up1` 拆成两段,其余 up2/up3/up4 为混合精度。
|
models/audiosr_unet.axmodel
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:bddeb3a64d6bc60ffa58e20e4d93640c19f3dfc2a28f8ce49f0648289fbf6fcb
|
| 3 |
+
size 285800456
|
models/audiosr_vae_decoder.axmodel
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:4e0a8ea7a0415128860be4ba01347c5d693b3065b5e79195c4edaeced316fde5
|
| 3 |
+
size 149047475
|
models/audiosr_vae_encoder_nhwc.axmodel
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:9d2cbb9dbbbf29f5e945adf3e96bd0395e6fd1a79c54634b52fd30fc542b71a5
|
| 3 |
+
size 106617540
|
models/voc_pre_fp32.axmodel
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:44efd5c09a3077eeb1c8a38da058ffb28150e791a94b89c63b5dd93bba7d899c
|
| 3 |
+
size 11040377
|
models/voc_split_nhwc_post_fp32.axmodel
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:e29e568cb2d2cdbbeecd9eafdfe46bcc6e8fd46ed8511d69f32ae45a73bd0c07
|
| 3 |
+
size 101017
|
models/voc_split_nhwc_up2_mix.axmodel
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:884c8a57cad5a15dce9b87ffeecd7a6fa9b1325fe76dc3c78e2b695ab8e839ac
|
| 3 |
+
size 20096630
|
models/voc_split_nhwc_up3_mix.axmodel
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:f7952e2ba46b30c6e9e3ae68f50f3fe1cb239147342b63e3f6b5331a2f38db45
|
| 3 |
+
size 8025046
|
models/voc_split_nhwc_up4_mix.axmodel
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:17c4576ea8252720c01188f1876e27a801eb987f101dffb0b68f77f409817ec2
|
| 3 |
+
size 6074768
|
models/voc_up0a_mix.axmodel
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:29dafebef2300121f4fea0afb9992cd373d09791296407895b64332b1ce11052
|
| 3 |
+
size 56930619
|
models/voc_up0b_mix.axmodel
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:6861cc29b8247fc3cf3753e48352092bca47bc61e958f460623c5ff714b25f16
|
| 3 |
+
size 211010547
|
models/voc_up1a_mix.axmodel
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:9e3762be4918dc25fcb99ba9807269bb8924b2bd8c7d18d645d98513858acb3d
|
| 3 |
+
size 11983860
|
models/voc_up1b_mix.axmodel
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:043c0efd16293aacd6f704a2bd06b733304c410f6f6de78b2fee367c47a0c3c8
|
| 3 |
+
size 53545172
|