jennyxue commited on
Commit
afec037
·
verified ·
1 Parent(s): b9befd5

Model card and the original license

Browse files
Files changed (1) hide show
  1. README.md +119 -0
README.md ADDED
@@ -0,0 +1,119 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: alibabasglab/MossFormerGAN_SE_16K
3
+ base_model_relation: quantized
4
+ license: apache-2.0
5
+ pipeline_tag: audio-to-audio
6
+ library_name: onnx
7
+ tags:
8
+ - onnx
9
+ - moonbox
10
+ - onnxruntime
11
+ - mossformergan
12
+ - clearervoice
13
+ - speech-enhancement
14
+ - denoising
15
+ - noise-suppression
16
+ - 16khz
17
+ ---
18
+
19
+ # MossFormerGAN ONNX
20
+
21
+ ONNX builds of [MossFormerGAN](https://huggingface.co/alibabasglab/MossFormerGAN_SE_16K), made by Moonbox for on-device use with ONNX Runtime (CPU, CUDA, DirectML). Each
22
+ precision is one folder; what the graphs take and return, and the signal processing that goes around them, is
23
+ under *How to run*.
24
+
25
+ ## Files
26
+
27
+ The last column is the benchmark on the VoiceBank-DEMAND test set (824 noisy utterances; details under *How to
28
+ run*): each precision's speech-quality scores, and how far its output is from the original PyTorch model's.
29
+
30
+ | Precision | File | Size | sha256 | Benchmark |
31
+ |---|---|---|---|---|
32
+ | fp16 | [`fp16/model.onnx`](https://huggingface.co/moonbox-dev/MossFormerGAN_SE_16K-ONNX/blob/main/fp16/model.onnx) | 12.2 MB | `4047ebd56656` | PESQ 3.342, STOI 0.9570, SI-SDR 19.77 dB; vs fp32 59.8 / 57.2 dB; worst file -0.05 dB (CUDA) |
33
+
34
+ ## How to run
35
+
36
+ MossFormerGAN SE 16K (ClearerVoice-Studio's speech enhancement: removes noise from speech at 16 kHz; a MossFormer
37
+ generator trained with a metric discriminator) as one ONNX graph: the generator that turns the compressed noisy
38
+ spectrum into the enhanced one. Input `spec` [1, 2, T, 201] (real and imaginary parts of a 400-point STFT, hop 100,
39
+ Hamming window, power-law compressed by 0.3), outputs `real` and `imag` [1, 1, T, 201]. T is free, up to 8192 frames
40
+ (51 s) per call. The normalization, the STFT, the compression and the inverse STFT stay outside the graph, exactly as
41
+ in ClearerVoice's own code. So the simplest correct pipeline is ClearerVoice's decode function with its network
42
+ swapped for this graph:
43
+
44
+ ```python
45
+ import numpy as np, onnxruntime as ort, soundfile as sf, torch
46
+ import clearvoice.utils.decode as dec # pip install clearvoice==0.1.2 (only for its decode helper)
47
+ from clearvoice.network_wrapper import network_wrapper
48
+
49
+ w = network_wrapper(); w.model_name = "MossFormerGAN_SE_16K"; w.load_args_se()
50
+ args = w.args; args.network = "MossFormerGAN_SE_16K"
51
+ sess = ort.InferenceSession("fp16/model.onnx", providers=["CUDAExecutionProvider", "CPUExecutionProvider"])
52
+
53
+ class Net: # what decode() calls: net(spec) -> (real, imag)
54
+ def __call__(self, spec):
55
+ return [torch.from_numpy(o) for o in sess.run(None, {"spec": spec.numpy().astype(np.float32)})]
56
+ def eval(self):
57
+ return self
58
+
59
+ noisy, sr = sf.read("noisy_16k.wav", dtype="float32") # 16 kHz mono
60
+ clean = dec.decode_one_audio(Net(), torch.device("cpu"), noisy[None], args)
61
+ sf.write("enhanced.wav", np.asarray(clean, np.float32).reshape(-1)[: len(noisy)], 16000)
62
+ ```
63
+
64
+ `decode_one_audio` itself processes files longer than 10 s in 10 s windows. Attention runs over the whole window, so
65
+ memory grows with the square of its length: about 1 GB per attention step on a 10 s clip. On CUDA, let ONNX Runtime
66
+ shrink its arena after each call (run option `memory.enable_memory_arena_shrinkage` = `gpu:0`), or its buffers pile
67
+ up across clips of different lengths.
68
+
69
+ **Precisions.** `fp16/`: weights and math in fp16, inputs and outputs fp32, for GPU providers. `int8/`: weight-only
70
+ int8 (every MatMul, Gemm and Conv weight int8 per output channel, activations fp32). It runs on the CPU provider, but
71
+ there the model is about 7 times slower than real time on a laptop CPU (the PyTorch original: 4 times), so use a GPU
72
+ for anything but short clips. **No int4**: 4-bit MatMul weights put 459 of the 824 test files more than 0.5 dB below
73
+ the original (worst 3.5 dB), and the file would be only 0.6 MB smaller than `int8/`. No bf16 (no bf16 convolution in
74
+ ONNX Runtime). The 12 attention modules' identical rotary tables are stored once.
75
+
76
+ **Benchmark.** VoiceBank-DEMAND test set (Valentini et al.: 824 utterances by two unseen speakers), resampled to 16
77
+ kHz, enhanced with ClearerVoice's own decode code. PESQ is wide-band (ITU-T P.862.2), STOI and SI-SDR against the
78
+ clean speech. The PyTorch original (`clearvoice` 0.1.2, fp32) runs 4 times slower than real time on this laptop's CPU,
79
+ so it was run on the first 50 files. There, the ONNX fp32 graph on CUDA matches it at 77.3 dB mean (75.6 dB worst
80
+ file), and SI-SDR differs by at most 0.002 dB. The full set then compares against that fp32 graph. "vs fp32" is the
81
+ SI-SDR of each output against the fp32 graph's, mean / worst file; "worst file" is the largest SI-SDR drop on any one
82
+ file. Pass (the speech-enhancement threshold used across these repositories): no file more than 0.5 dB below.
83
+ Real-time factor on an RTX 3080 Laptop GPU, or with 4 CPU threads (Intel laptop CPU) for the CPU rows, whole pipeline.
84
+
85
+ | | Files | PESQ | STOI | SI-SDR (dB) | vs fp32 (dB) | Worst file | Real-time factor |
86
+ |---|---|---|---|---|---|---|---|
87
+ | noisy input | 824 | 1.97 | 0.921 | 8.39 | - | - | - |
88
+ | original (PyTorch fp32, CPU) | first 50 | 3.442 | 0.9485 | 20.81 | - | - | 3.9 |
89
+ | ONNX fp32 (CUDA, not published) | first 50 | 3.442 | 0.9485 | 20.81 | - | -0.00 dB vs original | - |
90
+ | ONNX fp32 (CUDA, not published) | 824 | 3.342 | 0.9570 | 19.77 | - | - | 0.18 |
91
+ | `fp16/` (CUDA) | 824 | 3.342 | 0.9570 | 19.77 | 59.8 / 57.2 | -0.05 dB | 0.20 |
92
+ | `int8/` (CUDA) | 824 | 3.344 | 0.9570 | 19.75 | 44.4 / 41.3 | -0.16 dB | 0.18 |
93
+ | `int8/` (CPU) | first 20 | 3.514 | 0.9710 | 21.19 | 43.6 / 41.1 | -0.08 dB | 7.0 |
94
+ | int4 (not published) | 824 | 3.334 | 0.9560 | 19.02 | 28.0 / 20.1 | -3.53 dB (459 files below -0.5) | 0.18 |
95
+
96
+ ## How these were made
97
+
98
+ From [MossFormerGAN](https://huggingface.co/alibabasglab/MossFormerGAN_SE_16K) at revision `dcb1b20e1e`, starting from an fp32 ONNX graph (where it came from is said above).
99
+ fp16: onnxconverter-common with inputs and outputs kept fp32, or, where the network's activations leave fp16's
100
+ range, fp16 weights with fp32 math (said above when so). int8: weight-only, every Conv / MatMul / GRU weight int8
101
+ with a per-output-channel scale behind DequantizeLinear and activations in fp32, so it runs on the CPU provider
102
+ without a calibration set. int4: ONNX Runtime's MatMulNBits (weight-only, block 32) on the MatMul weights, the rest
103
+ int8 as above, where the graph has enough MatMul weights to matter. No bf16: ONNX Runtime has no bf16 convolution or
104
+ GRU kernels. Every graph was benchmarked before upload; the `sha256` column above is what the files must hash to.
105
+
106
+ ## License
107
+
108
+ Apache-2.0. These files are a derivative of the original model and carry its license: it is copied in this
109
+ repository, and it governs your use of these files. The original model card follows unchanged.
110
+
111
+ ---
112
+
113
+ # Original model card
114
+
115
+ The MossFormerGAN_SE_16K model weights for 16 kHz speech enhancement in [ClearerVoice-Studio](https://github.com/modelscope/ClearerVoice-Studio/tree/main) repo.
116
+
117
+ This model is trained on large scale datasets inclduing open-sourced and private data.
118
+
119
+ It enhances speech audios by removing background noise.