jennyxue commited on
Commit
14dc429
·
verified ·
1 Parent(s): 7a980e6

Model card and the original license

Browse files
Files changed (1) hide show
  1. README.md +95 -0
README.md ADDED
@@ -0,0 +1,95 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: alibabasglab/FRCRN_SE_16K
3
+ base_model_relation: quantized
4
+ license: apache-2.0
5
+ pipeline_tag: audio-to-audio
6
+ library_name: mlx
7
+ tags:
8
+ - mlx
9
+ - moonbox
10
+ - mlx
11
+ - frcrn
12
+ - clearervoice
13
+ - speech-enhancement
14
+ - denoising
15
+ - 16khz
16
+ ---
17
+
18
+ # FRCRN MLX
19
+
20
+ MLX builds of [FRCRN](https://huggingface.co/alibabasglab/FRCRN_SE_16K), made by Moonbox for Apple Silicon Macs. Each precision is a folder; keep a folder's
21
+ files together. Only the parts listed below are converted: the rest of the pipeline comes from the original
22
+ repository unless the section below says otherwise.
23
+
24
+ ## Files
25
+
26
+ The last column is how far each build is from the original PyTorch model on the same inputs (see the section below
27
+ for what was measured).
28
+
29
+ | Precision | File | Size | sha256 | Checked against the original |
30
+ |---|---|---|---|---|
31
+ | bf16 | [`bf16/config.json`](https://huggingface.co/moonbox-dev/FRCRN_SE_16K-MLX/blob/main/bf16/config.json) | 0.1 kB | `86f0962100d6` | below threshold (experimental): SI-SDR 9.66 dB, DNSMOS 3.06; output vs the original's 16.9 / 2.9 dB |
32
+ | bf16 | [`bf16/mlx_model.py`](https://huggingface.co/moonbox-dev/FRCRN_SE_16K-MLX/blob/main/bf16/mlx_model.py) | 12.3 kB | `307afe87cb76` | below threshold (experimental): SI-SDR 9.66 dB, DNSMOS 3.06; output vs the original's 16.9 / 2.9 dB |
33
+ | bf16 | [`bf16/model.safetensors`](https://huggingface.co/moonbox-dev/FRCRN_SE_16K-MLX/blob/main/bf16/model.safetensors) | 29.7 MB | `4f4e44804287` | below threshold (experimental): SI-SDR 9.66 dB, DNSMOS 3.06; output vs the original's 16.9 / 2.9 dB |
34
+
35
+ ## How to run
36
+
37
+ FRCRN SE 16K (ClearerVoice-Studio's speech enhancement for 16 kHz speech: two complex-valued UNets with FSMN memory
38
+ and squeeze-excitation layers on a 640-point STFT) ported to MLX. Each folder holds `model.safetensors`, `config.json`
39
+ and `mlx_model.py` (the port). Needs `mlx`, `numpy`, `soundfile` (and `librosa` for input that isn't 16 kHz):
40
+
41
+ ```python
42
+ import os; os.environ["MLX_ENABLE_TF32"] = "0" # float32 matmuls (MLX uses TF32 otherwise)
43
+ import sys; sys.path.insert(0, "fp32") # a downloaded folder
44
+ import mlx_model, soundfile as sf
45
+
46
+ model = mlx_model.load("fp32")
47
+ clean = mlx_model.enhance(model, "noisy.wav") # float32, 16 kHz, the input's length
48
+ sf.write("clean.wav", clean, 16000)
49
+ ```
50
+
51
+ Like ClearerVoice, `enhance` normalizes the input's loudness first and scales the result back, and decodes inputs
52
+ over 120 s in 1 s windows. **Use `fp32/`.** The network is unusually sensitive to its weights (its BatchNorm statistics
53
+ reach 3e5): every folder computes in float32, but storing the weights in 16 bits (`bf16/`, `fp16/`) or quantizing the
54
+ FSMN layers (`8bit/`, `6bit/`, `4bit/`, on bf16 storage) already moves the output far enough to miss the threshold, so
55
+ those are marked experimental. BatchNorm parameters and the STFT kernels are float32 in every folder.
56
+
57
+ **How they were checked.** 8 noisy 16 kHz clips (LibriSpeech speech with babble or pink noise, 0-10 dB SNR), enhanced
58
+ by the original (ClearerVoice's `clearvoice` 0.1.2, PyTorch float32 on the CPU) and by every MLX build: SI-SDR against
59
+ the clean speech, DNSMOS P.835, and how close each output is to the original's. Pass: SI-SDR on every clip at most
60
+ 0.5 dB below the original's, mean DNSMOS at most 0.1 below.
61
+
62
+ | Precision | Size | SI-SDR (dB) | DNSMOS | Output vs the original's, mean / worst clip (dB) |
63
+ |---|---|---|---|---|
64
+ | noisy input | - | 4.98 | 2.04 | - |
65
+ | original (PyTorch fp32, CPU) | - | 10.30 | 3.19 | - |
66
+ | `fp32/` | 56 MB | 10.30 | 3.19 | 89.3 / 79.2 |
67
+ | `fp16/` (experimental) | 30 MB | 10.31 (1.2 lower on one clip) | 3.16 | 25.2 / 8.2 |
68
+ | `bf16/` (experimental) | 30 MB | 9.66 | 3.06 | 16.9 / 2.9 |
69
+ | `8bit/` (experimental) | 28 MB | 9.88 | 3.07 | 18.0 / 4.1 |
70
+ | `6bit/` (experimental) | 27 MB | 10.18 | 3.04 | 13.8 / 3.0 |
71
+ | `4bit/` (experimental) | 27 MB | 4.92 | 2.25 | 4.7 / -1.3 |
72
+
73
+ About 10x faster than real time on an Apple M5 (the original on the CPU: 0.6x).
74
+
75
+ ## How these were made
76
+
77
+ From [FRCRN](https://huggingface.co/alibabasglab/FRCRN_SE_16K) at revision `3766e6a64b`: the original weights loaded in MLX and stored as bf16 or fp16, or
78
+ quantized with MLX's affine quantization (`8bit`, `6bit`, `4bit`, group size 64; layers MLX can't quantize, such as
79
+ convolutions and norms, stay bf16). Every build was checked against the original before upload; the `sha256` column
80
+ above is what the files must hash to.
81
+
82
+ ## License
83
+
84
+ Apache-2.0. These files are a derivative of the original model and carry its license: it is copied in this
85
+ repository, and it governs your use of these files. The original model card follows unchanged.
86
+
87
+ ---
88
+
89
+ # Original model card
90
+
91
+ The FRCRN_SE_16K model weights for 16 kHz speech enhancement in [ClearerVoice-Studio](https://github.com/modelscope/ClearerVoice-Studio/tree/main) repo.
92
+
93
+ This model is trained on large scale datasets inclduing open-sourced and private data.
94
+
95
+ It enhances speech audios by removing background noise.