Audio-to-Audio
MLX
Safetensors
moonbox
mlx-audio
mossformer2
clearervoice
speech-enhancement
denoising
48khz
Instructions to use moonbox-dev/MossFormer2_SE_48K-MLX with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use moonbox-dev/MossFormer2_SE_48K-MLX with MLX:
# Download the model from the Hub pip install huggingface_hub[hf_xet] hf download moonbox-dev/MossFormer2_SE_48K-MLX --local-dir MossFormer2_SE_48K-MLX
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Atomic Chat
|
Download README.md from moonbox-dev/MossFormer2_SE_48K-MLX: direct link, hf CLI and curl.
- Browser
- Download file 9.91 kB
-
https://huggingface.co/moonbox-dev/MossFormer2_SE_48K-MLX/resolve/main/README.md
- Command line
-
hf download hf://moonbox-dev/MossFormer2_SE_48K-MLX/README.md
-
curl -L -o README.md https://huggingface.co/moonbox-dev/MossFormer2_SE_48K-MLX/resolve/main/README.md
9.91 kB
| base_model: alibabasglab/MossFormer2_SE_48K | |
| base_model_relation: quantized | |
| license: apache-2.0 | |
| pipeline_tag: audio-to-audio | |
| library_name: mlx | |
| tags: | |
| - mlx | |
| - moonbox | |
| - mlx-audio | |
| - mossformer2 | |
| - clearervoice | |
| - speech-enhancement | |
| - denoising | |
| - 48khz | |
| # MossFormer2 MLX | |
| MLX builds of [MossFormer2](https://huggingface.co/alibabasglab/MossFormer2_SE_48K), made by Moonbox for Apple Silicon Macs. Each precision is a folder; keep a folder's | |
| files together. Only the parts listed below are converted: the rest of the pipeline comes from the original | |
| repository unless the section below says otherwise. | |
| ## Files | |
| The last column is how far each build is from the original PyTorch model on the same inputs (see the section below | |
| for what was measured). | |
| | Precision | File | Size | sha256 | Checked against the original | | |
| |---|---|---|---|---| | |
| | bf16 | [`bf16/config.json`](https://huggingface.co/moonbox-dev/MossFormer2_SE_48K-MLX/blob/main/bf16/config.json) | 0.4 kB | `667830c83b2d` | SI-SDR 11.30 dB (original 11.34), DNSMOS 3.08 (3.09); output vs the original's 45.6 dB mean / 35.5 dB worst clip | | |
| | bf16 | [`bf16/mlx_model.py`](https://huggingface.co/moonbox-dev/MossFormer2_SE_48K-MLX/blob/main/bf16/mlx_model.py) | 6.9 kB | `65e4ee73d3bc` | SI-SDR 11.30 dB (original 11.34), DNSMOS 3.08 (3.09); output vs the original's 45.6 dB mean / 35.5 dB worst clip | | |
| | bf16 | [`bf16/model.safetensors`](https://huggingface.co/moonbox-dev/MossFormer2_SE_48K-MLX/blob/main/bf16/model.safetensors) | 110.7 MB | `24ec32b445b3` | SI-SDR 11.30 dB (original 11.34), DNSMOS 3.08 (3.09); output vs the original's 45.6 dB mean / 35.5 dB worst clip | | |
| | fp16 | [`fp16/config.json`](https://huggingface.co/moonbox-dev/MossFormer2_SE_48K-MLX/blob/main/fp16/config.json) | 0.4 kB | `667830c83b2d` | SI-SDR 11.34 dB, DNSMOS 3.09; output vs the original's 52.2 / 41.8 dB | | |
| | fp16 | [`fp16/mlx_model.py`](https://huggingface.co/moonbox-dev/MossFormer2_SE_48K-MLX/blob/main/fp16/mlx_model.py) | 6.9 kB | `65e4ee73d3bc` | SI-SDR 11.34 dB, DNSMOS 3.09; output vs the original's 52.2 / 41.8 dB | | |
| | fp16 | [`fp16/model.safetensors`](https://huggingface.co/moonbox-dev/MossFormer2_SE_48K-MLX/blob/main/fp16/model.safetensors) | 110.7 MB | `8b53c4ae0cba` | SI-SDR 11.34 dB, DNSMOS 3.09; output vs the original's 52.2 / 41.8 dB | | |
| | 8bit | [`8bit/config.json`](https://huggingface.co/moonbox-dev/MossFormer2_SE_48K-MLX/blob/main/8bit/config.json) | 0.5 kB | `760230e0c620` | SI-SDR 11.30 dB, DNSMOS 3.08; output vs the original's 43.5 / 33.3 dB | | |
| | 8bit | [`8bit/mlx_model.py`](https://huggingface.co/moonbox-dev/MossFormer2_SE_48K-MLX/blob/main/8bit/mlx_model.py) | 6.9 kB | `65e4ee73d3bc` | SI-SDR 11.30 dB, DNSMOS 3.08; output vs the original's 43.5 / 33.3 dB | | |
| | 8bit | [`8bit/model.safetensors`](https://huggingface.co/moonbox-dev/MossFormer2_SE_48K-MLX/blob/main/8bit/model.safetensors) | 67.9 MB | `70ddb076cc82` | SI-SDR 11.30 dB, DNSMOS 3.08; output vs the original's 43.5 / 33.3 dB | | |
| | 6bit | [`6bit/config.json`](https://huggingface.co/moonbox-dev/MossFormer2_SE_48K-MLX/blob/main/6bit/config.json) | 0.5 kB | `c8834a4b1bed` | SI-SDR 11.39 dB, DNSMOS 3.08; output vs the original's 39.9 / 26.2 dB | | |
| | 6bit | [`6bit/mlx_model.py`](https://huggingface.co/moonbox-dev/MossFormer2_SE_48K-MLX/blob/main/6bit/mlx_model.py) | 6.9 kB | `65e4ee73d3bc` | SI-SDR 11.39 dB, DNSMOS 3.08; output vs the original's 39.9 / 26.2 dB | | |
| | 6bit | [`6bit/model.safetensors`](https://huggingface.co/moonbox-dev/MossFormer2_SE_48K-MLX/blob/main/6bit/model.safetensors) | 56.5 MB | `6f3698c77ecb` | SI-SDR 11.39 dB, DNSMOS 3.08; output vs the original's 39.9 / 26.2 dB | | |
| | 4bit | [`4bit/config.json`](https://huggingface.co/moonbox-dev/MossFormer2_SE_48K-MLX/blob/main/4bit/config.json) | 0.5 kB | `4a2d02aa9cc4` | below threshold (experimental): SI-SDR 11.43 dB mean but 0.7-0.8 dB lower on two clips, DNSMOS 3.06; output vs the original's 27.6 / 13.6 dB | | |
| | 4bit | [`4bit/mlx_model.py`](https://huggingface.co/moonbox-dev/MossFormer2_SE_48K-MLX/blob/main/4bit/mlx_model.py) | 6.9 kB | `65e4ee73d3bc` | below threshold (experimental): SI-SDR 11.43 dB mean but 0.7-0.8 dB lower on two clips, DNSMOS 3.06; output vs the original's 27.6 / 13.6 dB | | |
| | 4bit | [`4bit/model.safetensors`](https://huggingface.co/moonbox-dev/MossFormer2_SE_48K-MLX/blob/main/4bit/model.safetensors) | 45.1 MB | `5589ed314927` | below threshold (experimental): SI-SDR 11.43 dB mean but 0.7-0.8 dB lower on two clips, DNSMOS 3.06; output vs the original's 27.6 / 13.6 dB | | |
| ## How to run | |
| MossFormer2 SE 48K (ClearerVoice-Studio's speech enhancement: removes noise from speech at 48 kHz) in MLX. Each | |
| folder holds `model.safetensors`, `config.json` and `mlx_model.py`, the runtime: mlx-audio's MossFormer2 network | |
| (`mlx-audio==0.6.5`) with two corrections that make it match the original PyTorch model (below). Needs `mlx`, | |
| `mlx-audio`, `numpy`, `soundfile` (and `librosa` for input that isn't 48 kHz): | |
| ```python | |
| import sys; sys.path.insert(0, "8bit") # a downloaded folder | |
| import mlx_model, soundfile as sf | |
| model = mlx_model.load("8bit") | |
| clean = mlx_model.enhance(model, "noisy.wav") # float32, 48 kHz; dither=0.0 makes it deterministic | |
| sf.write("clean.wav", clean, 48000) | |
| ``` | |
| **Why not mlx-audio's own loader.** mlx-audio 0.6.5's `MossFormer2SEModel` loads these folders too, but its output | |
| is 18 dB (worst clip 6 dB) away from the original's and loses up to 3 dB SI-SDR: it applies rotary embeddings to | |
| the two halves of each vector (`nn.RoPE(traditional=False)`), where the original (`rotary_embedding_torch`) rotates | |
| interleaved pairs, and its Kaldi filterbank leaves the first sample of each frame un-pre-emphasized. `mlx_model.py` | |
| fixes both and computes the features exactly as `torchaudio.compliance.kaldi.fbank` (with the original's dither of | |
| 1.0 by default). | |
| `8bit/`, `6bit/` and `4bit/` quantize every linear layer (MLX affine, group size 64); convolutions and norms stay | |
| bf16. | |
| **How they were checked.** 8 noisy 48 kHz clips (LibriSpeech speech with babble or pink noise, 0-10 dB SNR), | |
| enhanced by the original PyTorch model (ClearerVoice's `clearvoice` 0.1.2, float32) and every MLX build: SI-SDR | |
| against the clean speech, DNSMOS P.835, and how close each output is to the original's (SNR in dB). Pass: SI-SDR on | |
| every clip at most 0.5 dB below the original's, mean DNSMOS at most 0.1 below. | |
| | Precision | Size | SI-SDR (dB) | DNSMOS | Output vs the original's, mean / worst clip (dB) | Real-time factor, M5 24 GB | | |
| |---|---|---|---|---|---| | |
| | noisy input | - | 4.99 | 2.10 | - | - | | |
| | original (PyTorch fp32, MPS) | - | 11.34 | 3.09 | - | 0.032 | | |
| | `bf16/` | 111 MB | 11.30 | 3.08 | 45.6 / 35.5 | 0.015 | | |
| | `fp16/` | 111 MB | 11.34 | 3.09 | 52.2 / 41.8 | 0.015 | | |
| | `8bit/` | 68 MB | 11.30 | 3.08 | 43.5 / 33.3 | 0.015 | | |
| | `6bit/` | 57 MB | 11.39 | 3.08 | 39.9 / 26.2 | 0.015 | | |
| | `4bit/` (experimental) | 45 MB | 11.43 (0.7-0.8 lower on two clips) | 3.06 | 27.6 / 13.6 | 0.015 | | |
| With the dither off on both sides, the MLX float32 network matches the original to 57 dB (48 dB worst clip). | |
| ## How these were made | |
| From [MossFormer2](https://huggingface.co/alibabasglab/MossFormer2_SE_48K) at revision `eff8c97925`: the original weights loaded in MLX and stored as bf16 or fp16, or | |
| quantized with MLX's affine quantization (`8bit`, `6bit`, `4bit`, group size 64; layers MLX can't quantize, such as | |
| convolutions and norms, stay bf16). Every build was checked against the original before upload; the `sha256` column | |
| above is what the files must hash to. | |
| ## License | |
| Apache-2.0. These files are a derivative of the original model and carry its license: it is copied in this | |
| repository, and it governs your use of these files. The original model card follows unchanged. | |
| --- | |
| # Original model card | |
| # Introduction | |
| The MossFormer2_SE_48K model weights for 48 kHz speech enhancement in [ClearerVoice-Studio](https://github.com/modelscope/ClearerVoice-Studio/tree/main) repo. | |
| This model is trained on large scale datasets inclduing open-sourced and private data. | |
| It enhances speech audios by removing background noise. | |
| # Install | |
| **Clone the Repository** | |
| ``` sh | |
| git clone https://github.com/modelscope/ClearerVoice-Studio.git | |
| ``` | |
| **Create Conda Environment** | |
| ``` sh | |
| cd ClearerVoice-Studio | |
| conda create -n clearvoice python=3.8 | |
| conda activate clearvoice | |
| pip install -r requirements.txt | |
| ``` | |
| **Run Script** | |
| Go to `clearvoice/` and use the following examples. The MossFormer2_SE_48K model will be downloaded from huggingface automatically. | |
| Sample example 1: use speech enhancement model `MossFormer2_SE_48K` to process one wave file of `samples/input.wav` and save the output wave file to `samples/output_MossFormer2_SE_48K.wav` | |
| ```python | |
| from clearvoice import ClearVoice | |
| myClearVoice = ClearVoice(task='speech_enhancement', model_names=['MossFormer2_SE_48K']) | |
| output_wav = myClearVoice(input_path='samples/input.wav', online_write=False) | |
| myClearVoice.write(output_wav, output_path='samples/output_MossFormer2_SE_48K.wav') | |
| ``` | |
| Sample example 2: use speech enhancement model `MossFormer2_SE_48K` to process all input wave files in `samples/path_to_input_wavs/` and save all output files to `samples/path_to_output_wavs` | |
| ```python | |
| from clearvoice import ClearVoice | |
| myClearVoice = ClearVoice(task='speech_enhancement', model_names=['MossFormer2_SE_48K']) | |
| myClearVoice(input_path='samples/path_to_input_wavs', online_write=True, output_path='samples/path_to_output_wavs') | |
| ``` | |
| Sample example 3: use speech enhancement model `MossFormer2_SE_48K` to process wave files listed in `samples/audio_samples.scp' file, and save all output files to 'samples/path_to_output_wavs_scp/' | |
| ```python | |
| from clearvoice import ClearVoice | |
| myClearVoice = ClearVoice(task='speech_enhancement', model_names=['MossFormer2_SE_48K']) | |
| myClearVoice(input_path='samples/scp/audio_samples.scp', online_write=True, output_path='samples/path_to_output_wavs_scp') | |
| ``` | |