SCNet base, Core ML fp16 and fp32

018240c verified about 15 hours ago

2.77 kB

license: mit
library_name: coreml
tags:
  - audio
  - music
  - source-separation
  - coreml
  - scnet

SCNet (base) – Core ML

Core ML conversion of SCNet, a music source separation model that splits a stereo mix into four stems: drums, bass, other, vocals. These files are downloaded on first use by the iOS app Preso!, which runs the separation entirely on device.

Attribution

The model and its pretrained weights are the work of the SCNet authors:

Paper: Weinan Tong et al., SCNet: Sparse Compression Network for Music Source Separation (2024)
Original code and weights: https://github.com/starrytong/SCNet
License: MIT. The author confirmed that the released pretrained weights are MIT licensed and may be redistributed, including format-converted versions, with attribution: https://github.com/starrytong/SCNet/issues/35

This repository only contains a format conversion of the original checkpoint. No retraining or fine-tuning was done. The original license text is in LICENSE-SCNet.

Files

Folder	Precision	Size
`SCNet_fp16.mlpackage`	float16	about 23 MB
`SCNet_fp32.mlpackage`	float32	about 45 MB

Each .mlpackage is a folder with three files: Manifest.json, Data/com.apple.CoreML/model.mlmodel and Data/com.apple.CoreML/weights/weight.bin.

What is inside the model

Only the network. The STFT and inverse STFT, input normalisation and chunking are done outside, by the caller:

Sample rate 44.1 kHz, stereo.
STFT: 4096 points, hop 1024, rectangular window, centred with reflect padding, normalised.
Fixed segments of 11 s (476 frames), 25% overlap, triangular cross-fade.
Input spec: shape (1, 4, 2049, 476), laid out as [channel][real/imag].
Output stems: shape (1, 16, 2049, 476), laid out as [source][channel][real/imag], sources in the order drums, bass, other, vocals.
The FFTs along the time axis inside the original network are replaced by constant DFT matrices, which is why the segment length is fixed.

Measured

Conversion fidelity against the PyTorch original: 109 dB (fp32), 59 dB (fp16).
Quality on 12 seven-second MUSDB18 test excerpts (global SDR, not comparable one to one with the BSSEval figures in the paper): drums 9.2, bass 7.6, other 5.3, vocals 10.0 dB.
Speed on a Mac with Apple M1, GPU, fp16: about 2 s for 30 s of audio.

Not measured: speed and memory on an iPhone, and the Neural Engine. In the iOS Simulator the fp16 variant returned all zeros while fp32 worked; on macOS both work.

Training data

SCNet was trained by its authors on MUSDB18-HQ, a dataset released for research and educational use. Check whether that matters for your use before relying on these weights commercially.