| # Validation |
|
|
| The catalog includes an individual JSON report for each of its seven models, linked below. The reports |
| record input and output shapes, inference settings, output peaks, file-writing results, and |
| conversion comparisons where available. |
|
|
| ## Catalog checks |
|
|
| All seven models passed a fresh-process inference check on the same 14-second stereo music |
| excerpt at 44.1 kHz, with overlapping chunks. Each produced the expected number of finite, |
| full-length stereo stems and wrote its output audio successfully. Network access was disabled, |
| and PyTorch, ONNX, and ONNX Runtime imports were blocked during inference. |
|
|
| | Model | Stems checked | Additional verification | Report | |
| |---|---:|---|---| |
| | BS-RoFormer-SW | 6 | Bit-identical unquantized output versus the original checkpoint in MLX | [JSON](validation/bs-roformer-sw.json) | |
| | Mel-RoFormer InstV7n | 2 | Bit-identical unquantized output versus the original checkpoint in MLX | [JSON](validation/mel-roformer-instv7n.json) | |
| | UVR-MDX-NET Inst HQ3 | 2 | Bit-identical unquantized output versus the original checkpoint in MLX | [JSON](validation/uvr-mdx-net-inst-hq3.json) | |
| | HTDemucs | 4 | Weights and config match the pinned published files | [JSON](validation/htdemucs.json) | |
| | HTDemucs_FT | 4 | Weights and config match the pinned published files | [JSON](validation/htdemucs_ft.json) | |
| | HTDemucs_6s | 6 | Weights and config match the pinned published files | [JSON](validation/htdemucs_6s.json) | |
| | ZFTurbo Vocals v1 | 2 | Weights and config match the pinned published files | [JSON](validation/mel-roformer-zfturbo-vocals-v1.json) | |
|
|
| The three new conversions were compared before output quantization, across all ten emitted |
| stems. The four reused models were downloaded into an empty cache and verified by size and |
| SHA-256 before inference. Their weights retain the original precision: float32 for Demucs, |
| float16 for ZFTurbo. |
|
|
| RoFormer checks used two overlapping windows. Demucs checks used two shifts, a fixed seed of |
| zero, and 25% overlap. The fixed seed makes the validation runs repeatable; normal package use leaves shift selection random unless a seed is supplied. The source audio and separated |
| music are not distributed with the models. |
|
|
| ## Automatic downloads and offline loading |
|
|
| The [public API integration report](validation/automatic-downloads.json) covers all seven models |
| through normal `Separator.load_model()` and `Separator.separate()` calls. Each test: |
|
|
| 1. Starts with an empty model directory and downloads the exact repository files from a local HTTP mirror. |
| 2. Loads the model using its existing package name and separates the 14-second fixture. |
| 3. Blocks all network connections, creates a new separator, and loads the same cache using the short catalog ID. |
| 4. Verifies full-length, finite output and bit-identical unquantized stems between both runs. |
|
|
| PyTorch, ONNX, and ONNX Runtime imports are blocked throughout. The test checks actual audio writes |
| and confirms that each model file is requested exactly once. This exercises download and cache |
| behavior independently of Hugging Face availability. The report records the tested implementation's |
| source hashes and identifies the local mirror as the download source. |
|
|
| CPU tests additionally cover corrupt downloads and caches, interrupted transfers, concurrent |
| first use, offline cache misses, older cache reuse, and unchanged routing for other models. |
|
|
| ## Earlier reference comparisons |
|
|
| The following evidence comes from the separator projects' reference tests, independently of |
| the catalog packaging checks. PyTorch references use the original |
| [python-audio-separator](https://github.com/nomadkaraoke/python-audio-separator). |
|
|
| - **BS-RoFormer-SW:** the standard core passes real-checkpoint tests against native |
| python-audio-separator references and cross-repository consistency checks. Earlier full-track |
| PCM comparisons include near-silent relative-error exceptions bounded by one PCM16 step. |
| - **InstV7n and Inst HQ3:** a twelve-track comparison on two hosts passed relative L2 ≤ 0.05 |
| for every compared stem. The worst values were 0.0120522 and 0.0047207, respectively. |
| - **HTDemucs and HTDemucs_FT:** full-track controls with identical shift offsets, following the |
| normalization correction, had worst relative L2 errors of 0.000775014 and 0.00106586. |
| These are single-track controls. Other corpus comparisons include stochastic shift differences. |
| - **HTDemucs_6s:** the existing Demucs model publication reports source-checkpoint parity. |
| The catalog check adds offline six-stem inference and exact file verification; it does not |
| rerun the PyTorch reference comparison. |
| - **ZFTurbo Vocals v1:** earlier release tests cover both stems, overlap, input sample rates, |
| and mixture reconstruction. This catalog uses the same published float16 conversion. |
|
|
| Detailed methods and results are in the mlx-audio-separator |
| [performance report](https://github.com/ssmall256/mlx-audio-separator/blob/main/docs/performance-snapshot.md), |
| [release validation](https://github.com/ssmall256/mlx-audio-separator/blob/main/docs/release-validation.md), |
| and [RoFormer core documentation](https://github.com/ssmall256/mlx-audio-separator/blob/main/docs/roformer-core.md). |
|
|
| ## Interpreting the results |
|
|
| The catalog checks verify that the packaged files load and preserve the tested MLX behavior. |
| They do not provide a new ground-truth SDR score, guarantee separation quality on every song, |
| or establish a speedup from the recorded validation wall times. Bit-identical conversion |
| results describe original-MLX versus safetensors-MLX outputs, not MLX versus PyTorch. |
|
|
| The [manifest](catalog.json) records source identities, runtime versions, and conversion source |
| fingerprints. The model cards list file hashes, and `SHA256SUMS` covers the distributed files. |
|
|
