sam-audio-base-swift-fp16 (MLX)

Meta's SAM-Audio base converted for mlx-sam-audio-swift, a Swift-MLX port for Apple Silicon. Describe a sound in a few words ("speech", "dog barking", "drums") and the model separates it from a 48 kHz mixture, returning the described sound (target) and everything else (residual).

Why this repository exists next to mlx-community/sam-audio-base

mlx-community already hosts SAM-Audio conversions (sam-audio-base, sam-audio-base-fp16 and the small and large sizes). They were made for mlx-audio's Python port, and they are the right choice if that is what you run. This repository is a separate conversion, for three reasons:

1. A different runtime. It is laid out for the Swift package, not for mlx-audio: three weight files (model, text_encoder, codec) with the Swift modules' key names, the T5 encoder and tokenizer included rather than fetched from t5-base at run time, and weight norm already folded into the codec. Neither port can load the other's files, so the two cannot share a repository, and the plain name was taken.
2. Tensors the existing bundles do not contain. When there is no video prompt, upstream SAM-Audio still runs its align_masked_video module on an all-zero video feature, which adds a learned constant to every frame. The existing conversions strip that module, and mlx-audio's port skips the step. Against upstream PyTorch that costs about 20 dB of agreement per model step (18.6–24.7 dB as shipped, 110.7–112.0 dB with the term restored), and in our tests it is what made some separations come back with a silent target (8 of 160 draws, against 0 of 58 with the term). It is reported as mlx-audio#1176. This bundle keeps the term's tensors, and the Swift port applies it.
3. The watermark branch. Upstream's codec embeds an AudioSeal watermark in every output. This bundle carries that branch's weights and the Swift port runs it, so its output is marked as upstream's is; mlx-audio's decode path does not apply it.

What is the same: the source checkpoint (facebook/sam-audio-base), the licence, and the scope — text and time-span prompts only, with no visual prompting, re-ranking or span prediction in either conversion.

If mlx-audio adopts the fix, its bundles will need re-converting from the upstream checkpoint, and this repository will still be needed for the Swift layout.

Licence — read before use

The weights are under Meta's SAM License (LICENSE, last updated November 19, 2025), as is the code that runs them. This summary does not replace the text:

• Commercial use is permitted. Redistribution, including of converted or modified versions, is allowed only under the SAM License, with a copy of it.
• Prohibited uses: ITAR-controlled activities and end uses prohibited by trade controls, "including those related to military or warfare purposes, nuclear industries or applications, espionage, or the development or use of guns or illegal weapons"; you must comply with sanctions and applicable law.
• You may not "involve or encourage others to reverse engineer, decompile or discover the underlying components".
• You indemnify Meta against third-party claims arising from your use or distribution; there is no warranty; Meta may amend the licence.
• Acknowledge the SAM Materials in research publications.

Outputs are watermarked. As in upstream, the codec's decoder embeds an AudioSeal watermark in every waveform it produces. The port has no switch to turn it off.

Contents

file	what	size
model.safetensors	the flow-matching transformer (16 layers × 2048), input projections, span anchors, the align_masked_video terms; fp16, 318 tensors, 1,150.3 M parameters	2.30 GB
text_encoder.safetensors	the T5-base encoder (from t5-base, Apache-2.0), fp32 (T5 overflows in half precision), 99 tensors	0.44 GB
codec.safetensors	the DAC-VAE codec with its watermark branch, fp32, weight norm folded, 251 tensors	0.43 GB
tokenizer.json, tokenizer_config.json, special_tokens_map.json, spiece.model	the t5-base tokenizer	3.2 MB
config.json	the upstream config's text path and the conversion's provenance	1.7 KB
LICENSE	the SAM License	—

Converted from facebook/sam-audio-base @ 81f64008f9f9 with the package's Tools/convert; upstream code facebookresearch/sam-audio @ bb4c699, codec facebookresearch/dacvae @ 414c207. Not included: the vision encoder (visual prompts), the re-ranking models, the judge, and the span predictor.

Use

// Package.swift: .package(url: "https://github.com/xocialize/mlx-sam-audio-swift", from: "0.1.0") → product "SAMAudioCore"
import SAMAudioCore

let pipeline = try await SAMAudioPipeline(bundle: bundleURL)           // a local copy of this repository
let out = try pipeline.separate(wav: mixture, prompt: "speech", seed: 7)   // MLXArray (samples,), 48 kHz mono
// out.target, out.residual: 48 kHz mono, the mixture's length

// optional: time spans where the sound is present, for intermittent sounds
let knocks = try pipeline.separate(wav: mixture, prompt: "door knock",
                                   spans: [SeparationSpan(start: 0.5, end: 1.9), SeparationSpan(start: 3.0, end: 4.4)])

What to expect

Measured through the Swift pipeline on an M5 Max, on 58 separations of 10 s mixtures with known sources, one draw each (gain in mel-spectrogram SNR over the unprocessed mixture; "missed" = worse than the mixture):

	median gain	missed
a sound effect or speech from speech + effect (24)	12.5 dB	1
speech from speech + music (8)	13.9 dB	0
music from speech + music (8)	15.4 dB	0
drums / bass / guitar from a trio (6 each)	13.5 / 5.8 / 1.7 dB	0 / 0 / 1

• Write prompts as lowercase noun or verb phrases — "speech", "rain", "door knock". Sentences ("a person speaking") miss where the short form works; upstream's README says the same.
• Guitar-like stems do not separate in these tests.
• Target and residual are generated, not filtered: they are not sample-aligned with the mixture, so do not derive anything by subtraction.
• 3.8 s per 10 s clip; phys_footprint 3.3 GB after load, 7.5 GB peak (this bundle as stored, 58 clips). Output is mono; a stereo input is downmixed by the caller. One clip of about 10 s per call in this release.
• Parity against upstream PyTorch at fp32: 97–111 dB end to end, watermark included (the package's MEASUREMENTS.md).

Citation

@article{shi2025samaudio,
    title={SAM Audio: Segment Anything in Audio},
    author={Bowen Shi and Andros Tjandra and John Hoffman and Helin Wang and Yi-Chiao Wu and Luya Gao and Julius Richter and Matt Le and Apoorv Vyas and Sanyuan Chen and Christoph Feichtenhofer and Piotr Doll{\'a}r and Wei-Ning Hsu and Ann Lee},
    year={2025},
    url={https://arxiv.org/abs/2512.18099}
}