README.md

---
license: other
license_name: stability-ai-community
license_link: LICENSE.md
tags: [audio, text-to-audio, core-ai, aimodel, apple, on-device, stable-audio]
---

# Stable Audio 3 Medium — Apple Core AI (`.aimodel`)

**Powered by Stability AI**

A community conversion of [`stabilityai/stable-audio-3-medium`](https://huggingface.co/stabilityai/stable-audio-3-medium) to Apple's
**Core AI** format, for on-device generation on iOS 27 / macOS 27. The high-quality tier: music and sound, Mac-class.

Not an official Stability AI release. The weights are unchanged — this is a format
conversion, not a retrain or a fine-tune.

## What's here

A **single `.aimodel`** holding four named inference functions, plus the tokenizer:

| Function | Signature |
|---|---|
| `conditioner` | `(input_ids, attention_mask, seconds)` → `cross_attn_cond`, `cross_attn_mask`, `global_cond` |
| `dit` | `(x, t, cross_attn_cond, cross_attn_mask, global_cond, local_add_cond)` → `v` — **any length** |
| `decoder_N` | `(latent)` → `audio` — several fixed lengths, see below |
| `encoder_N` | `(audio)` → `latent` — for audio-in, continuation and inpainting |

```swift
let model = try await AIModel.load(url, options)
let dit = model.loadFunction("dit")
```

`medium.aimodel` is 9.8 GB and the tokenizer folder is 34 MB. All weights are
**float32** — precision was chosen by measurement, not preference; see *Fidelity*.

The conditioner's output matches the original library's conditioning assembly exactly
(0.0 difference), so nothing about the prompt or duration handling is approximated.

**Why several decoder lengths.** The DiT takes any length, but the encoder and decoder are
compiled at fixed sizes — and a fixed-length decoder makes you pay its maximum on every
call, since short latents have to be padded up to it. Measured on the 0.6 B model,
decoding the same 24 s of content costs 0.30 s / 0.5 GB through a 47.6 s graph but
2.38 s / 2.1 GB through a 380.4 s one. So the asset ships a ladder — pick the smallest
`decoder_N` that fits your clip. Weights are shared between the functions, so the extra
rungs cost about 0.4 MB each rather than a full copy.

## Capabilities

Feature parity with the original, with one structural difference:

| | Original | This conversion |
|---|---|---|
| Text → audio | ✅ | ✅ |
| Audio length | up to 190.2 s | up to 190.2 s |
| Audio in / continuation / inpainting | ✅ | ✅ (via the `encode_N` functions) |
| Negative prompts / CFG | ✅ | ✅ vanilla CFG, verified (see note) |
| Prompt length | 256 tokens | 256 tokens |
| Steps, seeds, determinism | ✅ | ✅ |
| 44.1 kHz stereo | ✅ | ✅ |

**The structural difference:** the DiT takes any length (its shape is `[1, 256, -1]`),
but the encoder and decoder are compiled at a fixed maximum of **190.2 s**. For shorter
audio, zero-pad the latent to that length, decode, and trim. The padding influences only
the last 7–26 ms, which falls inside the margin you should be trimming anyway.

### Classifier-free guidance

Run the DiT twice per step and blend on the host:

```python
v = v_cond + (cfg_scale - 1) * (v_cond - v_uncond)
```

This reproduces the reference implementation's **vanilla** CFG exactly (cosine 1.0000000,
SNR 81.8 dB, measured). Note the reference defaults to *adaptive projected* guidance
(`apg_scale=1.0`), which projects out the parallel component; that variant is **not**
implemented here. `example.py --negative "..." --cfg-scale 3` uses the vanilla form.

## Getting good output — read this

Three settings are not optional. Getting them wrong produces audio that is clearly
broken, not subtly worse:

1. **Sampler must be `pingpong`.** This is an `rf_denoiser` model. Euler drives the
   output past full scale (+7.8 dBFS measured) and crushes dynamic range by ~10 dB.
2. **Generate at 256 latent frames (23.8 s) or longer.** The model's
   `distribution_shift_options.min_length` is 256. Below that, output picks up gross
   high-frequency content — we measured 16–27 % of energy above 10 kHz against ~1 %
   for correct output.
3. **8 steps, CFG scale 1.0.** These are the model's own defaults for this family.

Latent frames ↔ seconds: `frames = seconds × 44100 / 4096`.

## Usage

`example.py` in this repo is self-contained — four graphs, the tokenizer, numpy and
`coreai.runtime`. No `stable-audio-tools`, no PyTorch, no virtualenv beyond the Core AI
runtime itself.

```bash
pip install coreai-torch tokenizers numpy      # brings the Core AI runtime

# text to audio
python3 example.py "loud crackling campfire with crickets" out.wav --seconds 24

# continue or inpaint from existing audio (44.1 kHz WAV)
python3 example.py "a dog barking" out.wav --seconds 24 --init-audio in.wav --keep 12
```

`--keep` sets how many seconds of `--init-audio` are held as context; everything after it
is regenerated from the prompt. Omit `--init-audio` for plain text-to-audio.

The sampler, schedule and conditioning assembly are all in that one file, ~120 lines, if
you are porting to Swift. The ping-pong loop is four lines:

```python
denoised = x - t[i] * dit(x, t[i], **cond)
x = (1 - t[i+1]) * denoised + t[i+1] * randn_like(x)
```

## Fidelity

Measured against the original PyTorch model on identical inputs:

| Graph | Cosine | SNR |
|---|---|---|
| DiT | 1.000000000 | 105.5 dB |
| Decoder | 1.000000000 | 101.4 dB |
| Encoder | 1.000000000 | ~100 dB |
| **End-to-end audio** | **0.999999981** | **74.1 dB** |

## Performance

On an M-series Mac (36 GB), 23.8 s of 44.1 kHz stereo:

- **2.21 s** — about 10.8× realtime
- Roughly **2× faster** than the same model running under MLX with 8-bit weights

Memory scales with duration and it is the real limit, not disk. A rough guide from the
same hardware: ~7.7 GB fixed plus ~0.32 GB per second of audio for the 2 B model. Check
before you ask for long clips.

## First load compiles the model

Core AI specialises an `.aimodel` for your device the first time you load it, then caches
the result. That first load is slow and the cost grows sharply with decoder length —
measured on the 2 B decoder, per rung: 23.8 s of audio compiles in 12 s, 47.6 s in 44 s,
95.1 s in 8.5 min, and 190.2 s in over an hour. Subsequent loads are effectively instant.

To skip it, precompile ahead of time:

```bash
xcrun coreai-build compile medium.aimodel --platform macOS --preferred-compute gpu
```

That writes one `.aimodelc` per architecture, which loads without compiling (2.2 s versus
12 s on an M3 Max). `.aimodelc` runs only on the architecture it was built for, so the
portable `.aimodel` is what ships here. This is the workflow Apple recommends for apps
(WWDC26, "Integrate on-device AI models into your app using Core AI").

## Known issue

**fp16 crashes the Neural Engine**, whatever the shape (`ANE inference operation failed`,
Code=-19) — retested on macOS 27.0 build 26A5416b. Fixing the length at export does *not*
avoid it, and nor does asking for the GPU: `allowed_compute_unit_kinds` always includes
the Neural Engine and has no setter. These graphs therefore ship at float32, which sidesteps
the bug entirely since the Neural Engine cannot execute float32 at all. Reproduced on both
model sizes.

## Licence

Stability AI Community License — see `LICENSE.md`. Free for research, non-commercial and
limited commercial use; register with Stability AI for commercial use, and the licence
terminates above USD $1M annual revenue. The T5Gemma text encoder is additionally subject
to the Gemma Terms of Use — see `LICENSE_GEMMA.md`. See `NOTICE` for attribution.