| --- |
| license: apache-2.0 |
| library_name: mlx |
| base_model: HeartMuLa/HeartCodec-oss-20260123 |
| pipeline_tag: text-to-audio |
| language: |
| - en |
| - zh |
| tags: |
| - mlx |
| - swift |
| - music-generation |
| - heartmula |
| - audio-codec |
| --- |
| |
| # HeartCodec-oss-fp16 (MLX) |
|
|
| The decoder half of [HeartCodec-oss](https://huggingface.co/HeartMuLa/HeartCodec-oss-20260123) at float16, for |
| [mlx-heartmula-swift](https://github.com/xocialize/mlx-heartmula-swift). It turns HeartMuLa's 8 × 8192 RVQ codes (12.5 Hz) into 48 kHz stereo: a flow-matching LLaMA-DiT |
| (24 × 1536 + 6 × 3072, 10 Euler steps, CFG 1.25) to 25 Hz latents in 29.76 s windows, then the SQ-Codec decoder (1920× |
| upsampling). **This is the default codec:** half the size of the fp32 codec and 3.5× faster per window. |
|
|
| | file | size | |
| |---|---| |
| | `model.safetensors` (506 tensors, float16; decode path only, weight_norm folded, MLX conv layouts) | 3.15 GB | |
| | `config.json` (with an `mlx_heartmula` section) | 1.9 KB | |
| | `LICENSE` (Apache-2.0), `NOTICE` | — | |
|
|
| Never bf16: upstream says bf16 degrades the codec's audio. |
|
|
| ## Use |
|
|
| This codec is what `HeartMuLaConfiguration` picks unless told otherwise. It works with either LM repo. |
|
|
| ```swift |
| // Package.swift: .package(url: "https://github.com/xocialize/mlx-heartmula-swift", from: "0.1.1") → product "MLXHeartMuLa" |
| import MLXHeartMuLa |
| import MLXServeCore |
| import MLXToolKit |
| |
| let engine = MLXServeEngine() |
| await engine.useModelStore(ModelStore(root: modelsRoot)) |
| let id = try await engine.register(HeartMuLaPackage.registration, |
| configuration: HeartMuLaConfiguration(), // .int8 LM + .fp16 codec |
| id: "heartmula-oss-3b") |
| try await engine.prepare(.music, package: id) // mlx-community/HeartMuLa-oss-3B-8bit + this repo |
| ``` |
|
|
| **TF32 doesn't matter here.** With the fp16 codec, a song is byte-identical whether MLX's TF32 matmuls are on or off, so it |
| loads in a process where another model already ran. On the same codes it sits 23.6 dB from the fp32 codec (the SQ-Codec's |
| rounding step amplifies fp16 rounding), with lyric and aesthetic scores unchanged. |
|
|
| | LM + codec | declared memory (resident + peak working set) | fits | render speed (M5 Max) | |
| |---|---|---|---| |
| | **8-bit + fp16 (the default)** | 8.5 + 5.5 GB | 24 GB Mac | 3.0× real time | |
| | bf16 + fp16 | 11.5 + 6.0 GB | 32 GB Mac | 2.5× | |
| | 8-bit + fp32 | 11.5 + 10.0 GB | 32 GB Mac | 2.3× | |
| | bf16 + fp32 | 14.5 + 10.0 GB | 48 GB Mac | 1.85× | |
|
|
| Memory is phys_footprint through MLXEngine at a 120 s song and a 240 s worst case; "fits" is the engine's 0.7 × RAM |
| admission rule. A 4-bit LM failed its lyric gate (PER 0.220) and is not published. |
| |
| ## Measured character |
| |
| Read on the fleet's 12-song set (10 English, 2 Mandarin; two seed sets per tier) and in the operator's listening passes: |
| |
| - **Lyrics are followed closely in English.** The lyric error rate (Whisper PER) is 0.125 with the bf16 LM and 0.132 with the |
| 8-bit LM (the PyTorch reference: 0.165). |
| - **Style tags steer it weakly.** CLAP style retrieval ranks its songs about 4.5 of 12 for their own tags, against about 2 for |
| the other open music models measured. A higher CFG (3.0) does not help. |
| - **Voice gender drifts.** Most voices come out female whatever the tags ask: 1 of 6 male-voice prompts got a male voice. |
| - **Mandarin is weak and uneven.** The PyTorch reference read PER 0.457 in Mandarin (one verse lost) against 0.107 in |
| English. The Swift bf16 takes did better on the two Mandarin songs, which is too few to call it fixed. |
| - **The 8-bit LM runs to the length cap twice as often as bf16** (10 of 24 songs against 5 of 24 at a 120 s cap). The lyrics are |
| sung, then choruses repeat, and one song in 24 looped a single line until the cap. |
| - **Raw codec output can exceed full scale** (peaks to 1.37), so the Swift pipeline applies a single −1 dBFS limiter gain. |
| |
| ## Parity |
| |
| Every tensor is the release's own float32 value (weight_norm folded in float32) cast to float16. The fp32 codec it derives from |
| matches upstream heartlib at 115–118 dB end to end. Details: [MEASUREMENTS.md](https://github.com/xocialize/mlx-heartmula-swift/blob/main/MEASUREMENTS.md). |
|
|
| ## Licence and attribution |
|
|
| Apache-2.0. HeartMuLa's generator repositories carry `license: apache-2.0` on their model cards but no LICENSE file. The grant |
| is the card statement plus heartlib's README of 2026-01-20, which moves "all related model weights" to Apache-2.0. This repo |
| ships heartlib's Apache-2.0 text as `LICENSE`, and a `NOTICE` with that grant chain and exactly what was changed. Not endorsed |
| by the HeartMuLa team or MuLa Labs. Paper: arXiv 2601.10547. |
|
|
