| --- |
| license: apache-2.0 |
| library_name: mlx |
| base_model: HeartMuLa/HeartCodec-oss-20260123 |
| pipeline_tag: text-to-audio |
| language: |
| - en |
| - zh |
| tags: |
| - mlx |
| - swift |
| - music-generation |
| - heartmula |
| - audio-codec |
| --- |
| |
| # HeartCodec-oss-fp32 (MLX) |
|
|
| The decoder half of [HeartCodec-oss](https://huggingface.co/HeartMuLa/HeartCodec-oss-20260123) at float32, for |
| [mlx-heartmula-swift](https://github.com/xocialize/mlx-heartmula-swift). It turns HeartMuLa's 8 × 8192 RVQ codes (12.5 Hz) into 48 kHz stereo: a flow-matching LLaMA-DiT |
| (24 × 1536 + 6 × 3072, 10 Euler steps, CFG 1.25) to 25 Hz latents in 29.76 s windows, then the SQ-Codec decoder (1920× |
| upsampling). This is the exact codec, matching upstream heartlib at 115–118 dB end to end. It costs twice the memory and 3.5× |
| the time per window of the default [fp16 codec](https://huggingface.co/mlx-community/HeartCodec-oss-fp16). |
|
|
| | file | size | |
| |---|---| |
| | `model-0000{1,2}-of-00002.safetensors` + index (506 tensors, float32; decode path only, weight_norm folded, MLX conv layouts) | 6.29 GB | |
| | `config.json` (with an `mlx_heartmula` section) | 1.8 KB | |
| | `LICENSE` (Apache-2.0), `NOTICE` | — | |
|
|
| ## Use |
|
|
| **This codec needs TF32 off for the whole process.** On Apple's neural accelerators, MLX's default TF32 matmuls drift about |
| 1e-3. The SQ-Codec's `round(9x)/9` step turns that into 25–30 dB end to end. MLX reads the setting once, on the process's |
| first matmul, so call `configureProcess()` before any model runs. If TF32 is already latched on, the package refuses to load |
| this codec and names the fix. |
|
|
| ```swift |
| // Package.swift: .package(url: "https://github.com/xocialize/mlx-heartmula-swift", from: "0.1.1") → product "MLXHeartMuLa" |
| import MLXHeartMuLa |
| import MLXServeCore |
| import MLXToolKit |
| |
| HeartMuLaPackage.configureProcess() // MLX_ENABLE_TF32=0 — must precede the first matmul in the process |
| let engine = MLXServeEngine() |
| await engine.useModelStore(ModelStore(root: modelsRoot)) |
| let id = try await engine.register(HeartMuLaPackage.registration, |
| configuration: HeartMuLaConfiguration(quant: .bf16, codec: .fp32), // the exact pair |
| id: "heartmula-oss-3b") |
| try await engine.prepare(.music, package: id) // mlx-community/HeartMuLa-oss-3B-bf16 + this repo; 48 GB Mac |
| ``` |
|
|
| Running the bare Core instead of the engine? Set `MLX_ENABLE_TF32=0` in the environment before launch. |
|
|
| | LM + codec | declared memory (resident + peak working set) | fits | render speed (M5 Max) | |
| |---|---|---|---| |
| | **8-bit + fp16 (the default)** | 8.5 + 5.5 GB | 24 GB Mac | 3.0× real time | |
| | bf16 + fp16 | 11.5 + 6.0 GB | 32 GB Mac | 2.5× | |
| | 8-bit + fp32 | 11.5 + 10.0 GB | 32 GB Mac | 2.3× | |
| | bf16 + fp32 | 14.5 + 10.0 GB | 48 GB Mac | 1.85× | |
|
|
| Memory is phys_footprint through MLXEngine at a 120 s song and a 240 s worst case; "fits" is the engine's 0.7 × RAM |
| admission rule. A 4-bit LM failed its lyric gate (PER 0.220) and is not published. |
| |
| ## Measured character |
| |
| Read on the fleet's 12-song set (10 English, 2 Mandarin; two seed sets per tier) and in the operator's listening passes: |
| |
| - **Lyrics are followed closely in English.** The lyric error rate (Whisper PER) is 0.125 with the bf16 LM and 0.132 with the |
| 8-bit LM (the PyTorch reference: 0.165). |
| - **Style tags steer it weakly.** CLAP style retrieval ranks its songs about 4.5 of 12 for their own tags, against about 2 for |
| the other open music models measured. A higher CFG (3.0) does not help. |
| - **Voice gender drifts.** Most voices come out female whatever the tags ask: 1 of 6 male-voice prompts got a male voice. |
| - **Mandarin is weak and uneven.** The PyTorch reference read PER 0.457 in Mandarin (one verse lost) against 0.107 in |
| English. The Swift bf16 takes did better on the two Mandarin songs, which is too few to call it fixed. |
| - **The 8-bit LM runs to the length cap twice as often as bf16** (10 of 24 songs against 5 of 24 at a 120 s cap). The lyrics are |
| sung, then choruses repeat, and one song in 24 looped a single line until the cap. |
| - **Raw codec output can exceed full scale** (peaks to 1.37), so the Swift pipeline applies a single −1 dBFS limiter gain. |
| |
| ## Parity |
| |
| Every tensor is bit-equal to the release's own float32 value, with weight_norm folded the way PyTorch computes it. MLX conv |
| layouts match PyTorch within 3e-6. The Swift decode matches upstream heartlib on PyTorch-MPS at 117.8 / 115.0 dB end to end |
| on fixed codes and noise, with 0 rounding-grid flips (TF32 off). Details: [MEASUREMENTS.md](https://github.com/xocialize/mlx-heartmula-swift/blob/main/MEASUREMENTS.md). |
|
|
| ## Licence and attribution |
|
|
| Apache-2.0. HeartMuLa's generator repositories carry `license: apache-2.0` on their model cards but no LICENSE file. The grant |
| is the card statement plus heartlib's README of 2026-01-20, which moves "all related model weights" to Apache-2.0. This repo |
| ships heartlib's Apache-2.0 text as `LICENSE`, and a `NOTICE` with that grant chain and exactly what was changed. Not endorsed |
| by the HeartMuLa team or MuLa Labs. Paper: arXiv 2601.10547. |
|
|
