| --- |
| library_name: coreml |
| base_model: |
| - ResembleAI/chatterbox-nano |
| license: mit |
| pipeline_tag: text-to-speech |
| tags: |
| - coreml |
| - text-to-speech |
| - apple-silicon |
| - ios |
| - macos |
| language: |
| - en |
| --- |
| |
| # Chatterbox Nano — CoreML |
|
|
| CoreML export of [ResembleAI/chatterbox-nano](https://huggingface.co/ResembleAI/chatterbox-nano) |
| (**110M** GPT2-small T3 + distilled 2-step **meanflow** S3Gen, English, |
| paralinguistic tags) for Apple platforms, converted by |
| [FluidInference](https://github.com/FluidInference). |
|
|
| Each model ships as both `.mlpackage` (source) and compiled `.mlmodelc`. |
|
|
| ## Models |
|
|
| | File | Size (fp16) | Role | Compute | |
| |---|---:|---|---| |
| | `T3Nano-Prefill-T512-M1536-fp16` | 173 MB | GPT2-small prefill over ≤512-token context (batch 1, no CFG), initializes 1536-slot KV cache | CPU+GPU | |
| | `T3Nano-Decode-M1536-fp16` | 184 MB | Single-step AR decode, KV cache via I/O tensors (9 ms/step) | CPU+GPU | |
| | `T3Nano-Decode-M1536-fp16-stateful` | 184 MB | Single-step AR decode, KV cache in `MLState` (**3.3 ms/step**; macOS 15+/iOS 18+) | CPU+GPU | |
| | `FlowMean-N500-fp16` | 228 MB | S3Gen meanflow: 500-token bucket → 1000 mel frames, 2 plain Euler steps in-graph (0.38 s/call) | CPU+GPU | |
| | `HiFT-T1000-fp16` | 40 MB | HiFTNet vocoder: mel → 24 kHz waveform (0.09 s/call) | CPU+GPU | |
| | `FlowMean-N1000-fp16` | 242 MB | S3Gen meanflow, 1000-token bucket → 2000 mel frames (extended ~30 s output; FluidAudio #924) | CPU+GPU | |
| | `HiFT-T2000-fp16` | 40 MB | HiFTNet vocoder for the 2000-frame extended bucket | CPU+GPU | |
| | `tables/tables.safetensors` | 87 MB | text/speech embedding tables (host applies; GPT2 `wpe` is in-graph) | |
| | `tables/voice-default.safetensors` | 0.7 MB | precomputed built-in voice conditioning (T3 cond embeds + S3Gen ref dict) | |
| | `tokenizer/` | 1.4 MB | GPT2 BPE tokenizer (50276 ids incl. 20 paralinguistic tags: `[laugh]`, `[chuckle]`, `[sigh]`, …) | |
|
|
| ### Output budget / bucket choice |
|
|
| The flow bucket holds `voice prompt tokens + generated speech tokens + 3 |
| silence tokens`. The built-in voice uses 250 prompt tokens, so `N500` |
| yields **≤247 generated tokens ≈ 9.9 s of audio per call** and `N1000` |
| **≤747 ≈ 29.9 s**. Likewise the T3 prefill window holds |
| `voice conditioning (376) + text BPE + 1 BOS`, i.e. **≤135 text tokens**. |
| Pick ONE flow/vocoder pair at load time; the buckets are static shapes, so |
| the extended pair roughly doubles flow+vocoder latency even for short |
| outputs. |
|
|
| The AR stage runs at ~300 tok/s against the 25 Hz speech-token rate |
| (**~12× real-time**) with the stateful decode; the full compute path is |
| ~3.4× real-time with I/O-KV decode and ~6× with MLState, measured on an |
| M-series Mac with `.cpuAndGPU`. |
|
|
| ⚠️ The sibling Multilingual T3 packages hard-crash under `.cpuOnly`; the |
| Nano packages have not been matrix-tested there — prefer `.cpuAndGPU` or |
| `.all`. |
|
|
| ## Samples |
|
|
| [`samples/`](./tree/main/samples) has CoreML end-to-end renders |
| (`e2e_nano_*.wav`) next to stock PyTorch renders (`baseline_*.wav`), all |
| using the built-in voice. `*_tags.wav` exercises the `[chuckle]` |
| paralinguistic tag. Parakeet-v3 transcribes both e2e renders verbatim. |
|
|
| To synthesize locally without the upstream checkpoint (Apple silicon): |
|
|
| ```bash |
| git clone -b feat/chatterbox-nano-coreml https://github.com/FluidInference/mobius |
| cd mobius/models/tts/chatterbox/coreml |
| uv sync |
| uv run python verify/e2e_nano_coreml.py # models auto-download from this repo |
| ``` |
|
|
| ## Runtime boundary |
|
|
| The graphs cover T3 prefill/decode, the meanflow flow, and the HiFT |
| vocoder. The host runtime must provide: |
|
|
| - text normalization (`punc_norm`) + BPE tokenization (`tokenizer/`; no |
| BOT/EOT wrapping — the prefill context is cond ++ text ++ **one** BOS |
| speech embed) |
| - embedding prep from `tables.safetensors` (no positional add — GPT2 `wpe` |
| is applied in-graph) |
| - turbo sampling: temperature 0.8 → top-k 1000 → top-p 0.95 → repetition |
| penalty 1.2, EOS = 6562 (no CFG, no alignment analyzer) |
| - SineGen randomness (`phase_vec`, `noise` inputs to HiFT) and flow noise `z` |
| - flow bucket padding/cropping; MLState seeding from prefill KV for the |
| stateful decode |
|
|
| Voice cloning from a reference wav additionally needs the VoiceEncoder / |
| S3TokenizerV2 / CAMPPlus encoders, which are not converted here; voices can |
| be prepared offline in Python (`export-tables-nano.py --ref-wav`) and |
| shipped as `voice-*.safetensors`. |
|
|
| ## Parity (vs upstream PyTorch) |
|
|
| | Check | Result | |
| |---|---| |
| | T3 wrappers vs stock (fp32) | logits 1.4e-05 | |
| | T3 CoreML fp16 (I/O-KV / stateful) | logits 2.2e-02 / 1.9e-02 | |
| | Flow CoreML fp16 | mel max 5.9e-02, mean 1.8e-03 | |
| | HiFT CoreML fp16 | wav max 3.3e-03, mean 1.2e-04 | |
| | e2e ASR round-trip | verbatim transcripts; `[chuckle]` audibly realized | |
|
|
| ## License |
|
|
| MIT, following upstream |
| [ResembleAI/chatterbox-nano](https://huggingface.co/ResembleAI/chatterbox-nano). |
| Upstream embeds Resemble's Perth watermarker in its Python pipeline; this |
| CoreML export does not include a watermarking stage. |
|
|
