File size: 2,691 Bytes
108f5bd | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 | ---
language:
- zh
- en
license: apache-2.0
tags:
- mlx
- apple-silicon
- audio
- speech-recognition
- text-to-speech
- voice-cloning
- speech-editing
- voice-design
- multimodal
pipeline_tag: text-to-speech
---
# FireRedAudio-MLX-8bit
Apple Silicon MLX 8-bit quantized checkpoint for **FireRedAudio** (9B multimodal audio language model).
- 🐙 **GitHub Repository & WebUI Studio**: [vanch007/mlx-FireRedAudio](https://github.com/vanch007/mlx-FireRedAudio)
- ⚡ **Hardware Acceleration**: Metal GPU native unified memory, fused SDPA attention, and Gated Delta Metal kernel.
- 🚀 **Real-Time Factor (RTF)**:
- **ASR (Speech Recognition)**: **RTF 0.1396** (~7.2x real-time speedup)
- **TTS (Voice Cloning)**: **RTF 0.7529** (Real-time speech generation on Apple M3 Max)
- 💾 **Memory Footprint**: Only **13.54 GB** active Metal memory (runs smoothly on 16GB / 24GB+ Mac devices).
---
## Model Architecture & Quantization Details
- **Backbone LLM**: 9B Qwen3.5 language model with selective 8-bit affine quantization (group_size=64).
- **Continuous Audio Autoencoder / Vocoder**: RedAE VAE and Native ISTFT vocoder preserved in full FP32/BF16 precision for lossless audio phase fidelity.
- **Audio Understanding Encoder**: Whisper / Conv1D acoustic encoder preserved in high precision.
- **Audio Generation Head**: 11-layer Flow Matching DiT.
---
## Quick Start
### 1. Installation
```bash
git clone https://github.com/vanch007/mlx-FireRedAudio.git
cd mlx-FireRedAudio
uv venv .venv
source .venv/bin/activate
uv pip install -e .
```
### 2. Download Model
```bash
hf download vanch007/FireRedAudio-MLX-8bit --local-dir models/FireRedAudio-8bit
```
### 3. Run Inference (CLI & Python)
```python
from fireredaudio_mlx import FireRedAudioInference
engine = FireRedAudioInference(model_path="models/FireRedAudio-8bit")
# ASR Speech Transcription (RTF ~ 0.14)
res = engine.understand("assets/examples/asr_zh_fleurs.wav", task="asr")
print("Transcript:", res.answer)
# Zero-shot TTS Voice Cloning (RTF ~ 0.75)
audio_res = engine.tts(
prompt_text="收到你的来信，我很高兴。",
prompt_audio="assets/examples/tts_zh_prompt.wav",
target_text="你好，欢迎体验 FireRedAudio 8-bit MLX 实时语音生成！",
)
```
### 4. Launch WebUI Studio
```bash
python run_webui.py --model models/FireRedAudio-8bit
```
Open your browser at **http://127.0.0.1:7860**.
---
## Citation & References
- FireRedAudio Official: [FireRedTeam/FireRedAudio](https://github.com/FireRedTeam/FireRedAudio)
- Apple MLX: [MLX Framework](https://github.com/ml-explore/mlx)
- MLX Port & WebUI Studio: [vanch007/mlx-FireRedAudio](https://github.com/vanch007/mlx-FireRedAudio)
|