| --- |
| language: |
| - zh |
| - en |
| license: apache-2.0 |
| tags: |
| - mlx |
| - apple-silicon |
| - audio |
| - speech-recognition |
| - text-to-speech |
| - voice-cloning |
| - speech-editing |
| - voice-design |
| - multimodal |
| pipeline_tag: text-to-speech |
| --- |
| |
| # FireRedAudio-MLX-8bit |
|
|
| Apple Silicon MLX 8-bit quantized checkpoint for **FireRedAudio** (9B multimodal audio language model). |
|
|
| - 🐙 **GitHub Repository & WebUI Studio**: [vanch007/mlx-FireRedAudio](https://github.com/vanch007/mlx-FireRedAudio) |
| - ⚡ **Hardware Acceleration**: Metal GPU native unified memory, fused SDPA attention, and Gated Delta Metal kernel. |
| - 🚀 **Real-Time Factor (RTF)**: |
| - **ASR (Speech Recognition)**: **RTF 0.1396** (~7.2x real-time speedup) |
| - **TTS (Voice Cloning)**: **RTF 0.7529** (Real-time speech generation on Apple M3 Max) |
| - 💾 **Memory Footprint**: Only **13.54 GB** active Metal memory (runs smoothly on 16GB / 24GB+ Mac devices). |
|
|
| --- |
|
|
| ## Model Architecture & Quantization Details |
|
|
| - **Backbone LLM**: 9B Qwen3.5 language model with selective 8-bit affine quantization (group_size=64). |
| - **Continuous Audio Autoencoder / Vocoder**: RedAE VAE and Native ISTFT vocoder preserved in full FP32/BF16 precision for lossless audio phase fidelity. |
| - **Audio Understanding Encoder**: Whisper / Conv1D acoustic encoder preserved in high precision. |
| - **Audio Generation Head**: 11-layer Flow Matching DiT. |
| |
| --- |
| |
| ## Quick Start |
| |
| ### 1. Installation |
| |
| ```bash |
| git clone https://github.com/vanch007/mlx-FireRedAudio.git |
| cd mlx-FireRedAudio |
| uv venv .venv |
| source .venv/bin/activate |
| uv pip install -e . |
| ``` |
| |
| ### 2. Download Model |
| |
| ```bash |
| hf download vanch007/FireRedAudio-MLX-8bit --local-dir models/FireRedAudio-8bit |
| ``` |
| |
| ### 3. Run Inference (CLI & Python) |
| |
| ```python |
| from fireredaudio_mlx import FireRedAudioInference |
|
|
| engine = FireRedAudioInference(model_path="models/FireRedAudio-8bit") |
| |
| # ASR Speech Transcription (RTF ~ 0.14) |
| res = engine.understand("assets/examples/asr_zh_fleurs.wav", task="asr") |
| print("Transcript:", res.answer) |
| |
| # Zero-shot TTS Voice Cloning (RTF ~ 0.75) |
| audio_res = engine.tts( |
| prompt_text="收到你的来信，我很高兴。", |
| prompt_audio="assets/examples/tts_zh_prompt.wav", |
| target_text="你好，欢迎体验 FireRedAudio 8-bit MLX 实时语音生成！", |
| ) |
| ``` |
| |
| ### 4. Launch WebUI Studio |
|
|
| ```bash |
| python run_webui.py --model models/FireRedAudio-8bit |
| ``` |
| Open your browser at **http://127.0.0.1:7860**. |
|
|
| --- |
|
|
| ## Citation & References |
|
|
| - FireRedAudio Official: [FireRedTeam/FireRedAudio](https://github.com/FireRedTeam/FireRedAudio) |
| - Apple MLX: [MLX Framework](https://github.com/ml-explore/mlx) |
| - MLX Port & WebUI Studio: [vanch007/mlx-FireRedAudio](https://github.com/vanch007/mlx-FireRedAudio) |
|
|
|
|
