Add files using upload-large-folder tool

108f5bd verified 2 months ago

2.69 kB

language:
  - zh
  - en
license: apache-2.0
tags:
  - mlx
  - apple-silicon
  - audio
  - speech-recognition
  - text-to-speech
  - voice-cloning
  - speech-editing
  - voice-design
  - multimodal
pipeline_tag: text-to-speech

FireRedAudio-MLX-8bit

Apple Silicon MLX 8-bit quantized checkpoint for FireRedAudio (9B multimodal audio language model).

🐙 GitHub Repository & WebUI Studio: vanch007/mlx-FireRedAudio
⚡ Hardware Acceleration: Metal GPU native unified memory, fused SDPA attention, and Gated Delta Metal kernel.
🚀 Real-Time Factor (RTF):
- ASR (Speech Recognition): RTF 0.1396 (~7.2x real-time speedup)
- TTS (Voice Cloning): RTF 0.7529 (Real-time speech generation on Apple M3 Max)
💾 Memory Footprint: Only 13.54 GB active Metal memory (runs smoothly on 16GB / 24GB+ Mac devices).

Model Architecture & Quantization Details

Backbone LLM: 9B Qwen3.5 language model with selective 8-bit affine quantization (group_size=64).
Continuous Audio Autoencoder / Vocoder: RedAE VAE and Native ISTFT vocoder preserved in full FP32/BF16 precision for lossless audio phase fidelity.
Audio Understanding Encoder: Whisper / Conv1D acoustic encoder preserved in high precision.
Audio Generation Head: 11-layer Flow Matching DiT.

Quick Start

1. Installation

git clone https://github.com/vanch007/mlx-FireRedAudio.git
cd mlx-FireRedAudio
uv venv .venv
source .venv/bin/activate
uv pip install -e .

2. Download Model

hf download vanch007/FireRedAudio-MLX-8bit --local-dir models/FireRedAudio-8bit

3. Run Inference (CLI & Python)

from fireredaudio_mlx import FireRedAudioInference

engine = FireRedAudioInference(model_path="models/FireRedAudio-8bit")

# ASR Speech Transcription (RTF ~ 0.14)
res = engine.understand("assets/examples/asr_zh_fleurs.wav", task="asr")
print("Transcript:", res.answer)

# Zero-shot TTS Voice Cloning (RTF ~ 0.75)
audio_res = engine.tts(
    prompt_text="收到你的来信，我很高兴。",
    prompt_audio="assets/examples/tts_zh_prompt.wav",
    target_text="你好，欢迎体验 FireRedAudio 8-bit MLX 实时语音生成！",
)

4. Launch WebUI Studio

python run_webui.py --model models/FireRedAudio-8bit

Open your browser at http://127.0.0.1:7860.

Citation & References

FireRedAudio Official: FireRedTeam/FireRedAudio
Apple MLX: MLX Framework
MLX Port & WebUI Studio: vanch007/mlx-FireRedAudio