Sophon-Realtime-32B-Flash-4bit-MLX

No GPU? Try it through the hosted API: sign up and get 3 hours of free voice chat (OpenAI Realtime-compatible, docs).

Version 1.2.0 · changelog · code: mac/ · CUDA build: Sophon-Realtime-32B-Flash-4bit · 中文:README_CN.md

The full-duplex speech model Sophon-Realtime-32B-Flash-4bit, running locally on an Apple silicon Mac with MLX. You talk, it decides by itself when you have finished, answers with its own voice and sound tags, stops when you interrupt, and carries on when you just say "mm-hmm". Hearing, turn-taking and speaking are all done by the model on the Mac's GPU: no cloud, and no speech-to-text step in between.

The weights are the Flash-4bit weights repacked losslessly for MLX: the 4-bit values, scales, audio tower and voice stage are bit-identical to the CUDA build.

Requirements

  • Apple silicon (M1 or later), macOS 14 or later
  • 36 GB or more unified memory recommended; 32 GB works with other apps closed (about 26 GB in use while talking)
  • About 24 GB download
  • A microphone. Mac Studio and Mac mini have none built in: plug in a USB microphone or a headset. Speakers are fine, echo cancellation is built in

Quick start

git clone https://github.com/adventists-ai/sophon-realtime && cd sophon-realtime
bash mac/install.sh            # Python environment + this repo's weights, into ~/sophon-mac
~/sophon-mac/sophon            # talk in Chinese
~/sophon-mac/sophon --lang en  # talk in English

Options: --no-aec (headset, no echo cancellation), --in / --out (audio devices, --list shows them), --voice.

Speed (M4 Max, 36 GB)

You stop talking → it starts speaking 0.9–1.1 s when the model is sure you have finished; up to about 2.5 s when it waits to see whether you go on
Long replies real time, no stalls
You interrupt it stops within about 0.15 s
Model load about 10 s

The CUDA server answers in about 0.5 s. On a Mac most of the time goes into the turn decision: when you pause, the model listens to your whole utterance once on the Mac's GPU. Chips with more GPU cores are faster.

How turn-taking works on a Mac

To process your audio only once, the turn question is asked inside the reply's own context (system prompt, conversation, your audio, then a short "has the user finished? A / B" question); if the answer is yes, the reply continues from the same computation. The check schedule and thresholds are those of the server (0.2 s to 2 s of silence, then a fallback reply). Accuracy on the Smart Turn test set (400 clips per language):

English acc. / AUC Chinese acc. / AUC
Server turn question 92.5 / 0.975 94.0 / 0.983
Mac turn question 90.5 / 0.960 94.3 / 0.977

Right after you interrupt, the full four-option question is used instead, so that a short back-channel ("mm-hmm", "yeah") makes it resume the cut reply.

Echo cancellation

WebRTC AEC3 removes the reply's own sound from the microphone. In a simulated room (speaker sound back at the microphone at −6 dB and 0 dB, 6 s of reply), the reply never interrupted itself with echo cancellation on (3 / 3 runs) and did every time with it off; a real interruption was still detected within about 0.12 s.

Status and limits

  • Tested on an M4 Max (36 GB) with real-time simulated audio input (speech played into the engine at real-time pace, simulated room echo). Live microphone runs and numbers from other Macs are very welcome in issue #1.
  • One conversation at a time. Camera input, tool calls and zero-shot voice cloning are not in the Mac version yet (built-in voices only).

Files

Folder What
thinker_mlx/ the 4-bit thinker for mlx_lm (same weights as Flash-4bit)
ear/ the thinker's audio tower and projector
talker/ the voice stage (same as Flash)
voices/ built-in voices

License: code Apache-2.0; model weights Apache-2.0, see NOTICE for sources.

Downloads last month
208
MLX
Hardware compatibility
Log In to add your hardware

4-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Collection including adventists-ai/Sophon-Realtime-32B-Flash-4bit-MLX