Instructions to use adventists-ai/Sophon-Realtime-32B-Flash-4bit-MLX with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use adventists-ai/Sophon-Realtime-32B-Flash-4bit-MLX with MLX:
# Download the model from the Hub pip install huggingface_hub[hf_xet] hf download adventists-ai/Sophon-Realtime-32B-Flash-4bit-MLX --local-dir Sophon-Realtime-32B-Flash-4bit-MLX
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Atomic Chat
Sophon-Realtime-32B-Flash-4bit-MLX
No GPU? Try it through the hosted API: sign up and get 3 hours of free voice chat (OpenAI Realtime-compatible, docs).
Version 1.2.0 · changelog · code: mac/ · CUDA build: Sophon-Realtime-32B-Flash-4bit · 中文:README_CN.md
The full-duplex speech model Sophon-Realtime-32B-Flash-4bit, running locally on an Apple silicon Mac with MLX. You talk, it decides by itself when you have finished, answers with its own voice and sound tags, stops when you interrupt, and carries on when you just say "mm-hmm". Hearing, turn-taking and speaking are all done by the model on the Mac's GPU: no cloud, and no speech-to-text step in between.
The weights are the Flash-4bit weights repacked losslessly for MLX: the 4-bit values, scales, audio tower and voice stage are bit-identical to the CUDA build.
Requirements
- Apple silicon (M1 or later), macOS 14 or later
- 36 GB or more unified memory recommended; 32 GB works with other apps closed (about 26 GB in use while talking)
- About 24 GB download
- A microphone. Mac Studio and Mac mini have none built in: plug in a USB microphone or a headset. Speakers are fine, echo cancellation is built in
Quick start
git clone https://github.com/adventists-ai/sophon-realtime && cd sophon-realtime
bash mac/install.sh # Python environment + this repo's weights, into ~/sophon-mac
~/sophon-mac/sophon # talk in Chinese
~/sophon-mac/sophon --lang en # talk in English
Options: --no-aec (headset, no echo cancellation), --in / --out (audio devices, --list shows them), --voice.
Speed (M4 Max, 36 GB)
| You stop talking → it starts speaking | 0.9–1.1 s when the model is sure you have finished; up to about 2.5 s when it waits to see whether you go on |
| Long replies | real time, no stalls |
| You interrupt | it stops within about 0.15 s |
| Model load | about 10 s |
The CUDA server answers in about 0.5 s. On a Mac most of the time goes into the turn decision: when you pause, the model listens to your whole utterance once on the Mac's GPU. Chips with more GPU cores are faster.
How turn-taking works on a Mac
To process your audio only once, the turn question is asked inside the reply's own context (system prompt, conversation, your audio, then a short "has the user finished? A / B" question); if the answer is yes, the reply continues from the same computation. The check schedule and thresholds are those of the server (0.2 s to 2 s of silence, then a fallback reply). Accuracy on the Smart Turn test set (400 clips per language):
| English acc. / AUC | Chinese acc. / AUC | |
|---|---|---|
| Server turn question | 92.5 / 0.975 | 94.0 / 0.983 |
| Mac turn question | 90.5 / 0.960 | 94.3 / 0.977 |
Right after you interrupt, the full four-option question is used instead, so that a short back-channel ("mm-hmm", "yeah") makes it resume the cut reply.
Echo cancellation
WebRTC AEC3 removes the reply's own sound from the microphone. In a simulated room (speaker sound back at the microphone at −6 dB and 0 dB, 6 s of reply), the reply never interrupted itself with echo cancellation on (3 / 3 runs) and did every time with it off; a real interruption was still detected within about 0.12 s.
Status and limits
- Tested on an M4 Max (36 GB) with real-time simulated audio input (speech played into the engine at real-time pace, simulated room echo). Live microphone runs and numbers from other Macs are very welcome in issue #1.
- One conversation at a time. Camera input, tool calls and zero-shot voice cloning are not in the Mac version yet (built-in voices only).
Files
| Folder | What |
|---|---|
thinker_mlx/ |
the 4-bit thinker for mlx_lm (same weights as Flash-4bit) |
ear/ |
the thinker's audio tower and projector |
talker/ |
the voice stage (same as Flash) |
voices/ |
built-in voices |
License: code Apache-2.0; model weights Apache-2.0, see NOTICE for sources.
- Downloads last month
- 208
4-bit