Sophon-Realtime-32B-Flash-4bit

Version 1.1.6 · changelog · full-size build: Sophon-Realtime-32B-Flash

Sophon-Realtime-32B-Flash with its thinker compressed to 4-bit weights, so the whole full-duplex speech model (listen, decide turns, write the reply with sound tags, speak) fits on one 32 GB GPU instead of an 80 GB one. Same code, same API, same voices and voice cloning as Flash; only the thinker weights and the deploy config differ.

	Flash-4bit (this repo)	Flash
GPU memory in use	about 30 GB	about 55 GB
Meant for	one or two conversations at a time	up to about 40
Download	about 25 GB	about 41 GB
Image input	off (to save memory)	on
Context	4096 tokens	8192 tokens

What changes with 4-bit (one H100, same scripts for both builds, CUDA MPS on)

	Flash-4bit	Flash
CMMLU, 2000 questions (Chinese multiple choice)	85.5	87.1
MMLU-Pro, 2000 questions (English, 10 options)	57.1	58.4
Replies that use a sound tag (600 held-out spoken turns)	13.0%	14.2%
Speech error rate of the spoken reply, Chinese, 80 replies (median)	1.6%	0.8%
Speech error rate, English, 69 replies (median)	0%	0%
Turn-taking, 40 realtime turns: started too early / no reply	11 / 1	10 / 2

• Knowledge drops by about 1.5 points (statistically significant on 2000 paired questions). Tag writing, speech clarity and turn-taking are unchanged within noise.
• Long replies (around 100 s) get less accurate in their second half in both builds; this comes from the voice stage, not from the 4-bit thinker.

Speed

/v1/chat/completions, user speech → first audio chunk; "behind" = how far playback falls behind real time (0 = never stalls).

One H100, 32 GB config	First audio median / P90	Behind
1 conversation, 10 turns, Chinese / English	0.24 / 0.26 s · 0.24 / 0.27 s	0
2 conversations starting in the same second, long replies	0.29–0.32 s	0
2 conversations × 10 turns	0.27 / 0.34 s	0
/v1/realtime, end of user speech → first audio	0.60 / 0.75 s	–

One RTX PRO 6000 Blackwell (same generation as the RTX 5090), 32 GB config	First audio	Behind
1 conversation	0.27–0.41 s	0
2 conversations starting in the same second, long replies	0.37–0.48 s	0.3–0.75 s at the start of the reply

• On this GPU generation two simultaneous long replies stall briefly at the start; the full-size Flash behaves the same there. One conversation is clean.
• Not yet run on an actual RTX 5090. Memory was measured at 30.5 GB on the PRO 6000 with a 32 GB budget; a 5090 has 32.6 GB, so run it headless and with nothing else on the GPU. Feedback welcome in issues.
• 24 GB GPUs are not enough: the 4-bit thinker alone needs about 23 GB at start-up.

Quick start

git clone https://github.com/adventists-ai/sophon-realtime && cd sophon-realtime/flash
bash install.sh                                   # once: Python 3.12 venv, vLLM 0.30, vLLM-Omni 0.30, plugin
hf download adventists-ai/Sophon-Realtime-32B-Flash-4bit --local-dir ./model
CONFIG=sophon_realtime_flash_32g.yaml GPU=0 PORT=8000 bash start.sh
python client_example.py --wav my_question.wav --lang en
bash stop.sh

• The first start compiles kernels (about 5–10 min) and needs the CUDA toolkit (CUDA_HOME). start.sh turns CUDA MPS on so the three stages share the GPU.
• With more memory (48 GB and up) use the 4-bit weights with a larger budget: copy the config and raise gpu_memory_utilization, max_model_len and max_num_seqs. On an H100 the 4-bit weights with the full-size settings use about 42 GB and handle 8 simultaneous long replies with at most 0.2 s behind.
• Everything else (API, sound tags, voices, zero-shot voice cloning, system prompt) is the same as in the Flash card and the repository.

License: code Apache-2.0; model weights Apache-2.0, see NOTICE for sources.