Sophon-Realtime-32B-Flash

A full-duplex speech model served as one vLLM-Omni model: it listens to the user, writes a reply (optionally with sound tags [laughter] [cough] [sigh] [breath] <laughter>…</laughter> <strong>…</strong>), and speaks while it writes. The voice stage emits one frame every 80 ms; tags are performed, never read aloud.

Flash shares the tag-writing thinker with the Tag build; only the voice stage differs. Flash's voice stage is autoregressive with a single forward pass per step, so first audio is faster and concurrency is higher; the Tag build laughs more convincingly. Both have the same API.

Sophon-Realtime v1 family

Three builds share one brain (the same 32B model listens, decides turns and writes the reply). They differ only in how the reply becomes sound. Full comparison: github.com/adventists-ai/sophon-realtime.

	Tag (recommended)	Flash	E2E (preview)
How it speaks	an expressive speech model performs the sound tags	a lighter, faster speech model speaks the tagged reply	a talker inside the 32B model, no separate speech model
NVV-SuperBench (en / zh)	0.80 / 0.86	0.63 / 0.77	0.60 / 0.77
First audio, one user	~0.6 s	0.31 s	~1 s
Users per 80 GB GPU (recommended)	16	40	4–5
GPUs	1	1	2
Weights	Sophon-Realtime-32B-Tag	Sophon-Realtime-32B-Flash	Sophon-Realtime-32B-E2E

Pick Tag if you want the voice to act; pick Flash if latency and users per GPU matter most. Same APIs.

Highlights

• VoiceBench 82.8 and Big Bench Audio 81.8%: the strongest brain among open full-duplex models. #1 among open full-duplex models on both; level with GPT-4o-mini-Audio on VoiceBench.
• Paralinguistic understanding 48.6, #1 overall: it hears your tone. Ahead of Gemini 3.8 Flash (46.9) and Qwen3-Omni (44.6) on LISTEN, StressTest and MSPB. It hears emotion, emphasis, sarcasm, laughter and sighs straight from your voice.
• The fastest build: 0.31 s to first audio, up to 40 concurrent users per 80 GB GPU. Made for customer service, in-car assistants and other many-user deployments.
• Still laughs and clears its throat: NVV-SuperBench 0.77 (Chinese) and 0.63 (English). It speaks the same tagged replies as Tag with a lighter, faster voice.
• Full-Duplex-Bench v3 tool calling: Pass@1 48%, ahead of Grok (43%), with a 100% response rate. Live mode is compatible with the OpenAI GPT-Live interface.
• Real full duplex. It listens the whole time you talk and drafts its reply as you go; it picks up when you stop, stops when you cut in, and keeps going when you just say "mm-hm".
• Apache-2.0, free for commercial use. Fine-tune persona, tone and domain knowledge with ms-swift.

Benchmark methods and full comparison tables: v1 family home page. To just try it: talk to it in the ModelScope Studio.

Performance (one H100 80 GB, FP8 thinker, /v1/chat/completions, user speech → first audio chunk)

Load	First audio median / P90	Stall (median)
1 stream	0.31 / 0.34 s	0
20 streams (arriving over 10 s)	0.45 / 0.52 s	0.09 s
40 streams (over 10 s)	0.58 / 0.67 s	0.37 s
64 streams (over 8 s)	0.84 / 1.03 s	1.9 s (queueing)

• Up to about 40 concurrent streams per 80 GB GPU is recommended. For more, run one instance per GPU behind a load balancer.
• Memory: about 55 GB per instance. The first start compiles kernels (about 10 min); later starts take 5–8 min.

Sound tags (held-out set, Chelsie voice, judged by DuplexJev; not comparable with the AST hit rates on the Tag card — for a like-for-like comparison of the three builds, see NVV-SuperBench above)

Tag	zh	en
[laughter]	83%	77%
<laughter>…</laughter> laughing while speaking	78%	71%
[cough]	90%	50% (8 items)
[sigh] (AST)	90%	100%
Mean character / word error rate	0.9%	1.0%

• Tags are never read aloud; without a tag the model rarely laughs on its own.
• <strong> emphasis is still weak in this build; the next build will improve it.

Deployment code

One-command scripts and the vLLM-Omni plugin: github.com/adventists-ai/sophon-realtime/tree/main/flash. This repository holds the weights: config.json, thinker/ (FP8), talker/, voices/.

Quick start

bash install.sh                 # once: Python 3.12 venv, vLLM 0.30, vLLM-Omni 0.30, plugin
# put the weights in ./model (hf download adventists-ai/Sophon-Realtime-32B-Flash --local-dir ./model)
GPU=0 PORT=8000 bash start.sh
python client_example.py --wav my_question.wav --lang en --voice chelsie
bash stop.sh

Voices: chelsie (default), brand; set language to zh or en.

API

• /v1/chat/completions: use the system prompt in config/system_prompt.json (the model was tuned with it; without it fewer tags are written); put the user's speech in input_audio; "modalities": ["text","audio"], optional "voice" and "language". The text stream keeps the tags (strip them for display if you like); the audio stream is base64 WAV chunks (24 kHz).
• /v1/realtime: OpenAI Realtime-style session; keep sending input_audio_buffer.append; the model decides when the user has finished.

Live mode (OpenAI GPT-Live compatible)

Run one more command next to the engine and GPT-Live clients can connect by changing only the URL and key. Sophon keeps the voice; when a request needs a lookup or an action it says "Let me check that", hands the task to your backend and speaks the result in its own words.

cd sophon-realtime/live && BUILD=flash PORT=8000 LIVE_PORT=8780 bash start_live.sh   # GPT-Live clients: ws://HOST:8780/v1/live/sessions

Details and measurements: live/ in the GitHub repository.

Fine-tuning

Fine-tune the thinker's LLM backbone with LoRA in ms-swift to change persona, tone, domain knowledge and tool use. Hearing, the voice and turn-taking stay as released, and sound tags such as laughter and coughs are kept. One 80 GB GPU is enough; see finetune/ in the GitHub repository.

Layout

model/ (thinker, voice stage, voices) · plugin/ (vLLM-Omni plugin sophon_realtime) · config/ (deploy config, system prompt, VAD) · examples/

License: code Apache-2.0; model weights Apache-2.0, see NOTICE for sources.