PsarAI-S2S-GGUF

PsarAI-S2S speech recognition and speech synthesis for audio.cpp, in one GGUF per quantization:

• ASR: MMS-1B CTC with a code-switching adapter: Khmer, English, French and mixes of them in one sentence, no language setting, with word timestamps.
• TTS: VoxCPM2-Jarvis (one voice), Khmer, English and French, with voice cloning and voice design.

Family psarai_s2s. It needs audio.cpp with the patch in audiocpp/mms_asr.patch (families mms_asr and psarai_s2s; not in upstream audio.cpp yet).

Files

File	Size	ASR CER vs transformers	TTS CER (km / en / fr)
psarai-s2s-f16.gguf	6.70 GB	0.11%	1.4 / 0 / 0 %	reference
psarai-s2s-q8_0.gguf	4.00 GB	0.33%	1.4 / 0 / 0 %	recommended
psarai-s2s-q6_k.gguf	3.25 GB	0.36%	0 / 0 / 0 %
psarai-s2s-q5_k.gguf	2.84 GB	0.09%	1.4 / 0 / 0 %
psarai-s2s-q4_k.gguf	2.46 GB	0.64%	0 / 0 / 0 %	smallest at full quality
psarai-s2s-q3_k.gguf	2.05 GB	1.22%	2.9 / 0 / 0 %	small loss

ASR CER vs transformers: how far the GGUF's transcripts are from the transformers model's on 6 held-out clips (Khmer, English, mixed, French): quantization error, not recognition accuracy. TTS CER: one sentence per language synthesized and transcribed back by the ASR (single runs, so 0 vs 1.4% is run-to-run variation). A q2_k file was built and dropped: its speech was unintelligible.

Recognition accuracy of the ASR itself (transformers, content CER): grkpp Khmer news 9.89%, FLEURS Khmer 13.04%, FLEURS English 4.64%, FLEURS French 4.71%, synthetic Khmer–English mixed 2.18%, Khmer–French mixed 2.60%; see the PsarAI-S2S card.

Speed

RTX 3090, warm, same inputs for every engine (ASR: 59 s of Khmer, English, mixed and French speech; TTS: one Khmer, English and French sentence). RTF = compute seconds per second of audio (lower is faster).

Task	Engine	Wall	RTF
ASR	PsarAI-ASR-1B (V1) GGUF q8_0, audio.cpp server	0.45 s	0.008
ASR	PsarAI-S2S GGUF q8_0, audio.cpp server	0.39 s	0.007
ASR	PsarAI-ASR-1B (V1), transformers fp16	0.21 s	0.004
ASR	PsarAI-S2S, transformers fp16	0.24 s	0.004
TTS	VoxCPM2-Jarvis GGUF q8_0, audio.cpp server	2.84 s	0.206
TTS	PsarAI-S2S GGUF q8_0, audio.cpp server	2.93 s	0.213
TTS	VoxCPM2-Jarvis, Python voxcpm (10 steps)	7.12 s	0.484

The combined file runs each part as fast as the single-model files. audio.cpp's ASR times include the HTTP request and WAV decoding. The GGUF TTS is about 2.3× faster than Python VoxCPM2. The PsarAI-S2S transformers model also has a fast speech engine (speak, RTF 0.005), which is not in the GGUF.

Use with audio.cpp

Build audio.cpp with the patch (it applies to upstream commit 50749561; Unsloth Studio's audio.cpp v0.9.2 is 507495611b):

git apply audiocpp/mms_asr.patch
cmake -S . -B build -DENGINE_ENABLE_CUDA=ON -DAUDIOCPP_MODEL_SET=full && cmake --build build -j

One file, both tasks:

build/bin/audiocpp_cli --task asr --family psarai_s2s --model psarai-s2s-q8_0.gguf --backend cuda --audio speech.wav
build/bin/audiocpp_cli --task tts --family psarai_s2s --model psarai-s2s-q8_0.gguf --backend cuda \
  --text "សួស្តី លោក។ ខ្ញុំត្រៀមរួចរាល់ហើយ។" --out hello.wav

The server binds each model entry to one task, so list the file twice; each entry loads only its own part:

"models": [
  {"id": "psarai-s2s-asr", "family": "psarai_s2s", "path": "psarai-s2s-q8_0.gguf", "task": "asr", "mode": "offline"},
  {"id": "psarai-s2s-tts", "family": "psarai_s2s", "path": "psarai-s2s-q8_0.gguf", "task": "tts", "mode": "offline"}
]

• POST /v1/audio/transcriptions → text; POST /v1/audio/transcriptions/details → text + words (start_sample, end_sample, with sample_rate). Khmer is written without spaces between words, so Khmer timestamps are per phrase; English and French are per word.
• POST /v1/audio/speech ({"model": "psarai-s2s-tts", "input": "..."}) → 48 kHz WAV.

Unsloth Studio runs audio.cpp's prebuilt server; to use these files there, build it with the patch for Studio's audio.cpp commit and point Studio at the build (UNSLOTH_AUDIO_CPP_PATH).

Layout

The file holds the ASR part under asr/ and the asr_weights tensor prefix, and VoxCPM2-Jarvis under its own voxcpm2 names, plus copies of both families' specs (specs/), so each part is read by its family's own code. See audiocpp/psarai_s2s.md.

Limitations

• One synthetic voice that imitates a fictional character's voice; the rights to that voice belong to its owners.
• ASR output is lowercase; numbers are written as words.
• The ASR adapter for mixed speech drops some French apostrophes ("lusok" for "l'usok").
• Not in upstream audio.cpp: stock builds (and Unsloth's own prebuilt) cannot load psarai_s2s.

License

CC-BY-NC-4.0 (non-commercial), following facebook/mms-1b-all. VoxCPM2 is by OpenBMB; see PsarAI-S2S for data sources and credits.