PsarAI-Speech

Khmer, English and French (and mixed km+en / km+fr) speech recognition and synthesis, in the Jarvis voice or a voice cloned zero-shot from a few seconds of someone's speech. No translation: the output language follows the input.

mode	path
asr	MMS-1B CTC ASR ("auto" adapter) + character 6-gram beam search
tts	numbers -> words, characters -> speaker-conditioned T2U -> mHuBERT units (+ unit 4-gram beam search), durations, pitch -> fast: unit HiFi-GAN with speaker + pitch input, or flow (clone v3): UnitFront + S3Gen flow-matching decoder that continues the reference recording + HiFT, 24 kHz; listen-back check with the ASR

from psarai_speech import Speech
speech = Speech()
print(speech.asr("in.wav"))
wav = speech.tts("សួស្តី លោក។ Bonsoir, monsieur.")             # numpy float32, 24 kHz, Jarvis (fast)
voice = speech.voice("reference.wav")                            # prepare once, reuse
wav = speech.tts("Good evening, sir.", voice=voice)              # cloned voice (flow)
wav = speech.tts("Good evening, sir.", voice=voice, engine="fast")   # ~15x faster, copies only voice type

python psarai_speech.py asr --audio in.wav
python psarai_speech.py tts --text "Good evening, sir." --out out.wav --voice ref.wav   # --engine fast|flow, --fast: no listen-back

Requirements: torch, transformers, safetensors, kenlm, numpy, soundfile, soxr, rapidfuzz; for voice cloning also torchaudio, pyworld, librosa, scipy, einops, diffusers (conformer is not needed).

Listen

Generated by the audio.cpp server from the GGUF in this family (the Python package gives the same output). Heard is what the PsarAI-Speech ASR transcribes from the audio, automatically (its own spelling slips included).

Jarvis voice (default)

	Text	Listen	Heard
Khmer	សួស្តី! ថ្ងៃនេះអាកាសធាតុល្អណាស់ ហើយខ្ញុំរីករាយដែលបានជួបអ្នកម្តងទៀត។ Hello! The weather is lovely today, and I am happy to see you again.		សួស្តី ថ្ងៃអ្នកកាសធាតុល្អណាស់ ហើយខ្ញុំរីករាយដែលបានជួបអ្នកម្តងទៀត។
English	Hello! This voice was cloned from a ten-second sample. The weather is lovely today.		hello this voice was closed from a ten second sample the weather is lovely today
French	Bonjour ! Cette voix a été clonée à partir d'un court extrait. Il fait très beau aujourd'hui. Hello! This voice was cloned from a short sample. The weather is lovely today.		bonjour cette voix état clonée à partir d'un cours extrai. il fait très beau aujourd'hui.
Khmer + English	ថ្ងៃនេះ ខ្ញុំមាន meeting ជាមួយ team នៅម៉ោងដប់ ហើយបន្ទាប់មកទៅញ៉ាំបាយ។ Today I have a meeting with the team at ten, and then I'll go eat.		ថ្ងៃនេះ ខ្ញុំមានមីទិញជាមួយធីមនៅម៉ោងដប់ ហើយបន្ទាប់មកទិញញ៉ាំបាយ។
numbers (km + en)	តម្លៃ 25 ដុល្លារ ក្នុងឆ្នាំ 2026។ The price rose 15% to $1,234.50. Price: 25 dollars in 2026. (numbers are read as words in each language)		កម្លែង ម្ភៃប្រាំដាក្នុងឆ្នាំពីរពាន់ម្ភៃប្រាំមួយ។ the price rose fifteen percent to one thousand two hundred and thirty-four point fifty a dollars.

Zero-shot voice cloning (clone v3)

Each voice from one ~10 s Khmer clip; the same clip speaks Khmer, English and French. Similarity = CAM++ cosine to the reference (two real clips of one speaker ≈ 0.8).

Voice A: elderly woman, warm: reference (9.6 s)

	Text	Cloned	Heard	similarity
Khmer	សួស្តី! ថ្ងៃនេះអាកាសធាតុល្អណាស់ ហើយខ្ញុំរីករាយដែលបានជួបអ្នកម្តងទៀត។ Hello! The weather is lovely today, and I am happy to see you again.		សួស្តី ថ្ងៃនេះ អាកាសធាតុល្អណាស់ ហើយខ្ញុំរីករាយដែលបានជួបអ្នកម្តងទៀត។	0.74
English	Hello! This voice was cloned from a ten-second sample. The weather is lovely today.		this voice was cloned from a ten second sample the weather is lovely today	0.78
French	Bonjour ! Cette voix a été clonée à partir d'un court extrait. Il fait très beau aujourd'hui. Hello! This voice was cloned from a short sample. The weather is lovely today.		bon joue doit été cloné à partir un cours extrait il fait très beau aujourd'hui.	0.79

Voice B: middle-aged woman, deep: reference (8.5 s)

	Text	Cloned	Heard	similarity
Khmer	សួស្តី! ថ្ងៃនេះអាកាសធាតុល្អណាស់ ហើយខ្ញុំរីករាយដែលបានជួបអ្នកម្តងទៀត។ Hello! The weather is lovely today, and I am happy to see you again.		សួស្តី ថ្ងៃនេះ អាកាសធាតុល្អណាស់ ហើយខ្ញុំរីករាយដែលបានជួបអ្នកម្តងទៀត។	0.80
English	Hello! This voice was cloned from a ten-second sample. The weather is lovely today.		hellois voice was cloned from a ten second sample the weather is lovely today	0.75
French	Bonjour ! Cette voix a été clonée à partir d'un court extrait. Il fait très beau aujourd'hui. Hello! This voice was cloned from a short sample. The weather is lovely today.		mon jour cette voie était clonnée à partir d'un cours extrait il fait très beau aujourd'hui.	0.77

Voice C: young man, deep and smooth: reference (8.8 s)

	Text	Cloned	Heard	similarity
Khmer	សួស្តី! ថ្ងៃនេះអាកាសធាតុល្អណាស់ ហើយខ្ញុំរីករាយដែលបានជួបអ្នកម្តងទៀត។ Hello! The weather is lovely today, and I am happy to see you again.		សួស្តី ថ្ងៃនេះ អាកាសធាតុល្អណាស់ ហើយខ្ញុំរីករាយដែលបានជួបអ្នកម្តងទៀត។	0.83
English	Hello! This voice was cloned from a ten-second sample. The weather is lovely today.		hello this voice was closed from a tenecken sample the weather is lovely today	0.73
French	Bonjour ! Cette voix a été clonée à partir d'un court extrait. Il fait très beau aujourd'hui. Hello! This voice was cloned from a short sample. The weather is lovely today.		bonjour cette voit été clonnée appartient cours extrait. il fait très beau aujourd'hui.	0.77

Same voice, two engines (Voice C, Khmer):

Engine	Listen	similarity
fast (unit HiFi-GAN, ~100× real time)		0.56
flow (clone v3, default with voice_ref)		0.83

Voice cloning (clone v3)

speech.voice(clip) takes the first 10 s of the reference: a CAM++ speaker embedding (for the T2U: units, durations and pitch in that speaker's way) and the clip itself as a prompt (mHuBERT units, WORLD pitch, mel). The flow decoder (S3Gen from Chatterbox, CosyVoice2 design, fine-tuned 60k steps on our km/en/fr units with new input layers) renders the new sentence as a continuation of the recording, so timbre, accent and recording character follow the reference. Use clean speech: noise in the reference is copied too.

Results

20 held-out clips per group, text -> speech -> ASR content CER; similarity = CAM++ cosine to the reference clip (greedy units, pace 1.07, no listen-back).

reference voices	CER fast	CER flow	similarity fast	similarity flow
DDD Khmer speakers (held out)	12.1%	7.6%	0.47	0.73
designed voices (held out)	12.8%	7.0%	0.47	0.79
FLEURS Khmer (noisy real recordings)	41.9%	31.2%	0.35	0.70
FLEURS English	45.1%	24.9%	0.34	0.73
FLEURS French	26.4%	18.9%	0.40	0.73
Jarvis	4.9%	4.1%	0.74	0.81
3 real voices, km / en / fr / mixed	19.9%	13.9%	0.47	0.73

Two real clips of the same speaker give ~0.8 similarity (DDD 0.79, designed 0.83). CAM++ also conditions the flow decoder, so its similarity is somewhat optimistic: listen.

ASR with the character 6-gram: FLEURS km CER 12.4% -> 9.3%, mixed held-out 2.2% -> 1.6%.

License

CC-BY-NC-SA-4.0 (non-commercial, share-alike): mhubert/ is utter-project/mHuBERT-147 (CC-BY-NC-SA-4.0, first 9 layers), which also defines the speech units. s3gen/ code and the S3Gen / HiFT / CAM++ weights come from ResembleAI/chatterbox (MIT, licence in s3gen/LICENSE).