PsarAI-S2S

Speech and text in Khmer (ខ្មែរ), English and French with one model: speech recognition, translation, fast speech output, and speech-to-speech translation. Mixed speech (Khmer with English or French in the same sentence) needs no language setting.

Task	Method	Languages
asr speech → text	MMS-1B (wav2vec2, CTC) with a code-switching adapter	km, en, fr and any mix
translate text → text	NLLB-600M fine-tuned on km ↔ en ↔ fr	km, en, fr
speak text → speech	NLLB encoder → non-autoregressive unit decoder → unit HiFi-GAN (24 kHz)	km, en
s2st speech → text + speech	to Khmer: speech straight into the translator (no transcript in between); to English: ASR → MT	→ km, → en

Everything is in one model.safetensors (3.7 GB, fp16). The voice is a single synthetic voice ("Jarvis").

Usage

import torch, soundfile as sf
from transformers import AutoModel

model = AutoModel.from_pretrained("nphearum/PsarAI-S2S", trust_remote_code=True, dtype=torch.float16).to("cuda")

text = model.asr("speech.wav")                          # Khmer / English / French / mixed
text = model.asr("speech.wav", language="fra")          # "eng" or "fra": best accuracy for one language
model.translate("Good evening, sir.", to="km")          # -> សាយ័ណ្ហសួស្តី លោក។   (to: km | en | fr)
wav = model.speak("សួស្តី លោក។", lang="km", pace=1.15)  # numpy float32 at model.sample_rate (24 kHz)
text, wav = model.s2st("speech.wav", to="km")
sf.write("out.wav", wav, model.sample_rate)

Requirements: transformers, torch, soundfile, soxr. Audio input: a file path, or a numpy array with sampling_rate=. pace > 1 makes the voice slower (it speaks a little fast by default).

Results

Speech recognition, content CER % (letters only), greedy, compared with PsarAI-ASR-1B (V1):

Test set	V1	PsarAI-S2S asr	with language=
grkpp, Khmer news, new speakers	13.72	9.89
khmer10, unseen speaker	4.84	3.44
DDD held-out (clips not in training)	5.58	4.10
FLEURS Khmer test	14.05	13.04
FLEURS English test	4.74	4.64	3.89 (eng)
FLEURS French test	—	4.71	3.51 (fra)
Jarvis Khmer–English mixed (synthetic)	1.83	2.18
Jarvis Khmer–French mixed (synthetic)	21.23	2.60

Translation, chrF on 200 held-out segments per direction (higher is better), before → after fine-tuning: km→en 73.2 → 80.3, km→fr 64.8 → 74.0, en→km 77.0 → 82.4, en→fr 82.5 → 87.4, fr→km 68.9 → 76.6, fr→en 83.0 → 86.6 (mean 74.9 → 81.2). The references are machine translations of Khmer web text, so this measures agreement with them, not human-judged quality.

Speech output (speak), intelligibility: speech is generated from held-out Jarvis transcripts and transcribed back by ASR (CER, lower is better; the original recordings score 0.017 / 0.004 / 0.030): Khmer 0.055, English 0.042, mixed 0.189. Speed on an RTX 3090: about 0.01 s of compute per second of speech (a 5 s sentence in about 0.05 s); about 1 s on CPU.

Speech → Khmer (s2st(to="km")) on held-out Jarvis clips: CER 0.010 (Khmer speech), 0.037 (English), 0.100 (mixed) against reference translations; 0.6–1.3 s per clip on GPU.

How it works

• ASR: the MMS-1B base stays frozen; small adapters (~2.3M parameters each) and output heads are trained. The default adapter covers Khmer, English, French and code-switching (vocabulary: Khmer script, Latin letters, French accents, | word separator). eng and fra adapters are the MMS ones for single-language speech.
• Speech → Khmer uses a 9M-parameter bridge that maps all 49 ASR layers into the translator's input space, so the translator reads speech directly instead of a transcript (this avoids passing ASR mistakes on). It was trained with the original khmer-translate-600m decoder, which is kept for it (bridge_decoder).
• Speech output: a non-autoregressive decoder predicts discrete speech units (mHuBERT-147, k-means 1,000) and their durations from the translator's encoder states; a HiFi-GAN turns units into 24 kHz audio.

Limitations

• One synthetic voice that imitates a fictional character's voice; the rights to that character's voice belong to their owners. No voice cloning in this model.
• Speech output is Khmer and English only; French output is not trained yet. Long or rare words can come out blurred; mixed sentences are less clear than single-language ones.
• Speech → English goes through ASR, so recognition errors reach the translation. Speech → Khmer does not.
• Domain: speech output and the mixed-speech tests use synthetic "Jarvis" speech; real Khmer is tested on grkpp, DDD and FLEURS. The translation training data are machine translations of web text.
• Khmer លោក ("you", polite) is sometimes translated as "he".
• Output of the ASR is lowercase; spaces in Khmer follow phrase boundaries, not word boundaries.

License and credits

CC-BY-NC-4.0 (non-commercial), following facebook/mms-1b-all and NLLB-200. Speech units: utter-project/mHuBERT-147. Training data: Digital-Divide-Data/khmer-speech-dataset (CC-BY-SA-4.0), google/fleurs (CC-BY-4.0), Common Voice French via ele-sage/french-asr-quebec-eu (CC0), nphearum/dataset-asr-tts (CC-BY-NC-4.0), and machine-translated nphearum/khmer-raw-text-3M text. Check that your use is permitted before using this model.