zipformer-CarCMD-TH

Thai speech recognition for car voice commands. Say "เปิดแอร์ยี่สิบสี่องศา" or "ปิดกระจกฝั่งคนขับ" and it writes down what you said, offline, on a phone or car CPU. It's the speech front end of Lyra Car Assistant, which turns the text into a car action.

Under the hood: yfyeung/icefall-asr-gigaspeech2-th-zipformer-2024-06-20, a Thai Zipformer transducer trained with icefall on GigaSpeech2, fine-tuned here on car commands and everyday Thai. 4-bit encoder. 91 MB in all. sherpa-onnx loads it without conversion.

Files

file	size
encoder.int4.onnx + encoder.int4.onnx.data	6.5 + 81.7 MB	Zipformer encoder; MatMuls as 4-bit MatMulNBits (blocks of 128, int8 compute)
decoder.int8.onnx + .data	1.3 MB	stateless decoder, int8
joiner.int8.onnx + .data	1.0 MB	joiner, int8
tokens.txt, bpe.model		2,000 BPE pieces

Every weight sits in a .data file beside its graph. ONNX Runtime maps those from disk rather than copying them, so once it has decoded a clip the model holds about 170 MB of private memory.

Use it

import numpy as np, sherpa_onnx, soundfile as sf

rec = sherpa_onnx.OfflineRecognizer.from_transducer(
    encoder="encoder.int4.onnx", decoder="decoder.int8.onnx", joiner="joiner.int8.onnx",
    tokens="tokens.txt", num_threads=2, decoding_method="modified_beam_search")
audio, sr = sf.read("command.wav", dtype="float32")                # 16 kHz mono
lead = np.random.default_rng(0).standard_normal(sr // 2).astype(np.float32) * 1e-3
stream = rec.create_stream()
stream.accept_waveform(sr, np.concatenate([lead, audio]))
rec.decode_stream(stream)
text = stream.result.text.replace("\u0e4d\u0e32", "\u0e33")       # ํา -> ำ

Three details matter:

1. Put half a second of very quiet noise in front. Speech that starts on the first sample can lose its first word otherwise.
2. Turn ํา back into ำ. The BPE vocabulary went through NFKC normalization, so the model always writes ำ as ํ + า.
3. Trim long silence around the command before decoding, keeping about 500 ms on each side.

The text comes out as Thai without spaces, and numbers always as Thai words: "ขอแอร์ยี่สิบสี่องศา", never "24". If your app wants digits, convert them yourself, and watch for words that merely contain a number word, like สามารถ or ห้าม.

Tested with sherpa-onnx 2.0.1 (Python) on macOS. sherpa-onnx also has Android, iOS and WebAssembly builds; the 4-bit encoder needs one whose ONNX Runtime knows MatMulNBits (1.17 or newer).

Results

Character error rate, spaces ignored, numbers compared as Thai words. Lower is better.

	Gowajee test	FLEURS test (first 300)
starting model, its published int8 files	9.2%	6.1%
this model	6.3%	5.7%

Gowajee test: 2,717 spoken commands and requests from the corpus's 25 dev and test speakers, none of whom are in the training set. FLEURS is read, news-style Thai. It's in the table to catch the fine-tune making everyday Thai worse, which it didn't. Both were scored with the three steps above.

Training

Three epochs over 30.1 hours of audio (23,392 clips), from the published checkpoint, with icefall's zipformer/finetune.py. Base learning rate 0.002, 600 s of audio per batch, fp16, one RTX 4090. The training itself took about eight minutes. The exported weights average epochs 2 and 3.

Six epochs were tried too: better on synthetic voices, worse on real ones.

source	clips
VachanaTTS, Lyra's commands	5,286	1,762 Thai command sentences in its four voices, 3 renditions each
VachanaTTS, commands Lyra will get	2,019	windows, sunroof, A/C, defrost, screen, ambient light, apps, vehicle status
VoxCPM2 designed voices	2,913	47 voices made up from text descriptions (no real person's voice is cloned)
joined clips	3,000	2-3 short clips with pauses, so a pause isn't taken for the end
Gowajee train speakers	7,998	real people giving commands, half with car noise added
FLEURS Thai train	2,176	read Thai

Clean TTS sounds nothing like a car, so the synthetic clips got roughed up: speed changes, cabin reverb, a cheap-mic filter, and road, A/C fan, hiss, babble or music noise at 3-25 dB SNR. Labels write numbers as Thai words and English names in Thai letters (CarPlay → คาร์เพลย์). No recordings of private individuals went in.

Known issues

• Speech that starts on the very first sample sometimes loses its first word. The lead-in noise in Use it takes care of nearly every case.
• ำ comes out as ํา. See above.
• Loanwords are the weak spot. "Google Maps" tends to come out as "กูเกิล มาร์ท", and "ซันรูฟ" once in a while as "ซัน รู". CarPlay, Android Auto, ม่านบังแดด and เปอร์เซ็นต์ are fine now.
• Thai only: English names come out spelled in Thai.
• It's tuned for commands a few seconds long. Cut long recordings into pieces first; a VAD does that well.

License and credits

Apache-2.0, like the model it starts from.

• Starting model: yfyeung/icefall-asr-gigaspeech2-th-zipformer-2024-06-20 by Yifan Yang, Apache-2.0, revision e33c75c. This model changes its weights by fine-tuning and its encoder by 4-bit quantization.
• GigaSpeech2, what the starting model was trained on. Its dataset card says Apache-2.0, but SpeechColab doesn't own the audio and shares it for non-commercial research and education only. This model's fine-tuning didn't use that audio.
• FLEURS by Google, CC BY 4.0. Thai train clips used for fine-tuning, test clips for scoring; transcripts normalized and audio converted to 16 kHz mono.
• Gowajee corpus v0.9.3, Chulalongkorn University, MIT.
• VachanaTTS by VIZINTZOR, MIT, and VoxCPM2 by OpenBMB, Apache-2.0: the synthetic voices.
• icefall, k2, lhotse and sherpa-onnx by k2-fsa (Apache-2.0); ONNX Runtime (MIT); PyThaiNLP (Apache-2.0).
• Command sentences from Lyra Car Assistant.

@inproceedings{gigaspeech2,
  title={GigaSpeech 2: An Evolving, Large-Scale and Multi-domain ASR Corpus for Low-Resource Languages with Automated Crawling, Transcription and Refinement},
  author={Yifan Yang and Zheshu Song and Jianheng Zhuo and Mingyu Cui and Jinpeng Li and Bo Yang and Yexing Du and Ziyang Ma and Xunying Liu and Ziyuan Wang and Ke Li and Shuai Fan and Kai Yu and Wei-Qiang Zhang and Guoguo Chen and Xie Chen},
  booktitle={Proc. ACL},
  year={2025},
  address={Vienna},
}
@article{fleurs2022arxiv,
  title = {FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech},
  author = {Conneau, Alexis and Ma, Min and Khanuja, Simran and Zhang, Yu and Axelrod, Vera and Dalmia, Siddharth and Riesa, Jason and Rivera, Clara and Bapna, Ankur},
  journal={arXiv preprint arXiv:2205.12446},
  year = {2022},
}
@techreport{gowajee,
  title = {{Gowajee Corpus}},
  author = {Ekapol Chuangsuwanich and Atiwong Suchato and Korrawe Karunratanakul and Burin Naowarat and Chompakorn CChaichot and Penpicha Sangsa-nga and Thunyathon Anutarases and Nitchakran Chaipojjana and Yuatyong Chaichana},
  year = {2020},
  institution = {Chulalongkorn University, Faculty of Engineering, Computer Engineering Department},
  note = {Version 0.9.3},
  url = {https://github.com/ekapolc/gowajee_corpus}
}