---
library_name: transformers
license: other
license_name: modified-mit
datasets:
- Digital-Divide-Data/khmer-speech-dataset
- openslr/librispeech_asr
- KrorngAI/openslr-librispeech_asr-clean100-00000-19999-km-translate
- KrorngAI/DDD-Cambodia-khmer-speech-dataset-parquet-000-159-en-translate
- W4ng1204/Nonspeech7k
language:
- km
- en
metrics:
- wer
- cer
- ter
pipeline_tag: automatic-speech-recognition
---
# TrorYongASR2
> [!Note]
> This repository contains model weights and configuration files for the pre-trained model.
>
## Model Details
### Model Description
**TrorYongASR2**, an update to [TrorYongASR1](https://huggingface.co/KrorngAI/TrorYongASR-small), augments the auditory-lingual decoder to 1) improve speech-text alignment, 2) enhance the expressivity of rotary position embedding, 3) ameliorate the expressivity of multi-head attention, and 4) speed up training and inference times.
For speech-text alignment, Length-Aware RoPE (LARoPE) is implemented in the cross-attention to enhance monotonic structure alignment.
In addition, the lowest frequencies of RoPE in the context-position attention and LARoPE in speech-text alignment are removed to create robust semantic channels.
To improve multi-head attention, a gating mechanism is integrated in the self-attention and the cross-attention layers of the auditory-lingual decoder: apply a head-specific sigmoid gate after the scaled dot-product attention.
This introduces non-linearity upon the low-rank mapping in the softmax attention which mitigates 'massive activation' and 'attention sink'.
Finally, the auditory-lingual decoder adopts point-wise dynamic rescaled Gaussian cumulative distribution function (Derf) in place of normalization layers to save compute, hence accelerating training speed and inference process.
[Figure 1](#fig-archit) provides an overview of the TrorYongASR2's architecture.
Figure 1: TrorYongASR2 architecture. Dropout layers are omitted due to space constraints, [B], [L], [T], [E], and [P] are begin-of-sequence, language, task, end-of-sequence, and pad tokens respectively.
This figure presents the case having 16 distinct target prediction positions.
The auditory-lingual decoder is just a single normalization-free Transformer where the standard normalization layers are replaced by point-wise Dynamic rescaled Gaussian cumulative distribution function (Derf).
All projections in attention layers are explicitly shown here because specific to TrorYongASR2, the single position basis \\( \boldsymbol{p} \\) is used for each position to directly form query vectors (differently from the standard decoder, the linear layer for query projection is not required) while context vectors are the token embeddings multiplied by the position basis and are projected into key and value vectors.
With gated mechanism, masked multi-head attention (Masked MHA) generates gate score from context vectors while MHA for speech-text alignment computes gate score from the hidden state, the output of Masked MHA.
Inspired by QK-normalization, a Derf activation is applied to the query and key vectors of Masked MHA right before \\( p \\)-RoPE transformation.
For the cross-attention layer, the query and key vectors are transformed by first Derf and then \\( p \\)-LARoPE.
Attention masks are generated from randomly chosen permutations and are used only for the context-position attention.
The last linear layer outputs logits over the vocabulary set.
These logits are then used to compute cross-entropy loss.
The audio encoder is simply Whisper's Audio Encoder, but it can process audio of arbitrary length that is shorter than 30 seconds.
- **Developed by:** KHUN Kimang (Ph.D.)
- **Shared by:** KrorngAI
- **Model type:** ASR (Automatic Speech Recognition)
- **Language(s) (NLP):** Khmer and English
### Model Sources
- **Blog Post:** [https://kimang18.github.io/krorngai-blog/TrorYongASR2/](https://kimang18.github.io/krorngai-blog/TrorYongASR2/)
- **Demo:** [https://huggingface.co/spaces/Kimang18/TrorYongASR2-Demo](https://huggingface.co/spaces/Kimang18/TrorYongASR2-Demo)
- **Repository:** https://github.com/Kimang18/KrorngAI/tree/main/tror-yong-asr
## Evaluation
### Dataset
The evaluation assesses transcription capability on two datasets: 1) [`google/fleurs`](https://huggingface.co/datasets/Kimang18/google-fleurs-km-kh) for Khmer and 2) [`openslr/librispeech_asr`](https://huggingface.co/datasets/openslr/librispeech_asr) for English.
All results are from the **test split** of each dataset.
| Dataset | Language | Testing samples | Description |
|-----------------------|------------|------------------|---------------------------------------------------|
| **google/fleurs** | Khmer | 765 | Multi-lingual dataset with Khmer language samples |
| **librispeech_asr.clean** | English | 2611 | Clean speech dataset for English transcription |
Audio longer than 30 seconds are excluded from the evaluation (that is why `google/fleurs` has 765 samples instead of 771 and `librispeech_asr.clean` has 2611 instead of 2620).
### Metrics
For transcription task, 3 metrics below are used
- **Token Error Rate (TER)**: Proportion of incorrectly predicted tokens
- **Character Error Rate (CER)**: Proportion of characters that are incorrect
- **Word Error Rate (WER)**: Proportion of words that are incorrect
TER measures model's capability in predicting the next token given the audio input and the current sequence of tokens.
This metric is weaker than WER and CER because it doesn't account for insertions, deletions, substitutions, and autoregression as comprehensively.
### Results
| Model | Metric | Khmer (`fleurs`) | English (`librispeech_asr.clean`) |
|-----------|--------|------------------|-------------------------------|
| **Tiny** | WER | 57.07% | 20.35% |
| | CER | 43.95% | 12.42% |
| | TER | 49% | 11% |
| **Small** | WER | 36.70% | 10.12% |
| | CER | 29.26% | 6.05% |
| | TER | 38% | 5% |
Both configurations performance well on English language.
However, Khmer language remains a challenge.
**Note:** To compute `CER` and `WER`, whitespaces are added between words in Khmer text (Khmer text does not have word boundaries like English text). To do so, `khmercut` PyPI package is used to tokenize Khmer text into words, and then the words are joined back together with whitespaces.
It is due to no-boundary Khmer words that Token Error Rate is utilized as one of the metrics.
**Comparison with TrorYongASR1 and Whisper:**
| Size | Model | Parameters | Khmer (`fleurs`) | English (`librispeech_asr.clean`) |
| -------- | ------------ | ---------- | --------------------------- | ----------------------------- |
| **Tiny** | **TrorYongASR2** | 29M | **57.07%** | 20.35% |
| | TrorYongASR1 | 29M | 75.88% | 54.33% |
| | Whisper | 39M | 100.6% | **7.6%** |
| | | | | |
| **Small**| **TrorYongASR2** | 136M | **36.70%** | 10.12% |
| | TrorYongASR1 | 135M | 50.46% | 21.75% |
| | Whisper | 244M | 104.4% | **3.4%** |
**Key Observations:**
- As mentioned above, TrorYongASR2 clearly outperforms TrorYongASR1 and requires just a small increase in the number of parameters (the increase comes from additional length predictor and gate projection layers)
- Whisper has the highest number of parameters for comparable configuration (39M vs 29M for `tiny`, and 244M vs 136M for `small`)
- Whisper shows significantly lower word error rates on English (7.6% for `tiny`, 3.4% for `small`)
- TrorYongASR2 performs best on Khmer dataset (57.07% for `tiny`, and 36.70% for `small`)
**Note:** `WER` data of Whisper is taken directly from their [paper](https://arxiv.org/abs/2212.04356).
**Note on Translation Task:** According to [KrorngAI](https://huggingface.co/KrorngAI), the 2 translation datasets above are built using the API of [TranslateKH](https://translatekh.mptc.gov.kh/) which, in turn, uses an AI model for the translation (Khmer to English and vice-versa).
Without any means to check the correctness of the translation over 209 hours of audio, TrorYongASR2 is not evaluated on any translation datasets.
It can be assumed that the translation performance of TrorYongASR2 is upper bounded by the AI model used in the backend of [TranslateKH](https://translatekh.mptc.gov.kh/).
## Training
TrorYongASR2 is trained on 5 datasets:
- 234 hours of `Digital-Divide-Data/khmer-speech-dataset` (the whole dataset contains 727.94 hours) for Khmer audio and Khmer text (transcription)
- `train.360` split of `openslr/librispeech_asr` for English audio and English text (transcription)
- `KrorngAI/DDD-Cambodia-khmer-speech-dataset-parquet-000-159-en-translate` for Khmer audio and English text (translation)
- `KrorngAI/openslr-librispeech_asr-clean100-00000-19999-km-translate` for English audio and Khmer text (translation)
- `nospeech7k` for no speech audio (voice activity detection)
So, there are around 804 hours of audio in the training dataset.
TrorYongASR2 has 2 configurations as the following
| **Model Size** | Tiny | Small |
|:-----------------:|:-----------------:|:-------------------:|
| **Audio Encoder** | 4 layers, 6 heads | 12 layers, 12 heads |
| **Audio Context** | 1500 | 1500 |
| **Auditory-lingual Decoder** | 1 layer, 12 heads | 1 layer, 24 heads |
| **Text Context** | 448 | 448 |
| **Embedding Dimension** | 384 | 768 |
| **Parameters** | 29M | 136M |
The audio array is processed to log-mel spectrogram with 80 mel bins (the same as Whisper models of the same size).
These configurations are the same as the ones of TrorYongASR1.
The training hyperparameters are listed below:
- **Training regime:** 16-mixed precision training using `LightningAI` package
- **Number of epochs:** roughly 2 epochs
- **Optimizer:** MuonAdamW (custom implementation [here](https://github.com/Kimang18/adam_muon_optimizer))
- **Effective Batch size:** 64
- **Number of optimizer steps:** 10149
- **Learning rate:** Linear Warmup (153 optimizer steps) + CosineAnnealing (9996 optimizer steps)
- **Gradient Clip Value:** 0.5 (only for parameters trained by AdamW)
To underline the gain from architectural improvement, TrorYongASR1 is retrained on the same 5 datasets above using the same set of hyperparameters.
[Figure 2](#fig-loss) shows the training history of `tiny` configuration.
TrorYongASR2 has smaller training and validation losses, and higher validation token accuracy.
This shows the effectiveness of the new architectural design in TrorYongASR2.
Figure 2: Training history of TrorYongASR1 and TrorYongASR2 using the same training configuration.
TrorYongASR1 is represented by the blue color and TrorYongASR2 is represented by the red one.
The first row shows the training loss and the second one describes validation performance.
The top right plot shows the moving average of training loss.
All plots suggest that TrorYongASR2 has better performance compared to TrorYongASR1.
## How to Get Started with TrorYongASR2
First, install `tror-yong-asr` PyPI package:
```bash
pip install tror-yong-asr >= 0.2.0
```
Then, you can use TrorYongASR2 for `transcribe` or `translate` via the code below.
```python
from transformers import AutoProcessor
from tror_yong_asr import TrorYongASRModel, transcribe, translate, identify_language
model_id = "KrorngAI/TrorYongASR2-small"
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
model = TrorYongASRModel.from_pretrained(model_id, trust_remote_code=True)
result1 = identify_language('/path/to/audio_file.mp3', model, processor)
print(result1)
result2 = transcribe('/path/to/audio_file.mp3', model, processor, max_tokens=64)
print(result2)
result3 = translate('/path/to/audio_file.mp3', model, processor, max_tokens=64)
print(result3)
```
## Fine-tuning
You can follow this notebook below to learn to fine-tune TrorYongASR2.
[](https://colab.research.google.com/#fileId=https://huggingface.co/KrorngAI/TrorYongASR2-small/blob/main/fine_tune_TrorYongASR2.ipynb)
## Uses
### Direct Use
The model can be used directly for:
- **Speech-to-text transcription**: transcribe Khmer and English audio
- **Speech-to-text translation**: translate Khmer audio to English text and English audio to Khmer text
- **Language identification**: identify if the spoken language in the audio is Khmer or English
- **Edge computing**: Deploy on mobile devices, IoT devices, and embedded systems due to its small size (29M parameters for tiny or 136M for small)
- **Real-time applications**: Low latency inference suitable for real-time speech interfaces
### Downstream Use
The model can be integrated into:
- **Mobile applications**: Android/iOS apps with speech recognition
- **Web applications**: Browser-based speech-to-text using WebAssembly
- **IoT devices**: Smart speakers, voice assistants
- **Larger ASR systems**: As a component in multi-language ASR pipelines
## Bias, Risks, and Limitations
**Technical Limitations:**
- **No timestamp output**: The model does not support timestamp output
- **Noise robustness**: Performance may degrade in noisy environments
**Sociotechnical Limitations:**
- **Accent variability**: May not perform well on diverse Khmer accents
- **Background noise**: Limited robustness to background noise and reverberation
- **Speaker variability**: May struggle with different speaking styles and rates
## Citation
**BibTeX:**
```bibtex
@online{khun2026,
author = {Khun, Kimang},
title = {TrorYongASR2: {Permuted} {Language} {Modeling} with
{Monotonic} {Alignment,} {Gated} {Attention,} and
{Normalization-free} {Design}},
date = {2026-10-19},
url = {https://kimang18.github.io/krorngai-blog/TrorYongASR2/},
langid = {en}
}
```
## Model Card Author
- ឈ្មោះ: បណ្ឌិត ឃុន គីមអាង
- Name: KHUN Kimang (Ph.D.)
## Acknowledgement
Both models are trained thanks to the free credits from Kaggle, and `Google Colab`.
Thanks to [`KrorngAI`](https://huggingface.co/KrorngAI), [`openslr`](https://openslr.org), and [Mozilla Data Collective](https://mozilladatacollective.com/datasets/cml9h5vgc01bxmn075sjeftek) for their publicly available dataset.
## Model Card Contact
If you have any questions, please reach out at [Facebook Page](https://www.facebook.com/profile.php?id=61582509385293).
