Automatic Speech Recognition
Transformers
ONNX
Safetensors
Khmer
English
wav2vec2
speech
khmer
code-switching
mms
ctc
Eval Results (legacy)
File size: 12,314 Bytes
1a24b5a
 
 
 
 
8948417
 
1a24b5a
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
---
license: cc-by-nc-4.0
language:
- km
- en
base_model: nphearum/PsarAI-ASR-1B
base_model_relation: quantized
datasets:
- Digital-Divide-Data/khmer-speech-dataset
- nphearum/dataset-asr-tts
- nphearum/khmer-raw-text-3M
pipeline_tag: automatic-speech-recognition
library_name: transformers
tags:
- automatic-speech-recognition
- speech
- khmer
- code-switching
- wav2vec2
- mms
- ctc
- onnx
metrics:
- cer
- wer
model-index:
- name: PsarAI-ASR-1B
  results:
  - task:
      type: automatic-speech-recognition
      name: Automatic Speech Recognition
    dataset:
      type: seanghay/khmer_grkpp_speech
      name: Khmer news speech (grkpp, unseen speakers, 1-30 s clips)
    metrics:
    - type: cer
      value: 13.73
      name: CER (greedy)
  - task:
      type: automatic-speech-recognition
      name: Automatic Speech Recognition
    dataset:
      type: nphearum/dataset-asr-tts
      name: English (held-out synthetic speech)
    metrics:
    - type: wer
      value: 1.39
      name: WER (greedy)
---

# PsarAI-ASR-1B (PyTorch + ONNX)

Speech recognition for **Khmer (ខ្មែរ)**, **English**, and **mixed Khmer–English** speech
(wav2vec2, ~965M parameters, CTC).

This repository contains the original PyTorch weights of
[nphearum/PsarAI-ASR-1B](https://huggingface.co/nphearum/PsarAI-ASR-1B) plus **ONNX exports**
(fp32, fp16, int8) that run with `onnxruntime` only, without PyTorch.

## Files

| Path | Format | Size | Use it for |
|---|---|---:|---|
| `model.safetensors` (+ configs) | PyTorch, fp16 weights | 1.9 GB | 🤗 Transformers, GPU or CPU |
| `onnx/fp32/` | ONNX, fp32 | 3.9 GB | reference accuracy, CPU/GPU |
| `onnx/fp16/` | ONNX, fp16 (fp32 inputs/outputs) | 1.9 GB | GPU (CUDA / DirectML) |
| `onnx/int8/` | ONNX, int8 dynamic quantization | 1.0 GB | **CPU** (smallest, recommended for CPU) |
| `onnx/infer_onnx.py` | script | | command-line transcription with ONNX |
| `onnx/export_onnx.py` | script | | how the ONNX files were made |
| `results/` | benchmark | | CER / speed comparison of all variants |

Each `onnx/<variant>/` folder is self-contained (`model.onnx` + `model.onnx.data` + vocab/tokenizer files).

## Install

```bash
# PyTorch / Transformers
pip install torch transformers librosa soundfile

# ONNX only (no torch needed)
pip install onnxruntime librosa huggingface_hub     # CPU
pip install onnxruntime-gpu librosa huggingface_hub # NVIDIA GPU
```

## Quick test — 🤗 Transformers

```python
import librosa
import torch
from transformers import AutoProcessor, Wav2Vec2ForCTC

REPO = "phonsobon/PsarAI-ASR-1B"
device = "cuda" if torch.cuda.is_available() else "cpu"

processor = AutoProcessor.from_pretrained(REPO)
model = Wav2Vec2ForCTC.from_pretrained(REPO, dtype=torch.float32).to(device).eval()

audio, _ = librosa.load("speech.wav", sr=16_000, mono=True)  # any format librosa reads; 16 kHz mono
inputs = processor(audio, sampling_rate=16_000, return_tensors="pt").to(device)
with torch.inference_mode():
    logits = model(**inputs).logits
print(processor.decode(logits.argmax(-1)[0]))
```

Weights are stored in float16; load them as float32 as above, especially on CPU.
Clips of up to ~30 s fit comfortably on a 24 GB GPU.

### Long audio (pipeline with chunking)

```python
from transformers import pipeline

asr = pipeline(
    "automatic-speech-recognition",
    model="phonsobon/PsarAI-ASR-1B",
    device=0,             # -1 for CPU
    chunk_length_s=20,    # split long recordings into 20 s windows
    stride_length_s=4,    # overlap between windows
)
print(asr("long_recording.wav")["text"])
```

## Quick test — ONNX Runtime (no PyTorch)

### Command line

```bash
# Download only the int8 model + script (~1 GB instead of ~9 GB)
hf download phonsobon/PsarAI-ASR-1B --include "onnx/int8/*" "onnx/infer_onnx.py" --local-dir PsarAI-ASR-1B

python PsarAI-ASR-1B/onnx/infer_onnx.py speech.wav                                # int8 on CPU
python PsarAI-ASR-1B/onnx/infer_onnx.py speech.wav --variant fp16 --device cuda   # needs onnx/fp16/*
python PsarAI-ASR-1B/onnx/infer_onnx.py speech.wav --variant fp16 --device dml    # Windows DirectML
```

(With older `huggingface_hub`, use `huggingface-cli download` instead of `hf download`.)

### Python

```python
import json
from pathlib import Path

import librosa
import numpy as np
import onnxruntime as ort
from huggingface_hub import snapshot_download

VARIANT = "int8"  # "int8" for CPU, "fp16" for GPU, "fp32" for reference
local = snapshot_download("phonsobon/PsarAI-ASR-1B", allow_patterns=[f"onnx/{VARIANT}/*"])
model_dir = Path(local) / "onnx" / VARIANT

vocab = json.loads((model_dir / "vocab.json").read_text(encoding="utf-8"))
id_to_token = {i: t for t, i in vocab.items()}
skip_ids = {vocab[t] for t in ("<pad>", "<s>", "</s>", "<unk>") if t in vocab}

session = ort.InferenceSession(
    str(model_dir / "model.onnx"),
    providers=["CPUExecutionProvider"],  # ["CUDAExecutionProvider", "CPUExecutionProvider"] for GPU
)

def transcribe(path: str) -> str:
    audio, _ = librosa.load(path, sr=16_000, mono=True)
    x = ((audio - audio.mean()) / np.sqrt(audio.var() + 1e-7)).astype(np.float32)[None, :]
    mask = np.ones_like(x, dtype=np.int64)
    (logits,) = session.run(["logits"], {"input_values": x, "attention_mask": mask})

    # Greedy CTC decoding: merge repeats, drop blanks/special tokens, "|" = space.
    out, prev = [], None
    for i in logits[0].argmax(-1).tolist():
        if i != prev and i not in skip_ids:
            out.append(id_to_token[i])
        prev = i
    return " ".join("".join(out).replace("|", " ").split())

print(transcribe("speech.wav"))
```

## Test accuracy on a public Khmer test set

Computes CER on a few clips of [seanghay/khmer_grkpp_speech](https://huggingface.co/datasets/seanghay/khmer_grkpp_speech)
(real Khmer news speech, never used in training).

```bash
pip install "datasets[audio]" jiwer
```

```python
import re

import jiwer
import torch
from datasets import Audio, load_dataset
from transformers import AutoProcessor, Wav2Vec2ForCTC

REPO = "phonsobon/PsarAI-ASR-1B"
N_CLIPS = 20
device = "cuda" if torch.cuda.is_available() else "cpu"

processor = AutoProcessor.from_pretrained(REPO)
model = Wav2Vec2ForCTC.from_pretrained(REPO, dtype=torch.float32).to(device).eval()

ds = load_dataset("seanghay/khmer_grkpp_speech", split="train", streaming=True)
ds = ds.cast_column("audio", Audio(sampling_rate=16_000))

def norm(text: str) -> str:
    # letters only: drop spaces, punctuation and zero-width characters
    return re.sub(r"[\s​‌‍។៖!?.,\"'()\-]", "", text.lower())

refs, hyps = [], []
for sample in ds:
    audio = sample["audio"]["array"]
    if not 1 <= len(audio) / 16_000 <= 30:  # same 1–30 s range as the reported results
        continue
    inputs = processor(audio, sampling_rate=16_000, return_tensors="pt").to(device)
    with torch.inference_mode():
        pred = processor.decode(model(**inputs).logits.argmax(-1)[0])
    refs.append(norm(sample["transcription"]))
    hyps.append(norm(pred))
    print(f"REF: {sample['transcription']}\nHYP: {pred}\n")
    if len(refs) == N_CLIPS:
        break

print(f"CER on {len(refs)} clips: {jiwer.cer(refs, hyps):.2%}")
```

This simple normalization does not convert digits to words, so expect a CER somewhat above
the 13.73% reported below (which uses the full normalization described in *Results*).

## Results

Character error rate (CER, letters only: spaces and punctuation ignored) and word error rate (WER),
lower is better. Before scoring, references and transcripts are normalized the same way: canonical
Khmer character order, one spelling per word (ឲ្យ/អោយ → ឱ្យ), numbers as spoken words (២០១៩ → ពីរពាន់ដប់ប្រាំបួន).

| Test set | Clips | What it is | Greedy | With Khmer LM decoder* |
|---|---:|---|---:|---:|
| grkpp (news) | 348 | [seanghay/khmer_grkpp_speech](https://huggingface.co/datasets/seanghay/khmer_grkpp_speech), new speakers and topics | 13.73% CER | 9.99% CER |
| grkpp, no digits | 125 | same, clips whose transcript has no digits | 11.68% CER | 8.95% CER |
| khmer10 | 10 | DDD speaker never used in training | 4.84% CER | 3.01% CER |
| DDD held-out | 150 | Digital-Divide-Data clips not used in training | 5.87% CER | 2.12% CER |
| Jarvis Khmer | 100 | held-out synthetic voice, Khmer | 1.27% CER | 0.17% CER |
| Jarvis English | 100 | held-out synthetic voice, English | 1.39% WER | 0.49% WER |
| Jarvis mixed | 100 | held-out synthetic voice, Khmer–English | 12.43% WER | 2.99% WER |

\* Greedy is what this model outputs on its own. The second column adds a separate decoder: CTC beam
search with a Khmer character 6-gram and a word trigram, which is not included in this repository.

### PyTorch vs ONNX variants

Benchmark on 136 clips (42 min: English, Khmer, and mixed Khmer–English voice-over).
RTF = processing time / audio duration (lower is faster). Full table: [`results/summary.md`](results/summary.md).

| Variant | Device | CER | RTF | Size |
|---|---|---:|---:|---:|
| PyTorch | CUDA | 6.28% | 0.009 | 1.93 GB |
| ONNX fp32 | CPU | 6.28% | 0.370 | 3.86 GB |
| ONNX fp16 | CPU | 6.28% | 0.732 | 1.93 GB |
| ONNX int8 | CPU | 6.34% | 0.545 | 1.02 GB |

- All ONNX variants match PyTorch accuracy (int8 loses < 0.1 pt CER).
- On **CPU**, use **int8** (smallest) or fp32 (fastest on CPU in this test). fp16 is meant for GPU; on CPU it is slow.
- On **GPU**, use PyTorch or ONNX fp16 with `CUDAExecutionProvider`.

![Benchmark dashboard](results/charts/dashboard.png)

## Training

| | |
|---|---|
| Base | facebook/mms-1b-all, Khmer adapter (`khm`) |
| Trained | adapter layers + output head (~2.3M of ~965M parameters); base model frozen |
| Output vocabulary | the Khmer adapter's 152 characters + `\|` word separator (153) |
| Data | ~15 h of [Digital-Divide-Data/khmer-speech-dataset](https://huggingface.co/datasets/Digital-Divide-Data/khmer-speech-dataset) (real read speech, many speakers); [nphearum/dataset-asr-tts](https://huggingface.co/datasets/nphearum/dataset-asr-tts) synthetic "Jarvis" voice: 3,000 Khmer, 2,000 English and 800 mixed clips |
| Steps | 4,000; AdamW, learning rate 1e-3 with warm-up and linear decay; bf16 autocast |
| Hardware | 1 × RTX 3090 |

See the original model card at [nphearum/PsarAI-ASR-1B](https://huggingface.co/nphearum/PsarAI-ASR-1B) for full details.

ONNX export: opset 17, dynamic batch and length axes, weights in an external `model.onnx.data` file
(the fp32 graph is above protobuf's 2 GB limit). int8 uses dynamic quantization of the transformer
MatMuls only; the convolutional feature encoder stays fp32. See [`onnx/export_onnx.py`](onnx/export_onnx.py).

## Limitations

- **English**: trained and tested only on one synthetic voice; expect clearly weaker results on real English speakers and accents. Output is lowercase.
- **Numbers**: the vocabulary has the Khmer digits ០ ១ ២ ៣ ៥ ៧ but not ៤ ៦ ៨ ៩, so the model sometimes writes numbers in mixed scripts (e.g. `២០១9`).
- **Spaces**: Khmer is written without spaces between words; the model places spaces where its training text had them (between phrases), not between every word.
- **Missing characters**: a few rare Khmer characters are not in the vocabulary (e.g. ៛ ៕ ឰ ឝ ឞ).
- **Domain words and names** it has not heard are often misspelled; an LM decoder with domain text or hotwords helps.

## License and credits

Released under **CC-BY-NC-4.0** (non-commercial), following the base model
[facebook/mms-1b-all](https://huggingface.co/facebook/mms-1b-all).

- Model: [nphearum/PsarAI-ASR-1B](https://huggingface.co/nphearum/PsarAI-ASR-1B)
- Training data: [Digital-Divide-Data/khmer-speech-dataset](https://huggingface.co/datasets/Digital-Divide-Data/khmer-speech-dataset) (CC-BY-SA-4.0),
  [nphearum/dataset-asr-tts](https://huggingface.co/datasets/nphearum/dataset-asr-tts) (CC-BY-NC-4.0; its synthetic voice imitates a
  character voice whose rights belong to their owners).

Check that your use is permitted before using this model.

```bibtex
@article{pratap2023mms,
  title   = {Scaling Speech Technology to 1,000+ Languages},
  author  = {Pratap, Vineel and Tjandra, Andros and Shi, Bowen and others},
  journal = {arXiv preprint arXiv:2305.13516},
  year    = {2023}
}
```