Blame for README.md

---
language:
- vi
- en
license: cc-by-nc-sa-4.0
library_name: pytorch
pipeline_tag: automatic-speech-recognition
base_model: google/embeddinggemma-2
tags:
- ctc
- vietnamese
- code-switching
- streaming
- embeddinggemma
- asr
metrics:
- wer
---

# VieGem-ASR 🇻🇳💎

<p align="center"><img src="assets/cover.jpg" width="460" alt="VieGem-ASR"></p>

Fast Vietnamese and Vietnamese–English speech recognition: a 359M-parameter streaming CTC model on Google's
**EmbeddingGemma 2** audio encoder. Inspired by Quan Nguyen's **QUAN-Transcribe-1.0**.

- **Fast:** about **3,500× real time** on one H100 (batched, greedy); streams in 0.48 s chunks with no look-ahead.
- **Vietnamese:** mean WER **6.34** over 8 common test sets (**5.29** with the included n-gram LM).
- **Code-switching:** ViMedCSS WER **14.3**; **66%** of English terms written exactly.

## Demo

<video controls width="540" src="https://huggingface.co/5CD-AI/VieGem-ASR/resolve/main/demo/viegem_asr_promo.mp4"></video>

## Results

### Vietnamese test sets

WER %, lower is better, greedy unless marked "+ LM". VieGem-ASR, QUAN-Transcribe-1.0 and Qwen3-ASR-0.6B were run by
us with one normalizer; † = reported in the QUAN-Transcribe-1.0 model card (our scorer matches its numbers within 0.1).

| Test set | What it contains | VieGem-ASR | VieGem-ASR + LM | QUAN-Transcribe-1.0 | Qwen3-ASR-0.6B | Gemini-Flash-3.8 † | ElevenLabs Scribe v2 † | PhoWhisper-large † | Whisper large-v3 † |
|---|---|---|---|---|---|---|---|---|---|
| VIVOS | read speech, quiet room | 6.63 | _4.78_ | 7.52 | 10.22 | 9.36 | 9.14 | **4.52** | 9.18 |
| FLEURS vi | read Wikipedia sentences | 7.33 | _5.85_ | 7.88 | 8.40 | 5.90 | 5.60 | **5.31** | 7.09 |
| Common Voice 17 | read, crowd-sourced microphones | 8.61 | **6.21** | 9.69 | 16.07 | 10.30 | 10.50 | 9.12 | 18.43 |
| PhoAudiobook | audiobooks | 1.39 | 1.04 | 1.39 | 3.23 | 1.60 | 1.45 | **0.87** | 1.85 |
| LSVSC | large mixed-domain corpus | 5.16 | **4.71** | 5.35 | 7.78 | 5.70 | 6.25 | 6.81 | 13.96 |
| Bud500 | mixed domains, tightly cut clips | 3.08 | 2.75 | 1.82 | 6.79 | **1.80** | 3.15 | 9.51 | 64.03 |
| GigaSpeech2 vi | YouTube, spontaneous | 9.50 | **8.52** | 11.60 | 11.37 | 10.70 | 10.50 | 10.14 | 14.19 |
| ViMD | dialects of 63 provinces | 9.05 | **8.46** | 9.84 | 13.98 | 10.00 | 10.05 | 11.38 | 15.56 |
| **Mean of the 8** | | 6.34 | **5.29** | 6.89 | 9.73 | 6.92 | 7.08 | 7.21 | 18.04 |

Bold = best greedy result per row. Text is lowercased, without punctuation, numbers as words.

### Vietnamese–English code-switching (ViMedCSS)

| System | Test WER | Hard WER |
|---|---|---|
| Whisper-large-v3 (paper) | 34.47 | 39.35 |
| PhoWhisper-large (paper) | 31.24 | 37.37 |
| PhoWhisper-small + AG, fine-tuned (paper) | 23.67 | 33.73 |
| QUAN-Transcribe-1.0 | 22.26 | 26.56 |
| **VieGem-ASR** | **14.30** | **21.08** |
| **VieGem-ASR + LM** | **13.00** | **19.84** |

"Paper" = numbers from the ViMedCSS paper (its own normalization, so approximate).

### Streaming and speed

- Streaming (0.48 s chunks): VIVOS 8.73, CV17 10.84, FLEURS 9.02, LSVSC 5.84, ViMD 10.00, ViMedCSS 15.99.
- A 10-minute video is transcribed in **0.49 s** on one H100 (median of 20 runs, ~1,220× real time).

## Usage

```python
import torch, soundfile as sf
from transformers import AutoTokenizer
from asr.model import CTCModel, ModelConfig, greedy_decode     # asr/ folder of this repo
from asr.text import Units

ck = torch.load("viegem_asr.pt", map_location="cpu")
cfg = ModelConfig.from_dict(ck["model_config"])
units = Units(ck["gemma_ids"])
tok = AutoTokenizer.from_pretrained(cfg.encoder_id)               # google/embeddinggemma-2 (accept its terms)
model = CTCModel(cfg, units.gemma_ids)
model.load_state_dict(ck["model"], strict=False)
model = model.cuda().eval()

wav, sr = sf.read("audio.wav", dtype="float32")                  # 16 kHz mono, ≤ 30 s
x = torch.from_numpy(wav)[None].cuda()
with torch.no_grad(), torch.autocast("cuda", dtype=torch.bfloat16):
    logits, lens = model(x, torch.tensor([x.shape[1]], device="cuda"), streaming=False)  # streaming=True: 0.48 s chunks
print(units.decode(tok, greedy_decode(logits, lens)[0]))
```

With the LM (`pip install kenlm`):

```python
from asr.lm_decode import LMDecoder

words = open("lm/viegem_lm.unigrams.txt", encoding="utf-8").read().split()
dec = LMDecoder("lm/viegem_lm.bin", unigrams=words, alpha=0.35, beta=1.5)
pieces = ["", ""] + tok.convert_ids_to_tokens(units.gemma_ids)
lp = torch.log_softmax(logits[0, : lens[0]].float(), -1).cpu().numpy()
print(units.decode(tok, dec.decode(lp, pieces, beam_width=32)))
```

## Model

- **Encoder:** EmbeddingGemma 2 audio tower (12 conformer layers, 25 frames/s); top 4 layers fine-tuned.
- **Reader:** layer mix + 6 transformer blocks (d=768) with chunked attention, so one model serves offline and streaming.
- **Head:** 21,031 Gemma tokens (Vietnamese, English, single characters), initialized from Gemma's text embeddings.
- **Files:** `viegem_asr.pt` (fp16 weights, 311 MB), `asr/` (code), `lm/` (4-gram KenLM from our training transcripts, 71 MB), `demo/`.

## Training data

About **5,800 hours**, every set filtered against all test and dev sentences.

| Type | Sources | Hours |
|---|---|---|
| Audiobooks | PhoAudiobook, InfoRe2 | ~1,900 |
| Social media and YouTube | VietSpeech, viVoice, GigaSpeech2 (subset), VietSuperSpeech | ~2,820 |
| Mixed domains | Bud500, VLSP2020, LSVSC, ViMD, FPT FOSD, InfoRe1, FLEURS, Common Voice 17 | ~780 |
| English | LibriSpeech-100 | ~100 |
| Vietnamese–English (medical) | ViMedCSS | ~23 |

## Limitations

- Output is lowercase, without punctuation; numbers are written as words.
- Clips cut in the middle of a word (as in Bud500) lose the last word more often.
- English is meant for terms mixed into Vietnamese; plain English is weak (LibriSpeech test-clean 10.7).

## License

Non-commercial research use (**CC-BY-NC-SA-4.0**), because the training data includes Bud500, viVoice and PhoAudiobook.
Use must also follow the **Gemma Terms of Use** and Prohibited Use Policy.

## Credits

Google DeepMind (EmbeddingGemma 2); Quan Nguyen (QUAN-Transcribe-1.0, the inspiration for this work); the authors of
VIVOS, Common Voice, FLEURS, LSVSC, ViMD, Bud500, PhoAudiobook, VietSpeech, viVoice, GigaSpeech2, VietSuperSpeech,
VLSP2020, FPT FOSD, InfoRe, ViMedCSS and LibriSpeech. ViMedCSS baselines: Nguyen et al., LREC 2026.