Malagasy 4-gram language model (KenLM) for CTC speech recognition
A word-level 4-gram KenLM model of Malagasy, built to decode the CTC output of Flo976/w2v-bert-malagasy. It brings that model from 14.33 % to 13.63 % word error rate on short utterances, and from 12.01 % to 10.60 % on full-length clips.
Files
| File | Content |
|---|---|
mg_4gram.bin |
KenLM binary (trie, 8-bit quantized), 89 MB |
unigrams.txt |
478,401 word forms seen at least twice, one per line, for pyctcdecode |
Training text
17.4 million words, normalized exactly as the ASR output: lowercase, punctuation removed, apostrophes and hyphens kept inside words, numbers as digits.
| Source | Words | License |
|---|---|---|
| Malagasy Wikipedia, via ffatty/plaintext-wikipedia-full-malagasy | 16.5 M | CC-BY-SA-3.0 |
| Transcripts of the train split of badrex/malagasy-speech-full | 0.9 M | not declared on the Hub |
No validation or test transcript was used, so the scores above are not inflated by the language model. Built with lmplz -o 4 --prune 0 0 1 1, then build_binary -q 8 -b 7 -a 256 trie.
Usage
pip install pyctcdecode https://github.com/kpu/kenlm/archive/master.zip
from huggingface_hub import hf_hub_download
from pyctcdecode import build_ctcdecoder
from transformers import AutoProcessor
processor = AutoProcessor.from_pretrained("Flo976/w2v-bert-malagasy")
tok = processor.tokenizer
vocab = tok.get_vocab()
labels = ["" if t == tok.pad_token else " " if t == tok.word_delimiter_token else "⁇" if t == tok.unk_token else t
for t in sorted(vocab, key=vocab.get)]
repo = "Flo976/malagasy-kenlm-4gram"
unigrams = [w.strip() for w in open(hf_hub_download(repo, "unigrams.txt"), encoding="utf-8") if w.strip()]
decoder = build_ctcdecoder(labels, kenlm_model_path=hf_hub_download(repo, "mg_4gram.bin"),
unigrams=unigrams, alpha=0.3, beta=-1.0)
# text = decoder.decode(logits[0].numpy(), beam_width=100)
Tuned on the validation split: α = 0.3, β = −1.0 for utterances of 10 s or less, β = 0.0 for longer clips. A beam of 100 is used: in an earlier tuning on full-length clips, beams of 300 and 500 brought no gain over it. After decoding, re-attaching apostrophes (amin' ny → amin'ny) gains another 0.24 point on short utterances.
pyctcdecode 0.5.0 declares numpy<2 but works with numpy 2; if that pin conflicts with your environment, install it with pip install --no-deps pyctcdecode pygtrie.
License
CC-BY-SA-3.0, inherited from Malagasy Wikipedia: share-alike applies to derivatives of this language model. The acoustic model Flo976/w2v-bert-malagasy is licensed separately under CC-BY-4.0.
Built by Flo976 for Milo Voice.