Request access to moore-asr-eval-w2vbert
This repository is publicly accessible, but you have to accept the conditions to access its files and content.
moore-asr-eval-w2vbert is a Mooré speech recognition model used to score text-to-speech systems, shared for non-commercial research (CC BY-NC 4.0). Commercial use needs written permission from nyantudrealban@gmail.com. Please read the card, its terms and licenses before requesting access, and make sure you understand them.
Log in or Sign Up to review the conditions and access this model content.
Mooré eval ASR (w2v-BERT 2.0 + CTC)
A character-level Mooré speech recognizer. We use it to score how intelligible Mooré text-to-speech systems are, and it never saw the test data.
- 0.46 % CER on the held-out dev chapters, 0.60 % on the test chapters and 0.73 % on two voices it never heard, with greedy decoding and no language model
- Trained only on the training chapters of the Mooré TTS benchmark. No dev or test chapter and no held-out voice was used, in either Bible translation.
- Decoding is plain CTC. A language model trained on the Bible would fix some TTS mistakes before they are counted, so we left it out.
Résumé en français. Reconnaissance de la parole en mooré au niveau des caractères (w2v-BERT 2.0 + CTC). Nous l'utilisons pour mesurer l'intelligibilité des systèmes de synthèse vocale en mooré, et il n'a jamais vu les données de test. CER de 0,46 % sur les chapitres de dev réservés, 0,60 % sur les chapitres de test et 0,73 % sur deux voix jamais entendues, en décodage glouton sans modèle de langue. Il est entraîné uniquement sur les chapitres d'entraînement du benchmark de synthèse vocale mooré : aucun chapitre de dev ou de test ni aucune voix réservée, dans aucune des deux traductions de la Bible. Le décodage est un CTC simple : un modèle de langue entraîné sur la Bible corrigerait certaines erreurs de synthèse avant qu'elles soient comptées, nous l'avons donc écarté. Licence CC BY-NC 4.0 ; pour un usage commercial, écrivez à nyantudrealban@gmail.com. Accès sur demande.
Usage
import torch, soundfile as sf, soxr
from transformers import Wav2Vec2BertForCTC, Wav2Vec2BertProcessor
repo = "anyantudre/moore-asr-eval-w2vbert"
proc = Wav2Vec2BertProcessor.from_pretrained(repo)
model = Wav2Vec2BertForCTC.from_pretrained(repo).eval()
wav, sr = sf.read("sample.wav", dtype="float32")
wav = soxr.resample(wav, sr, 16000)
feats = proc(wav, sampling_rate=16000, return_tensors="pt")
with torch.no_grad():
ids = model(**feats).logits.argmax(-1)
print(proc.batch_decode(ids)[0])
Compute CER on text normalized the same way on both sides: NFC, lowercase, hyphens and apostrophes turned into spaces, punctuation removed (mtts.text.normalize_cer in the project code).
Training
| Base model | facebook/w2v-bert-2.0 |
| Parameters | ~600M |
| Input | 16 kHz mono |
| Output | 32 characters: Mooré letters (ɛ ɩ ʋ, nasal vowels ã ẽ ĩ õ ũ) plus word delimiter |
| Data | ~130 h of read Mooré Bible speech: the train split of the jw.org (24 kHz) and Bible-society (16 kHz) recordings |
- 8 epochs, AdamW, lr 5e-5 with 500 warmup steps and linear decay, effective batch 32, bf16, light time masking (p=0.05), adapter on
- Best checkpoint chosen on dev CER (step 15,000 of 17,168)
- One RTX 5090, about 4.5 hours
Results
Scored on the real recordings of the frozen benchmark sets (jw.org translation), one utterance at a time.
| Set | Utterances | Chapters | CER | WER | CER on ɛ ɩ ʋ and nasal vowels |
|---|---|---|---|---|---|
| dev | 402 | 21 | 0.46 % | 1.77 % | 1.27 % |
| test | 541 | 23 | 0.60 % | 1.93 % | 1.62 % |
| test, unseen voices V17 and V18 | 836 | 40 | 0.73 % | 2.57 % | 2.43 % |
These numbers are the floor for any TTS system scored with this model: a perfect synthesis of these sentences would still get about this CER.
The dev CER logged during training was higher (1.30 % at the released step) because it was computed on padded batches, where the padding frames add spurious characters.
Limitations
- It was trained on read Bible speech only and will do much worse on conversational Mooré. It is meant for comparing TTS systems on in-domain text.
- Mooré orthography does not mark tone, so this model cannot tell whether a TTS system got the tones right.
- The output vocabulary has no ɛ̃, ɩ̃ or ʋ̃ (letters written with a combining tilde), because they never occur in the training text.
License
These weights are released under CC BY-NC 4.0; the base model (facebook/w2v-bert-2.0) is MIT. The training audio comes from Bible recordings whose terms allow non-commercial use only and restrict redistribution, so the dataset stays private. For commercial use, contact nyantudrealban@gmail.com.
- Downloads last month
- 10
Model tree for anyantudre/moore-asr-eval-w2vbert
Base model
facebook/w2v-bert-2.0