Modularcomputing/dynamic-asr

Dynamic ASR

Speech recognition for Nigerian English, Nigerian Pidgin, Igbo, Yoruba and Hausa — and for speech that switches between them inside a sentence. By Modular, part of the Dynamic Accent project (see also the Dynamic Accent TTS and the Dynamic Accent Duplex voice agent, which uses this model as its ASR).

Dynamic ASR is Meta's Omnilingual ASR omniASR_CTC_1B_v2 (1B-parameter wav2vec2 CTC, Apache-2.0) fully fine-tuned on 187.2 hours of licensed Nigerian speech. The architecture, tokenizer and decoding are unchanged, so it is a drop-in replacement: language-agnostic (no language hint), greedy CTC, cheap enough to re-decode a growing buffer for streaming partials.

Status: experimental research release. Read the limitations.

Two models

folder	what it is	use it when
real/ (default)	fine-tuned on licensed real speech only	general use; as good as the experimental model on real code-switched speech and better on the selection dev set (mean CER 0.071 vs 0.074)
codeswitch-experimental/	same recipe, plus ~4.99 h of synthetic code-switched speech made with our own Dynamic Accent Flash-DPO TTS (12 % of training audio)	research on synthetic code-switch data; see the ablation below before choosing it

Both are the same size (1B, ~1.95 GB bf16 state dict) and load the same way.

Results

All test data is held out (never trained on). Scores are WER / CER after lower-casing and stripping punctuation; "no-diacritic" CER folds tone marks and dots. Decoding: greedy CTC with a Latin-script logit mask (the duplex setting).

Per language (duplex evaluation set: 229 clips — WAXAL test ibo/yor/hau/pcm × 40, FLEURS validation English × 29, 40 clips made by joining two held-out clips of different languages)

CER

model	English	Pidgin	Igbo	Yoruba	Hausa	code-switch (concatenated)
omniASR_CTC_1B (stock; previous duplex ASR)	0.065	0.169	0.206	0.191	0.090	0.128
omniASR_CTC_1B_v2 (stock base)	0.024	0.123	0.133	0.135	0.106	0.104
Dynamic ASR real (real/)	0.025	0.074	0.093	0.121	0.082	0.081
Dynamic ASR code-switch, experimental (codeswitch-experimental/)	0.024	0.076	0.095	0.118	0.081	0.072

WER

model	English	Pidgin	Igbo	Yoruba	Hausa	code-switch (concatenated)
omniASR_CTC_1B (stock; previous duplex ASR)	0.144	0.380	0.636	0.498	0.299	0.330
omniASR_CTC_1B_v2 (stock base)	0.097	0.336	0.456	0.401	0.317	0.292
Dynamic ASR real (real/)	0.095	0.193	0.315	0.355	0.253	0.238
Dynamic ASR code-switch, experimental (codeswitch-experimental/)	0.092	0.193	0.321	0.353	0.252	0.216

Real code-switched speech

Licensed corpora contain little real code-switched speech, so we built the most honest test we could from held-out data: Igbo/Yoruba/Hausa utterances whose transcripts contain English words. Natural conversational = spontaneous Igbo speech with English (IgboSynCorp; those speakers were removed from training). Insertional = read Igbo/Yoruba/Hausa sentences with English names/terms (WAXAL test, NaijaS2ST dev). CER (no-diacritic CER in brackets), plus English-word recall (share of the reference's English words that appear in the output; strict exact-match, so low in absolute terms). The last column is a held-out set of synthetic code-switched clips (unseen texts) from the TTS.

model	real CS, all (n=94)	natural conversational Igbo-English (n=36)	insertional, read (n=58)	English-word recall	synthetic CS test (n=109)
omniASR_CTC_1B (stock; previous duplex ASR)	0.224 (0.195)	0.372 (0.326)	0.178 (0.154)	0.3009	0.159 (0.129)
omniASR_CTC_1B_v2 (stock base)	0.193 (0.172)	0.353 (0.315)	0.143 (0.128)	0.2832	0.132 (0.118)
Dynamic ASR real (real/)	0.148 (0.137)	0.248 (0.234)	0.117 (0.107)	0.3186	0.105 (0.091)
Dynamic ASR code-switch, experimental (codeswitch-experimental/)	0.149 (0.137)	0.247 (0.234)	0.119 (0.106)	0.3805	0.038 (0.035)

Did synthetic code-switched audio help?

Short answer: not on real code-switched speech — at least not measurably at this scale. The two models share everything (data, seed, 4,000 steps) except that codeswitch-experimental/ sampled 12 % of its audio from 2572 synthetic code-switched clips (4.99 h).

• Real code-switched speech (94 clips): a tie. CER 0.148 (real) vs 0.149 (codeswitch-experimental); natural conversational Igbo–English 0.248 vs 0.247; insertional 0.117 vs 0.119; WER 0.436 vs 0.438. All differences are well inside the noise of a 94-clip test.
• English-word recall did go up, 0.319 → 0.381 (share of the reference's English words the model writes out exactly). This is the one real-speech signal in favour of synthetic data: the model spells embedded English words as English more often. It did not translate into lower CER, and the test is small, so treat it as a lead, not a result.
• Per-language real tests: no change. Differences of at most 0.003 CER on English, Pidgin, Igbo, Yoruba and Hausa (B slightly better on Yoruba/Hausa/English, slightly worse on Pidgin/Igbo). On the selection dev set (WAXAL validation + OpenSLR-70) the experimental model is slightly worse (mean CER 0.074 vs 0.071).
• Concatenated code-switch clips (real audio of two languages joined): small gain, CER 0.081 → 0.072, WER 0.238 → 0.216. These are inter-sentential switches at a hard boundary, closer to what the synthetic data looks like than to natural speech.
• Synthetic code-switched test (109 held-out clips, unseen texts): large gain, CER 0.105 → 0.038 (stock v2 base: 0.132). This mostly measures adaptation to our own TTS voices and recording style, so it is not evidence about real speakers.

Verdict: synthetic code-switched audio from our TTS teaches the model the synthetic domain and slightly improves English-word spelling, but it does not lower error on real code-switched or monolingual speech. real/ stays the default. A real natural code-switching test set (and more real code-switched training speech) is what would settle this.

AfriSwitch (in-the-wild code-switched speech) — pending

We planned to score both models (and the stock v2 base) zero-shot on the Nigerian subsets of Intron Health's AfriSwitch benchmark (Igbo, Yoruba, Hausa, Nigerian Pidgin; evaluation only, never trained on). The dataset is gated with manual approval; we requested access on 2026-10-24 and it had not been granted when the models were finalised, so no AfriSwitch scores are reported yet. When we run it we will use our scorer's normalisation (lowercase, punctuation stripped; the paper scores diacritic-preserving WER, and our models drop Igbo tone marks, so Igbo WER will be inflated against its references and we will report the diacritic-folded figure alongside), corpus-level WER and CER.

For reference, the paper's zero-shot WER (%) on these subsets (Table 2; systems given the matrix language, diacritics preserved):

language	Sahara V2.5 (best overall, avg 35.93 over 12 languages)	Omnilingual LLM 7B	Gemini 3.6	ElevenLabs	Sahara V2
Igbo	46.94	75.52	75.88	65.37	83.58
Yoruba	45.54	70.27	77.26	74.62	89.66
Hausa	31.77	39.27	45.44	41.01	45.56
Nigerian Pidgin	32.83	42.71	38.84	35.33	42.19

AfriSwitch: Ashungafac, Awobade & Olatunji (Intron Health), AfriSwitch: A Benchmark for In-the-Wild African Code-Switched Speech Recognition, arXiv:2608.26434 (2026). Dataset: https://huggingface.co/datasets/intronhealth/AfriSwitch (dataset card licence: CC-BY-NC-SA-4.0; paper CC BY 4.0).

@misc{ashungafac2026afriswitch,
  title={AfriSwitch: A Benchmark for In-the-Wild African Code-Switched Speech Recognition},
  author={Ashungafac, Gabrial Zencha and Awobade, Busayo and Olatunji, Tobi},
  year={2026}, eprint={2608.26434}, archivePrefix={arXiv}, primaryClass={cs.CL}
}

Usage

pip install "omnilingual-asr>=0.2.0" huggingface_hub soundfile scipy
python transcribe.py audio.wav                      # real/ model (default)
python transcribe.py audio.wav --model codeswitch   # codeswitch-experimental/

from transcribe import DynamicASR
asr = DynamicASR("real")          # downloads real/model.pt; base architecture omniASR_CTC_1B_v2
print(asr(waveform, sr))          # mono float32 array, any sample rate

The checkpoint is a plain PyTorch state dict ({"model": state_dict, ...}) for omniASR_CTC_1B_v2: transcribe.py loads the base through fairseq2.models.hub.load_model("omniASR_CTC_1B_v2") and overwrites its weights (the base download is needed once for the model skeleton and tokenizer). On omnilingual-asr 0.1.0, run code/training/patch_v2.sh (adjust the venv path) to register the v2 card first.

Streaming (duplex agent). code/asr_server.py is the FastAPI worker used by Dynamic Accent Duplex: POST /asr with raw float32 16 kHz mono → {"text", "ms"}; partial hypotheses come from re-decoding the growing buffer every 0.64 s (~30–90 ms per call on an H100).

cd code && ASR_MODEL=dynamic-asr uvicorn asr_server:app --port 8002      # real/ (default)
#          ASR_MODEL=dynamic-asr-cs  -> codeswitch-experimental/
#          ASR_MODEL=omni            -> stock omniASR_CTC_1B (also the automatic fallback if Dynamic ASR cannot load)

Output format: lower-case, no punctuation, digits as spoken or written in the training transcripts. Igbo is written without tone marks (dotted letters ị ọ ụ ṅ kept, standard everyday orthography); Yoruba with tone marks; Hausa as in the sources (hooked letters often missing, as in WAXAL).

Training data (licensed only)

source	language	hours used	licence
bible_hau	hau	27.01	CC-BY-SA-4.0 (Open.Bible / Biblica)
bible_yor	yor	20.43	CC-BY-SA-4.0 (Open.Bible / Biblica)
fleurs	eng	5.63	CC-BY-4.0 (Google FLEURS)
fleurs	hau	7.6	CC-BY-4.0 (Google FLEURS)
fleurs	ibo	6.09	CC-BY-4.0 (Google FLEURS)
fleurs	yor	5.74	CC-BY-4.0 (Google FLEURS)
igboapi	ibo	34.98	CC-BY-4.0 (IgboAPI)
igbosyncorp	ibo	2.16	CC0-1.0 (IgboSynCorp)
naijas2st_igbo	ibo	14.81	CC-BY-4.0 (NaijaS2ST)
open_bible_igbo	ibo	22.02	CC-BY-SA-4.0 (Open.Bible / Biblica)
slr70	eng	4.0	CC-BY-SA-4.0 (OpenSLR 70, Nigerian English)
slr86	yor	2.98	CC-BY-SA-4.0 (OpenSLR 86, Yoruba)
waxal	hau	6.58	CC-BY-SA-4.0 (WAXAL Media Trust subsets)
waxal	ibo	10.58	CC-BY-SA-4.0 (WAXAL Media Trust subsets)
waxal	pcm	6.07	CC-BY-SA-4.0 (WAXAL Media Trust subsets)
waxal	yor	10.48	CC-BY-SA-4.0 (WAXAL Media Trust subsets)

Per-epoch sampling down-weights Bible narration and dictionary reads and up-weights Pidgin (≈ 115.6 h per epoch; Igbo 39, Yoruba 26, Hausa 24, Pidgin 18, English 8 h). Held out and never trained on: WAXAL test + validation, FLEURS dev/test, NaijaS2ST dev, and every IgboSynCorp speaker whose speech contained English.

Synthetic code-switched data (codeswitch-experimental/ only). 2797 code-switched sentences (English with Igbo/Yoruba/Hausa/Pidgin, Pidgin with Igbo/Yoruba/Hausa; spans tagged per language) written by Qwen3-30B-A3B-Instruct-2507 (Apache-2.0), read by the Dynamic Accent Flash-DPO TTS (itself trained on licensed data only) in 32 voices (zero-shot prompts from WAXAL speakers), with random speaking rate (slow/normal/fast), speed perturbation 0.9–1.1, and on-the-fly reverb, phone/cheap-mic band-limiting, clipping and noise (SNR 5–30 dB). Clips the stock model could not read (CER > 0.35, i.e. TTS failures) were dropped: 2572 clips / 4.99 h kept for training, 109 unseen-text clips kept as the synthetic test.

Not used, on licence grounds: African Voices / WaZoBia, NaijaVoices (non-commercial), AfriSpeech-200 (non-commercial). Common Voice was not used (not fetched). No African Voices–trained TTS was used to make synthetic data.

Training

Full fine-tune of all transformer layers (CNN feature encoder frozen), bf16 autocast, AdamW (lr 2e-5, β 0.9/0.98, wd 0.01), 300 warm-up steps then linear decay, 4000 steps × 160 s of audio per batch (~177.8 h of audio seen), CTC loss, LayerDrop 0.1, 15 % light augmentation of real clips. One H100; the two models differ only in the synthetic share (0 % vs 12 %), same seed and steps. Checkpoint selected on a WAXAL-validation/OpenSLR-70 dev set. Training, data-prep, synthesis and evaluation code is in code/training/.

Limitations

• Small, read-speech-heavy test sets. 40 clips per language; real code-switched test = 94 clips. Differences of ~0.01 CER are within noise.
• Convention matching is part of the gain. WAXAL train and test share transcription conventions (e.g. Igbo without tones, Hausa without hooked letters, digits as numerals); some of the improvement on WAXAL tests is learning those conventions, not only better hearing.
• Little natural conversational speech (≈2 h IgboSynCorp in training; none for Yoruba/Hausa). Expect higher error rates on spontaneous, noisy, phone or far-field speech, and on accents unlike the training speakers.
• Code-switching is the weakest area. Real intra-sentential code-switched test data is scarce; the "natural conversational" CER is still ~0.25. English inside Igbo/Yoruba/Hausa is often spelt phonetically.
• Yoruba tone marks are frequently wrong or missing (Yoruba CER without diacritics is much lower than with); Igbo output has no tone marks by design.
• Not evaluated yet on the developer's own voice or other unseen real users; not evaluated for fairness across gender/age/region.
• No punctuation or casing (unlike the stock v2 model's written output).
• Not for high-stakes use (medical, legal, emergency) without human review.

Licence and attribution

• Weights: CC-BY-SA-4.0. Several training sources are CC-BY-SA-4.0 (WAXAL Media Trust subsets, Open.Bible, OpenSLR 70/86), so the fine-tuned weights are shared alike under CC-BY-SA-4.0. The base model omniASR_CTC_1B_v2 is Apache-2.0 (© Meta Platforms).
• Code (transcribe.py, code/): Apache-2.0 (LICENSE-CODE).
• Attribution: Google WAXAL (Media Trust subsets), Google FLEURS (CC-BY-4.0), OpenSLR 70 Nigerian English and OpenSLR 86 Yoruba (Google, CC-BY-SA-4.0), Open.Bible / Biblica (CC-BY-SA-4.0), NaijaS2ST (CC-BY-4.0), IgboAPI (CC-BY-4.0), IgboSynCorp (CC0-1.0, Harvard Dataverse doi:10.7910/DVN/YB9FWK), Meta Omnilingual ASR (Apache-2.0), Qwen3 (Apache-2.0).

Citation

@misc{modular2026dynamicasr,
  title  = {Dynamic ASR: Nigerian multilingual and code-switched speech recognition},
  author = {Modular (Chima Emmanuel Akuegbo)},
  year   = {2026},
  url    = {https://huggingface.co/Modularcomputing/dynamic-asr}
}