--- license: cc-by-sa-4.0 language: - en - pcm - ig - yo - ha pipeline_tag: automatic-speech-recognition library_name: fairseq2 tags: - asr - speech-recognition - code-switching - nigerian-languages - igbo - yoruba - hausa - nigerian-pidgin - ctc - omnilingual-asr - streaming - onnx --- # Dynamic ASR **Speech recognition for Nigerian English, Nigerian Pidgin, Igbo, Yoruba and Hausa — and for speech that switches between them inside a sentence.** By [Modular](https://huggingface.co/Modularcomputing), part of the Dynamic Accent project (see also the [Dynamic Accent TTS](https://huggingface.co/Modularcomputing/dynamic-accent-preview) and the [Dynamic Accent Duplex](https://huggingface.co/Modularcomputing/dynamic-accent-duplex) voice agent, which uses this model as its ASR). Dynamic ASR is Meta's [Omnilingual ASR](https://github.com/facebookresearch/omnilingual-asr) `omniASR_CTC_1B_v2` (1B-parameter wav2vec2 CTC, Apache-2.0) fully fine-tuned on **187.2 hours of licensed Nigerian speech**. The architecture, tokenizer and decoding are unchanged, so it is a drop-in replacement: language-agnostic (no language hint), greedy CTC, cheap enough to re-decode a growing buffer for streaming partials. **Status: experimental research release.** Read the limitations. ## Two models | folder | what it is | use it when | |---|---|---| | `real/` (**default**) | fine-tuned on licensed real speech only | general use; as good as the experimental model on real code-switched speech and better on the selection dev set (mean CER 0.071 vs 0.074) | | `codeswitch-experimental/` | same recipe, plus ~4.99 h of **synthetic code-switched speech** made with our own Dynamic Accent Flash-DPO TTS (12 % of training audio) | research on synthetic code-switch data; see the ablation below before choosing it | Both are the same size (1B, ~1.95 GB bf16 state dict) and load the same way. ## Variants | folder | model | parameters | files | runtime | |---|---|---|---|---| | `real/` (**default**) | omniASR_CTC_1B_v2 fine-tuned on licensed real speech | 1B | `model.pt` (1.95 GB, bf16) | PyTorch / fairseq2 (`transcribe.py`) | | `codeswitch-experimental/` | same, plus synthetic code-switched speech | 1B | `model.pt` (1.95 GB, bf16) | PyTorch / fairseq2 (`transcribe.py`) | | `pocket/` | omniASR_CTC_300M_v2 student distilled from a 1B Dynamic ASR teacher | 300M | `model_int8.onnx` (367 MB), `model_fp32.onnx` (1.30 GB), `model.pt` (1.30 GB), `vocab.json` | onnxruntime on CPU (`pocket/transcribe_pocket.py`) | The Pocket numbers below use a different scorer from the 1B tables in Results, so compare Pocket only with the teacher row in its own table. ## Pocket (300M, CPU) `pocket/` is a smaller model for CPU and offline use. It is Meta's `omniASR_CTC_300M_v2` (300M-parameter wav2vec2 CTC, Apache-2.0, same tokenizer and 20 ms frame rate as the 1B models), distilled from a 1B Dynamic ASR teacher. The teacher is an internal checkpoint (`real/` further fine-tuned with real-voice spliced code-switch clips); it is not one of the files in this repository. Checkpoint: P2, step 2500. **Training.** Loss = CTC on the reference transcripts + KL divergence to the teacher's per-frame distribution; CNN feature encoder frozen. Data: the licensed real speech listed under Training data, code-switch clips spliced from that same licensed speech, and an oversample of 0.9 h of real code-switched training clips. No synthetic TTS audio and no African Voices data. **Files** | file | size | what it is | |---|---|---| | `pocket/model_int8.onnx` | 367 MB | int8-quantised ONNX (recommended for CPU) | | `pocket/model_fp32.onnx` | 1.30 GB | fp32 ONNX | | `pocket/model.pt` | 1.30 GB | PyTorch checkpoint: a dict with `model` (state dict for `omniASR_CTC_300M_v2`), `step` and the training `args` | | `pocket/vocab.json` | 127 KB | CTC vocabulary (index 0 = blank, 1–3 = special tokens) | | `pocket/transcribe_pocket.py` | | CPU inference: onnxruntime, greedy CTC decoding, 16 kHz mono input | | `pocket/results.json` | | the scores and CPU benchmark below | ONNX graph: input `wav` (float32, `[1, samples]`, 16 kHz mono), output `logprobs` (`[1, frames, 10288]`). **Usage** ```bash pip install onnxruntime numpy soundfile scipy huggingface_hub huggingface-cli download Modularcomputing/dynamic-asr --include "pocket/model_int8.onnx" "pocket/vocab.json" "pocket/transcribe_pocket.py" --local-dir . python pocket/transcribe_pocket.py audio.wav # int8 (default) python pocket/transcribe_pocket.py audio.wav --model pocket/model_fp32.onnx # fp32 ``` Any wav is mixed to mono and resampled to 16 kHz; clips over ~20 s are cut at quiet points and transcribed in pieces. Output format is the same as the 1B models. **Results** (CER, greedy CTC, no language model). *gdev* = the general dev set (WAXAL validation + OpenSLR-70, 187 clips); it was used for checkpoint selection, so it is a dev set, not a test set. *Real code-switch* = the held-out 94-clip real code-switched test set described above, never used for training or selection. Scoring: NFC, lowercase, punctuation and apostrophes removed, English number words to digits, Igbo tone marks removed from both sides, Yoruba tone marks kept. | model | English | Hausa | Igbo | Pidgin | Yoruba | gdev mean | real code-switch | |---|---|---|---|---|---|---|---| | Pocket int8 ONNX | 0.0571 | 0.0522 | 0.0595 | 0.0963 | 0.1359 | 0.0802 | 0.1558 | | Pocket fp32 ONNX | 0.0571 | 0.0514 | 0.0595 | 0.0925 | 0.1350 | 0.0791 | 0.1558 | | 1B teacher (greedy) | 0.0294 | 0.0489 | 0.0598 | 0.0823 | 0.1288 | 0.0698 | 0.1375 | - int8 adds about 0.001 to the gdev mean (mostly Pidgin) and nothing on real code-switch. - English is the largest gap to the teacher (0.057 vs 0.029). **CPU speed** (Intel Xeon Platinum 8480+, batch 1, 20 clips totalling 110 s, one fresh process per setting) | model | threads | RTF | median latency per clip | peak RAM | |---|---|---|---|---| | int8 ONNX | 4 | 0.071 | 358 ms | 939 MB | | fp32 ONNX | 4 | 0.186 | 952 ms | 2287 MB | Pocket shares the limitations listed below, with higher error than the 1B models. ## Samples Hear the input and read what the model wrote. The input clips are **synthetic speech** from our [Dynamic Accent](https://huggingface.co/Modularcomputing/dynamic-accent-pro-v1.1) Flash-DPO TTS (female `waxal_ibo_5` or male anchor voice), not real recordings, so they are cleaner than real-world audio. The transcripts come from an internal Dynamic ASR checkpoint newer than `real/` (it adds a Yoruba tone-mark fix and is not one of the files in this repository yet): **with LM** = decoding with a 4-gram KenLM, as the [Duplex](https://huggingface.co/Modularcomputing/dynamic-accent-duplex) agent uses on final segments; **greedy** = plain CTC, no LM. Output is lower-case with no punctuation, and Igbo has no tone marks by design. Audio: mono MP3, CC-BY-SA-4.0. **1. Igbo** (female TTS voice) | | text | |---|---| | reference | Ndewo, kedụ ka ị mere? Anyị ga-aga ahịa echi n'ụtụtụ. | | Dynamic ASR, with LM | nde wo kedu ka i mere anyị ga agaghi echi n'ụtụtụ | | Dynamic ASR, greedy | ndewo kedu ka ị mere anyị ga aga ahị echi n'ụtụtụ | **2. Yoruba** (female TTS voice) | | text | |---|---| | reference | Ẹ káàárọ̀! Ṣé dáadáa ni ẹ wà? Mo fẹ́ lọ sí ọjà lọ́la. | | Dynamic ASR, with LM | ẹkáàárọ̀ ṣé dáadáa ni ẹwà mo fẹ́ lọ sí ọjà lọ́lá | | Dynamic ASR, greedy | ẹká àrọ̀ ṣé dáadá ni ẹwà mo fẹ́ lọ sí ọjà lọla | **3. Hausa** (male TTS voice) | | text | |---|---| | reference | Sannu da zuwa. Yaya aiki? Zan tafi Kano gobe da safe. | | Dynamic ASR, with LM | sannu da zuwa yaya aiki zan tafi kano gobe a safe | | Dynamic ASR, greedy | sanuda zuwa yaya aiki zan tafi kano gube da safe | **4. Pidgin** (female TTS voice) | | text | |---|---| | reference | How far, my guy? Abeg make we see for evening, I dey come. | | Dynamic ASR, with LM | how far my guy abeg make we see for evenin i dey come | | Dynamic ASR, greedy | how far my guy abeg make we see for evening i dey come | **5. Igbo + English (code-switch)** (male TTS voice) | | text | |---|---| | reference | Biko, can you send me the report before Friday? Daalụ. | | Dynamic ASR, with LM | biko can you send me the report before friday talo | | Dynamic ASR, greedy | biko can you send me the report before friday daalo | ## Results All test data is held out (never trained on). Scores are WER / CER after lower-casing and stripping punctuation; "no-diacritic" CER folds tone marks and dots. Decoding: greedy CTC with a Latin-script logit mask (the duplex setting). ### Per language (duplex evaluation set: 229 clips — WAXAL test ibo/yor/hau/pcm × 40, FLEURS validation English × 29, 40 clips made by joining two held-out clips of different languages) **CER** | model | English | Pidgin | Igbo | Yoruba | Hausa | code-switch (concatenated) | |---|---|---|---|---|---|---| | omniASR_CTC_1B (stock; previous duplex ASR) | 0.065 | 0.169 | 0.206 | 0.191 | 0.090 | 0.128 | | omniASR_CTC_1B_v2 (stock base) | 0.024 | 0.123 | 0.133 | 0.135 | 0.106 | 0.104 | | **Dynamic ASR real** (`real/`) | 0.025 | 0.074 | 0.093 | 0.121 | 0.082 | 0.081 | | Dynamic ASR code-switch, experimental (`codeswitch-experimental/`) | 0.024 | 0.076 | 0.095 | 0.118 | 0.081 | 0.072 | **WER** | model | English | Pidgin | Igbo | Yoruba | Hausa | code-switch (concatenated) | |---|---|---|---|---|---|---| | omniASR_CTC_1B (stock; previous duplex ASR) | 0.144 | 0.380 | 0.636 | 0.498 | 0.299 | 0.330 | | omniASR_CTC_1B_v2 (stock base) | 0.097 | 0.336 | 0.456 | 0.401 | 0.317 | 0.292 | | **Dynamic ASR real** (`real/`) | 0.095 | 0.193 | 0.315 | 0.355 | 0.253 | 0.238 | | Dynamic ASR code-switch, experimental (`codeswitch-experimental/`) | 0.092 | 0.193 | 0.321 | 0.353 | 0.252 | 0.216 | ### Real code-switched speech Licensed corpora contain little real code-switched speech, so we built the most honest test we could from held-out data: Igbo/Yoruba/Hausa utterances whose transcripts contain English words. *Natural conversational* = spontaneous Igbo speech with English (IgboSynCorp; those speakers were removed from training). *Insertional* = read Igbo/Yoruba/Hausa sentences with English names/terms (WAXAL test, NaijaS2ST dev). CER (no-diacritic CER in brackets), plus English-word recall (share of the reference's English words that appear in the output; strict exact-match, so low in absolute terms). The last column is a held-out set of synthetic code-switched clips (unseen texts) from the TTS. | model | real CS, all (n=94) | natural conversational Igbo-English (n=36) | insertional, read (n=58) | English-word recall | synthetic CS test (n=109) | |---|---|---|---|---|---| | omniASR_CTC_1B (stock; previous duplex ASR) | 0.224 (0.195) | 0.372 (0.326) | 0.178 (0.154) | 0.3009 | 0.159 (0.129) | | omniASR_CTC_1B_v2 (stock base) | 0.193 (0.172) | 0.353 (0.315) | 0.143 (0.128) | 0.2832 | 0.132 (0.118) | | **Dynamic ASR real** (`real/`) | 0.148 (0.137) | 0.248 (0.234) | 0.117 (0.107) | 0.3186 | 0.105 (0.091) | | Dynamic ASR code-switch, experimental (`codeswitch-experimental/`) | 0.149 (0.137) | 0.247 (0.234) | 0.119 (0.106) | 0.3805 | 0.038 (0.035) | ### Did synthetic code-switched audio help? **Short answer: not on real code-switched speech — at least not measurably at this scale.** The two models share everything (data, seed, 4,000 steps) except that `codeswitch-experimental/` sampled 12 % of its audio from 2572 synthetic code-switched clips (4.99 h). - **Real code-switched speech (94 clips): a tie.** CER 0.148 (`real`) vs 0.149 (`codeswitch-experimental`); natural conversational Igbo–English 0.248 vs 0.247; insertional 0.117 vs 0.119; WER 0.436 vs 0.438. All differences are well inside the noise of a 94-clip test. - **English-word recall did go up**, 0.319 → 0.381 (share of the reference's English words the model writes out exactly). This is the one real-speech signal in favour of synthetic data: the model spells embedded English words as English more often. It did not translate into lower CER, and the test is small, so treat it as a lead, not a result. - **Per-language real tests: no change.** Differences of at most 0.003 CER on English, Pidgin, Igbo, Yoruba and Hausa (B slightly better on Yoruba/Hausa/English, slightly worse on Pidgin/Igbo). On the selection dev set (WAXAL validation + OpenSLR-70) the experimental model is slightly worse (mean CER 0.074 vs 0.071). - **Concatenated code-switch clips (real audio of two languages joined): small gain**, CER 0.081 → 0.072, WER 0.238 → 0.216. These are inter-sentential switches at a hard boundary, closer to what the synthetic data looks like than to natural speech. - **Synthetic code-switched test (109 held-out clips, unseen texts): large gain**, CER 0.105 → 0.038 (stock v2 base: 0.132). This mostly measures adaptation to our own TTS voices and recording style, so it is not evidence about real speakers. **Verdict:** synthetic code-switched audio from our TTS teaches the model the synthetic domain and slightly improves English-word spelling, but it does not lower error on real code-switched or monolingual speech. `real/` stays the default. A real natural code-switching test set (and more real code-switched training speech) is what would settle this. ### AfriSwitch (in-the-wild code-switched speech) — pending We planned to score both models (and the stock v2 base) zero-shot on the Nigerian subsets of Intron Health's **AfriSwitch** benchmark (Igbo, Yoruba, Hausa, Nigerian Pidgin; evaluation only, never trained on). The dataset is gated with manual approval; we requested access on 2026-10-24 and it had not been granted when the models were finalised, so **no AfriSwitch scores are reported yet**. When we run it we will use our scorer's normalisation (lowercase, punctuation stripped; the paper scores diacritic-preserving WER, and our models drop Igbo tone marks, so Igbo WER will be inflated against its references and we will report the diacritic-folded figure alongside), corpus-level WER and CER. For reference, the paper's zero-shot WER (%) on these subsets (Table 2; systems given the matrix language, diacritics preserved): | language | Sahara V2.5 (best overall, avg 35.93 over 12 languages) | Omnilingual LLM 7B | Gemini 3.6 | ElevenLabs | Sahara V2 | |---|---|---|---|---|---| | Igbo | 46.94 | 75.52 | 75.88 | 65.37 | 83.58 | | Yoruba | 45.54 | 70.27 | 77.26 | 74.62 | 89.66 | | Hausa | 31.77 | 39.27 | 45.44 | 41.01 | 45.56 | | Nigerian Pidgin | 32.83 | 42.71 | 38.84 | 35.33 | 42.19 | AfriSwitch: Ashungafac, Awobade & Olatunji (Intron Health), *AfriSwitch: A Benchmark for In-the-Wild African Code-Switched Speech Recognition*, arXiv:2608.26434 (2026). Dataset: https://huggingface.co/datasets/intronhealth/AfriSwitch (dataset card licence: CC-BY-NC-SA-4.0; paper CC BY 4.0). ```bibtex @misc{ashungafac2026afriswitch, title={AfriSwitch: A Benchmark for In-the-Wild African Code-Switched Speech Recognition}, author={Ashungafac, Gabrial Zencha and Awobade, Busayo and Olatunji, Tobi}, year={2026}, eprint={2608.26434}, archivePrefix={arXiv}, primaryClass={cs.CL} } ``` ## Usage ```bash pip install "omnilingual-asr>=0.2.0" huggingface_hub soundfile scipy python transcribe.py audio.wav # real/ model (default) python transcribe.py audio.wav --model codeswitch # codeswitch-experimental/ ``` ```python from transcribe import DynamicASR asr = DynamicASR("real") # downloads real/model.pt; base architecture omniASR_CTC_1B_v2 print(asr(waveform, sr)) # mono float32 array, any sample rate ``` The checkpoint is a plain PyTorch state dict (`{"model": state_dict, ...}`) for `omniASR_CTC_1B_v2`: `transcribe.py` loads the base through `fairseq2.models.hub.load_model("omniASR_CTC_1B_v2")` and overwrites its weights (the base download is needed once for the model skeleton and tokenizer). On `omnilingual-asr` 0.1.0, run `code/training/patch_v2.sh` (adjust the venv path) to register the v2 card first. **Streaming (duplex agent).** `code/asr_server.py` is the FastAPI worker used by Dynamic Accent Duplex: `POST /asr` with raw float32 16 kHz mono → `{"text", "ms"}`; partial hypotheses come from re-decoding the growing buffer every 0.64 s (~30–90 ms per call on an H100). ```bash cd code && ASR_MODEL=dynamic-asr uvicorn asr_server:app --port 8002 # real/ (default) # ASR_MODEL=dynamic-asr-cs -> codeswitch-experimental/ # ASR_MODEL=omni -> stock omniASR_CTC_1B (also the automatic fallback if Dynamic ASR cannot load) ``` **Output format:** lower-case, no punctuation, digits as spoken or written in the training transcripts. Igbo is written **without tone marks** (dotted letters ị ọ ụ ṅ kept, standard everyday orthography); Yoruba **with** tone marks; Hausa as in the sources (hooked letters often missing, as in WAXAL). ## Training data (licensed only) | source | language | hours used | licence | |---|---|---|---| | bible_hau | hau | 27.01 | CC-BY-SA-4.0 (Open.Bible / Biblica) | | bible_yor | yor | 20.43 | CC-BY-SA-4.0 (Open.Bible / Biblica) | | fleurs | eng | 5.63 | CC-BY-4.0 (Google FLEURS) | | fleurs | hau | 7.6 | CC-BY-4.0 (Google FLEURS) | | fleurs | ibo | 6.09 | CC-BY-4.0 (Google FLEURS) | | fleurs | yor | 5.74 | CC-BY-4.0 (Google FLEURS) | | igboapi | ibo | 34.98 | CC-BY-4.0 (IgboAPI) | | igbosyncorp | ibo | 2.16 | CC0-1.0 (IgboSynCorp) | | naijas2st_igbo | ibo | 14.81 | CC-BY-4.0 (NaijaS2ST) | | open_bible_igbo | ibo | 22.02 | CC-BY-SA-4.0 (Open.Bible / Biblica) | | slr70 | eng | 4.0 | CC-BY-SA-4.0 (OpenSLR 70, Nigerian English) | | slr86 | yor | 2.98 | CC-BY-SA-4.0 (OpenSLR 86, Yoruba) | | waxal | hau | 6.58 | CC-BY-SA-4.0 (WAXAL Media Trust subsets) | | waxal | ibo | 10.58 | CC-BY-SA-4.0 (WAXAL Media Trust subsets) | | waxal | pcm | 6.07 | CC-BY-SA-4.0 (WAXAL Media Trust subsets) | | waxal | yor | 10.48 | CC-BY-SA-4.0 (WAXAL Media Trust subsets) | Per-epoch sampling down-weights Bible narration and dictionary reads and up-weights Pidgin (≈ 115.6 h per epoch; Igbo 39, Yoruba 26, Hausa 24, Pidgin 18, English 8 h). Held out and never trained on: WAXAL test + validation, FLEURS dev/test, NaijaS2ST dev, and every IgboSynCorp speaker whose speech contained English. **Synthetic code-switched data (`codeswitch-experimental/` only).** 2797 code-switched sentences (English with Igbo/Yoruba/Hausa/Pidgin, Pidgin with Igbo/Yoruba/Hausa; spans tagged per language) written by Qwen3-30B-A3B-Instruct-2507 (Apache-2.0), read by the Dynamic Accent **Flash-DPO** TTS (itself trained on licensed data only) in 32 voices (zero-shot prompts from WAXAL speakers), with random speaking rate (slow/normal/fast), speed perturbation 0.9–1.1, and on-the-fly reverb, phone/cheap-mic band-limiting, clipping and noise (SNR 5–30 dB). Clips the stock model could not read (CER > 0.35, i.e. TTS failures) were dropped: 2572 clips / 4.99 h kept for training, 109 unseen-text clips kept as the synthetic test. **Not used, on licence grounds:** African Voices / WaZoBia, NaijaVoices (non-commercial), AfriSpeech-200 (non-commercial). Common Voice was not used (not fetched). No African Voices–trained TTS was used to make synthetic data. ## Training Full fine-tune of all transformer layers (CNN feature encoder frozen), bf16 autocast, AdamW (lr 2e-5, β 0.9/0.98, wd 0.01), 300 warm-up steps then linear decay, 4000 steps × 160 s of audio per batch (~177.8 h of audio seen), CTC loss, LayerDrop 0.1, 15 % light augmentation of real clips. One H100; the two models differ only in the synthetic share (0 % vs 12 %), same seed and steps. Checkpoint selected on a WAXAL-validation/OpenSLR-70 dev set. Training, data-prep, synthesis and evaluation code is in `code/training/`. ## Limitations - **Small, read-speech-heavy test sets.** 40 clips per language; real code-switched test = 94 clips. Differences of ~0.01 CER are within noise. - **Convention matching is part of the gain.** WAXAL train and test share transcription conventions (e.g. Igbo without tones, Hausa without hooked letters, digits as numerals); some of the improvement on WAXAL tests is learning those conventions, not only better hearing. - **Little natural conversational speech** (≈2 h IgboSynCorp in training; none for Yoruba/Hausa). Expect higher error rates on spontaneous, noisy, phone or far-field speech, and on accents unlike the training speakers. - **Code-switching is the weakest area.** Real intra-sentential code-switched test data is scarce; the "natural conversational" CER is still ~0.25. English inside Igbo/Yoruba/Hausa is often spelt phonetically. - Yoruba tone marks are frequently wrong or missing (Yoruba CER without diacritics is much lower than with); Igbo output has no tone marks by design. - Not evaluated yet on the developer's own voice or other unseen real users; not evaluated for fairness across gender/age/region. - No punctuation or casing (unlike the stock v2 model's written output). - Not for high-stakes use (medical, legal, emergency) without human review. ## Licence and attribution - **Weights: CC-BY-SA-4.0.** Several training sources are CC-BY-SA-4.0 (WAXAL Media Trust subsets, Open.Bible, OpenSLR 70/86), so the fine-tuned weights are shared alike under CC-BY-SA-4.0. The base model omniASR_CTC_1B_v2 is Apache-2.0 (© Meta Platforms). - **Code (`transcribe.py`, `code/`, `pocket/transcribe_pocket.py`): Apache-2.0** (`LICENSE-CODE`). - **Pocket weights (`pocket/`): CC-BY-SA-4.0**, like the 1B weights. Base model omniASR_CTC_300M_v2 is Apache-2.0 (© Meta Platforms). - Attribution: Google WAXAL (Media Trust subsets), Google FLEURS (CC-BY-4.0), OpenSLR 70 Nigerian English and OpenSLR 86 Yoruba (Google, CC-BY-SA-4.0), Open.Bible / Biblica (CC-BY-SA-4.0), NaijaS2ST (CC-BY-4.0), IgboAPI (CC-BY-4.0), IgboSynCorp (CC0-1.0, Harvard Dataverse doi:10.7910/DVN/YB9FWK), Meta Omnilingual ASR (Apache-2.0), Qwen3 (Apache-2.0). ## Citation ```bibtex @misc{modular2026dynamicasr, title = {Dynamic ASR: Nigerian multilingual and code-switched speech recognition}, author = {Modular (Chima Emmanuel Akuegbo)}, year = {2026}, url = {https://huggingface.co/Modularcomputing/dynamic-asr} } ```