--- license: apache-2.0 library_name: pytorch language: - en - pcm - ig - yo - ha tags: - speech-to-speech - full-duplex - voice-agent - code-switching - nigerian-languages - asr - tts - experimental pipeline_tag: audio-to-audio --- # Dynamic Accent Duplex (experimental) **An experimental cascaded full-duplex voice agent for Nigerian code-switched speech** (English, Nigerian Pidgin, Igbo, Yoruba, Hausa) by Modular. You talk however you naturally mix languages; it answers **in the same mix**. Mostly English with an Igbo greeting gets a reply in mostly English with an Igbo greeting. Pidgin gets Pidgin. This is **not** a native speech-to-speech model (no Moshi-style joint model). It is a pipeline of permissively licensed components with a language-mix controller in the middle: ``` mic ─► Silero VAD ─► Dynamic ASR (CTC, streaming partials) ─► per-word language ID + mix estimate │ end-of-turn, barge-in, backchannels │ {eng: 0.8, ibo: 0.2, pattern: greeting:ibo} ▼ ▼ stop playback ◄── barge-in Qwen3-30B-A3B reply (3 candidates, inline //... tags) ▼ validator: mix distance, tags, greeting, length, loops, echo ─► regenerate / fallback ▼ speaker ◄── Dynamic Accent Flash-DPO TTS (chunk streaming, code-switching voice) ◄──┘ ``` **Status: research demo.** Igbo, Yoruba and Hausa beyond greetings and simple phrases are weak (the reply LLM's limit, not the voice's). Read the limitations before using it for anything. ## Listen Five scripted conversations from the offline driver. **Stereo: the user is on the LEFT channel, the agent on the RIGHT** (headphones help). **User turns are TTS-synthesised** with Dynamic Accent Flash female voices, not real people; the agent speaks with the Flash-DPO male anchor voice. These are **cascaded** demos (ASR → reply LLM → TTS), not a native speech-to-speech model, and the reply LLM is **weak in Igbo, Yoruba and Hausa beyond greetings and short phrases**, which you will hear. Each clip has a JSON log next to it in `samples/` (ASR text, target mix, reply, validator outcome, latency). Audio samples: CC-BY-SA-4.0. **1. Igbo greeting + English.** Mostly English with an Igbo greeting and closing ("Ndewo!" … "Daalụ!"): planning a day (bank, market, call mum). Includes a long turn with a mid-turn pause and a barge-in ("Wait, sorry, the bank closes early today"). The agent greets back in Igbo ("Ụtụtụ ọma!") and replies in English; its last line contains a garbled name. **2. Yoruba greeting + English.** "Ẹ káàárọ̀!" then English: buying a phone for a sister on a ₦100k budget, closing with "Ẹ ṣé!". The English replies are fine; the Yoruba the agent adds ("Ọmọ rẹ wà?") is not appropriate here. **3. Nigerian Pidgin.** Pidgin throughout ("How far? Abeg, wetin I fit cook tonight…"), with a barge-in ("No be rice o…"). The agent stays in Pidgin the whole way. **4. Hausa greeting + English.** "Sannu, barka da yamma" then English about a trip to Kano and what to pack; Hausa closing "To, na gode". The English advice is fine; the agent's extra Hausa phrases are not real Hausa, and the final Hausa-only turn falls back to "Sorry, I did not catch all of that". **5. Igbo-heavier.** More Igbo from the user ("Kedu? Biko, nyere m aka" … "Daalụ, ka ọ dị"): booking a taxi to the airport. Shows the weak spot: the agent mostly echoes the user's Igbo phrases rather than writing new Igbo. ## Components and licences | stage | component | licence | |---|---|---| | voice activity / turn-taking | [Silero VAD v5](https://github.com/snakers4/silero-vad) | MIT | | speech recognition | **[Dynamic ASR](https://huggingface.co/Modularcomputing/dynamic-asr)** `real` (Meta `omniASR_CTC_1B_v2` fine-tuned on licensed Nigerian speech; language-agnostic CTC + Latin-script mask). Fallback: stock [Meta Omnilingual ASR](https://github.com/facebookresearch/omnilingual-asr) `omniASR_CTC_1B` | weights CC-BY-SA-4.0 (base Apache-2.0) / Apache-2.0 | | language ID / mix | Dynamic Accent `tagger.py` (rules) + [GlotLID v3](https://huggingface.co/cis-lmu/glotlid) sliding windows | Apache-2.0 / Apache-2.0 | | reply LLM | [Qwen/Qwen3-30B-A3B-Instruct-2507-FP8](https://huggingface.co/Qwen/Qwen3-30B-A3B-Instruct-2507-FP8) served with [vLLM](https://github.com/vllm-project/vllm) | Apache-2.0 / Apache-2.0 | | text-to-speech | Dynamic Accent **Flash-DPO** ([Modularcomputing/dynamic-accent-pro-v1.1](https://huggingface.co/Modularcomputing/dynamic-accent-pro-v1.1), `checkpoints/flash`) on [Fun-CosyVoice3-0.5B-2512](https://huggingface.co/FunAudioLLM/Fun-CosyVoice3-0.5B-2512) | weights CC-BY-SA-4.0 (base Apache-2.0) | | diacritic restoration (only when a Yoruba/Igbo span has no tone marks) | Dynamic Accent ByT5 restorer (base `google/byt5-small`) | CC-BY-SA-4.0 (base Apache-2.0) | | this repo's code | orchestration, mix controller, validator, servers | **Apache-2.0** | Evaluation-only models (not part of the agent): Whisper large-v3-turbo (MIT), omniASR CTC-300M / LLM-1B / LLM-3B (Apache-2.0), Mistral-Small-3.2-24B-Instruct-2506 (Apache-2.0), MADLAD-400-3B-MT (Apache-2.0, back-translation). Evaluation data: WAXAL test splits (Media Trust ibo/yor/hau/pcm subsets, CC-BY-SA-4.0), FLEURS validation (CC-BY-4.0). Not used, on licence grounds: African Voices / WaZoBia data, NaijaVoices (non-commercial), N-ATLaS (1,000-user cap, share-alike). Gemma 3 was not tested (gated). This repo ships **code and config only**; each model is downloaded from its own repository under its own licence. Audio samples are made with the CC-BY-SA-4.0 TTS and are CC-BY-SA-4.0. ## How the language mix works 1. **Per-word language ID.** The release tagger labels every word (`eng/pcm/ibo/yor/hau`) from diacritics, lexicons and Pidgin markers; ASR text has no punctuation or tone marks, so GlotLID scores 4-word sliding windows and confident windows override the rule labels. A sentence-level pass turns English-looking sentences with Pidgin markers into Pidgin. 2. **Mix estimate.** Word shares plus patterns: `greeting:`, `closing:`, `discourse:`, intra- vs inter-sentential switching, blended 30 % with the conversation so far. ASR-misspelt greetings ("nde wo", "sanno", "dalo") are matched against a variant list. 3. **Constrained reply.** The LLM gets a natural-language target ("English about 80 %, Igbo about 20 %; open with a short Igbo greeting"), a phrasebook of correct greetings/thanks/closings, four examples, and must tag every span (`… …`), which is exactly the input format of the Dynamic Accent TTS. 4. **Validator.** Each of 3 candidates is scored: tag format, total-variation distance between target and reply mix (declared tags averaged with re-measured LID), every target language ≥ 15 % present, greeting kept, 2–40 words, no loops, no echo of the user's words. The best passing candidate wins; otherwise one stricter round of 3 more; otherwise a fallback (best candidate, or for an Igbo/Yoruba/Hausa/Pidgin-led turn a correct phrasebook opener in that language plus a short clarification request). ## Duplex behaviour - 32 ms Silero frames; speech start after 96 ms; end of turn after 600 ms of silence. - **Speculative start:** ASR → LLM → first TTS chunk begin at 256 ms of silence; audio is released at end of turn, and dropped if the user resumes. - **Barge-in:** 192 ms of user speech while the agent speaks (or has committed to speak) stops playback at once (client queue flushed); the interrupted reply is kept in history as what was actually played + `[interrupted]`. - **Backchannels:** during a long user turn (≥ 4.5 s), at a 288–448 ms pause, at most every 10 s: "Mm-hmm" (English) / "Ehen" (Pidgin, Igbo, Yoruba) / "To" (Hausa). - **Streaming TTS:** reply cut into a short first chunk (≤ 6–10 words, at a comma/greeting/conjunction), then sentences ≤ 22 words, each synthesised while the previous one plays. ## Run it Tested on one H100 80 GB (vLLM 0.32, torch 2.x). Set up the Dynamic Accent TTS from `Modularcomputing/dynamic-accent-pro-v1.1` as its README describes, then: ```bash pip install -r requirements.txt # duplex server env (the TTS env) pip install "omnilingual-asr>=0.2.0" huggingface_hub fastapi uvicorn # ASR worker env (can be a separate venv) # ASR choice: ASR_MODEL=dynamic-asr (default) | dynamic-asr-cs (experimental code-switch) | omni (stock omniASR_CTC_1B) pip install vllm # LLM server env DA_REL=/path/to/dynamic-accent-pro-v1.1 ./launch.sh # -> http://localhost:7860 (use headphones) ``` Offline scripted conversations (same components, simulated real-time clock; latency = end-of-turn wait + measured compute): ```bash cd code && python offline.py convo scenarios.json out/ # demo conversations -> wav + stereo (user L / agent R) + json log python offline.py eval list.jsonl out/ 40 # batch latency / mix / barge-in eval on audio clips ``` ## Evaluation All numbers from one H100 80 GB with the three processes sharing the GPU. Latency = end of the user's speech (VAD) to the first agent audio sample, from the offline driver: the end-of-turn wait is simulated frame by frame and every compute stage is timed on the real GPU (network and audio-device buffers excluded; the live WebSocket path measured 1.9-2.5 s over the public internet on two test turns). Held-out data only; whether Meta's ASR saw WAXAL/FLEURS in training is unknown. ### ASR (229 held-out clips; WER / CER; CER without diacritics in brackets) | model | eng | pcm | ibo | yor | hau | cs | latency p50 / p90 per clip | CS mix TVD after ASR (rules / +GlotLID windows) | |---|---|---|---|---|---|---|---|---| | omniASR_CTC_1B | 0.14 / 0.07 (0.07) | 0.38 / 0.17 (0.17) | 0.63 / 0.20 (0.16) | 0.49 / 0.19 (0.12) | 0.30 / 0.09 (0.09) | 0.33 / 0.13 (0.10) | 44 / 50 ms | 0.402 / 0.277 | | omniASR_CTC_300M | 0.13 / 0.05 (0.05) | 0.58 / 0.23 (0.23) | 0.70 / 0.21 (0.17) | 0.59 / 0.21 (0.14) | 0.40 / 0.13 (0.13) | 0.40 / 0.14 (0.12) | 27 / 30 ms | 0.381 / 0.381 | | omniASR_LLM_1B | 0.07 / 0.03 (0.03) | 0.33 / 0.15 (0.15) | 0.54 / 0.17 (0.12) | 0.40 / 0.16 (0.11) | 0.30 / 0.09 (0.09) | 0.32 / 0.14 (0.11) | 1037 / 1802 ms | 0.4 / 0.4 | | omniASR_LLM_3B | 0.06 / 0.03 (0.03) | 0.43 / 0.18 (0.18) | 0.63 / 0.21 (0.15) | 0.35 / 0.14 (0.10) | 0.30 / 0.09 (0.09) | 0.32 / 0.15 (0.13) | 848 / 1760 ms | 0.352 / 0.352 | | whisper_turbo | 0.03 / 0.01 (0.01) | 0.42 / 0.21 (0.21) | 1.06 / 0.41 (0.34) | 1.05 / 0.52 (0.29) | 0.90 / 0.33 (0.32) | 0.81 / 0.62 (0.61) | 119 / 208 ms | 0.51 / 0.51 | | omniASR_CTC_1B_latin | 0.14 / 0.07 (0.07) | 0.38 / 0.17 (0.17) | 0.63 / 0.20 (0.16) | 0.49 / 0.19 (0.12) | 0.30 / 0.09 (0.09) | 0.33 / 0.13 (0.10) | 45 / 54 ms | 0.402 / 0.277 | | omniASR_CTC_1B_v2 (stock base) | 0.10 / 0.02 (0.02) | 0.34 / 0.12 (0.12) | 0.46 / 0.13 (0.11) | 0.40 / 0.13 (0.10) | 0.32 / 0.11 (0.11) | 0.29 / 0.10 (0.09) | same CTC-1B arch, not re-timed (≈ CTC_1B) | not measured | | **Dynamic ASR real** (default) | 0.10 / 0.03 (0.03) | 0.19 / 0.07 (0.07) | 0.32 / 0.09 (0.08) | 0.35 / 0.12 (0.08) | 0.25 / 0.08 (0.08) | 0.24 / 0.08 (0.06) | same CTC-1B arch, not re-timed (≈ CTC_1B) | not measured | | Dynamic ASR code-switch (experimental) | 0.09 / 0.02 (0.02) | 0.19 / 0.08 (0.08) | 0.32 / 0.10 (0.08) | 0.35 / 0.12 (0.08) | 0.25 / 0.08 (0.08) | 0.22 / 0.07 (0.06) | same CTC-1B arch, not re-timed (≈ CTC_1B) | not measured | **Note:** the end-to-end latency tables below (and the per-clip ASR latency column) were measured with `omniASR_CTC_1B` as the ASR, before Dynamic ASR existed. Dynamic ASR is the same size and architecture (1B wav2vec2 CTC, greedy decoding), so latency is expected to be similar but was not re-measured. Dynamic ASR rows use the same 229 clips and scorer; see the [Dynamic ASR card](https://huggingface.co/Modularcomputing/dynamic-asr) for the code-switch tests. LID on reference CS text (upper bound): word accuracy 0.9, mix TVD 0.133. ### Reply LLM (28 prompts: 16 mixed/greeting/Pidgin/English written prompts + 12 native WAXAL sentences ibo/yor/hau) | model | subset | first candidate passes | regen rate | fallback | mean mix TVD | GlotLID agrees with span tag | judge 1-5 (back-translated) | LLM s (n=3) | |---|---|---|---|---|---|---|---|---| | mistral-small-3.2-24b | mixed | 0.93 | 0.07 | 0.0 | 0.027 | 1.0 | 4 | 0.433 | | mistral-small-3.2-24b | native | 0.92 | 0.08 | 0.0 | 0.008 | 1.0 | 1 | 0.584 | | mistral-small-3.2-24b | all | 0.89 | 0.07 | 0.0 | 0.017 | 1.0 | 2.79 | 0.497 | | qwen3-30b-a3b-2507 | mixed | 0.86 | 0.07 | 0.0 | 0.018 | 1.0 | 4.43 | 0.277 | | qwen3-30b-a3b-2507 | native | 0.92 | 0.0 | 0.0 | 0.029 | 1.0 | 1.58 | 0.41 | | qwen3-30b-a3b-2507 | all | 0.89 | 0.04 | 0.0 | 0.021 | 1.0 | 3.25 | 0.317 | ### Latency, batch eval (80 turns; seconds, p50 / p90) | stage | p50 | p90 | |---|---|---| | end-of-turn wait (VAD silence) | 0.576 | 0.576 | | ASR final | 0.036 | 0.042 | | LID + mix | 0.013 | 0.03 | | LLM + validator rounds | 0.389 | 0.791 | | TTS first chunk | 0.863 | 1.69 | | compute total (ASR->first chunk) | 1.381 | 2.211 | | **end of user speech -> first agent audio** | 1.637 | 2.467 | Mix adherence (batch_eval): mean TVD(target, reply) 0.124 (p90 0.451); first candidate passes 0.588; regenerated 0.037; fallback 0.15 (safe phrasebook 0.113); 39/80 turns had a multi-language target. ### Latency, demos (19 turns; seconds, p50 / p90) | stage | p50 | p90 | |---|---|---| | end-of-turn wait (VAD silence) | 0.576 | 0.576 | | ASR final | 0.057 | 0.069 | | LID + mix | 0.005 | 0.016 | | LLM + validator rounds | 0.246 | 0.537 | | TTS first chunk | 0.795 | 1.168 | | compute total (ASR->first chunk) | 1.139 | 1.488 | | **end of user speech -> first agent audio** | 1.395 | 1.744 | Mix adherence (demos): mean TVD(target, reply) 0.084 (p90 0.162); first candidate passes 0.789; regenerated 0.0; fallback 0.053 (safe phrasebook 0.053); 11/19 turns had a multi-language target. Barge-in (batch eval, n=40): user speech onset -> agent audio stopped p50 0.235 s, p90 0.3 s (VAD detection p50 0.195 s + 40 ms output flush). ## Demos `samples/` holds five scripted conversations (playable in **Listen** above; stereo, user left / agent right) made with the offline driver (agent: Flash-DPO male anchor). **User turns in these samples are TTS-synthesised** (Flash female voices), not real people. Each has a JSON log with the ASR text, the target mix, the reply, the validator outcome and per-stage latency. ## Limitations (read before use) - **Igbo, Yoruba and Hausa content is weak beyond greetings, thanks and short phrases.** The reply LLM (and every permissive LLM we tried) parrots or garbles longer native-language replies; a back-translation judge scored native-language replies 1.6/5 versus 4.4/5 for mixed/greeting/Pidgin prompts. The validator catches echoing and drift and falls back to an honest short reply, which means native-led turns often get "Sannu! Sorry, I did not catch all of that…". - ASR on accented or synthetic speech misspells Igbo/Yoruba greetings (and loses tone marks), which can hide the user's language from the mix estimate; a variant list covers common greetings only. - Pidgin vs English is the hardest LID boundary; Pidgin detection uses marker words. - Latency p90 is driven by the TTS first chunk (autoregressive 0.5B LM, ~17 ms per audio token, ~0.4-0.6 x real time on a shared H100). - Barge-in uses VAD only: without headphones or echo cancellation the agent's own voice can interrupt it. Backchannels are rule-based and can be too frequent on long read speech. - No safety filtering beyond the LLM's own; replies can be wrong. Research demo, not a product. ## Citation / credits Built by Modular on Meta Omnilingual ASR, Silero VAD, GlotLID, Qwen3, vLLM, CosyVoice3 and the Dynamic Accent TTS (trained on licensed data only: WAXAL, FLEURS, Open.Bible, NaijaS2ST, IgboAPI, IgboSynCorp, BibleTTS, OpenSLR).