UncleanCode
Dynamic Accent Flash: add Flash-DPO (CosyVoice3) as --version flash (default), diacritic front-end, Flash results, samples
e4ae2bb verified 2 days ago
preview code
Raw History Blame Contribute Delete
31.5 kB
---
license: cc-by-sa-4.0
language:
  - en
  - pcm
  - yo
  - ig
  - ha
base_model:
  - FunAudioLLM/Fun-CosyVoice3-0.5B-2512
  - FunAudioLLM/CosyVoice2-0.5B
pipeline_tag: text-to-speech
library_name: pytorch
tags:
  - text-to-speech
  - code-switching
  - nigerian-languages
  - igbo
  - yoruba
  - hausa
  - nigerian-pidgin
  - cosyvoice2
  - cosyvoice3
  - dpo
datasets:
  - google/WaxalNLP
  - McGill-NLP/NaijaS2ST
  - AfriSpeech/open-bible-speech-african
  - nkowaokwu/ibo-dict
  - google/fleurs
---
<p align="center"><img src="assets/dynamic_accent_banner.jpg" width="600"/></p>

# Dynamic Accent: Code-switching TTS

**Code-switching TTS for Nigerian languages.** English, Nigerian Pidgin, Igbo, Yoruba and Hausa, mixed freely inside one sentence and spoken by one consistent voice, with no model swapping between languages.

**New: Dynamic Accent Flash** (`--version flash`, now the default) is the same recipe ported to [FunAudioLLM/Fun-CosyVoice3-0.5B-2512](https://huggingface.co/FunAudioLLM/Fun-CosyVoice3-0.5B-2512), then preference-tuned with DPO for naturalness, intelligibility, pace and pauses, with a text front-end that restores Yoruba/Igbo diacritics. It is trained on licensed data only. See **Flash results** below.

Dynamic Accent (Phase 2/3) is a fine-tune of [FunAudioLLM/CosyVoice2-0.5B](https://huggingface.co/FunAudioLLM/CosyVoice2-0.5B) by Modular. It adds:
- per-token **language embeddings**, a **speaker token** and an adversarial (gradient-reversal) speaker classifier on the LM, so the voice stays the same across a language switch (after Zhang et al., Google, 2019, *Learning to Speak Fluently in a Foreign Language*);
- training on about 178 h of Nigerian-language and code-switched speech, including about 102 h of Igbo;
- learned `[pause]`/`[breath]` events and a speaking-rate instruction (Phase 3);
- a **flow-matching decoder fine-tuned on the anchor voice** for tone and pitch detail.

It is a research release, not a production system. Read **Limitations** before you use it.

## Versions and voices

Three model versions ship side by side. Pick one with `--version`, or render several with `--compare` and A/B before use.

| version | LM | flow | decoding | what changed |
|---|---|---|---|---|
| `flash` (**default**) | `checkpoints/flash/flash_dpo` on **Fun-CosyVoice3-0.5B-2512**: Flash SFT (CV3 LM, the Phase 3 data re-weighted towards natural speech, plus IgboSynCorp pseudo-transcripts) + DPO (1,206 preference pairs; reward = naturalness + CER + articulation rate + pause rate) | stock CV3 flow + vocoder, zero-shot from the voice clip | text front-end (numbers verbalised, missing Yoruba/Igbo diacritics restored by `checkpoints/flash/diacritics_byt5`), one pass per paragraph, silence-aware repetition check, no audio post-processing | pace about native, fewest clipped endings for every voice, lower anchor CER (see **Flash results**) |
| `phase3` | `checkpoints/phase3/p3_lm_full` (Phase 2 LM + 5,000 steps on a 178 h re-cut with ~102 h Igbo, `[pause]`/`[breath]` tags and a speed instruction, rate `normal`) | shared `checkpoints/phase2/p2_flow_sft` | one pass per paragraph, silence-aware repetition check, no audio post-processing | lower CER in Igbo/Hausa/Yoruba/code-switch/paragraphs, comma pauses at the anchor's length, clipped endings 63% → 11% |
| `phase2` | `checkpoints/phase2/p2_lm_B` (~120 h, pause profile B) | `checkpoints/phase2/p2_flow_sft` | sentence by sentence, internal silences capped at 0.6 s | the Phase 2 final system |

| voice (`--voice`) | who | notes |
|---|---|---|
| `anchor_male` (default) | WAXAL Yoruba speaker 4, male | the voice both versions were trained around |
| `waxal_ibo_5_female` | WAXAL Igbo speaker 5, female, native Igbo | zero-shot (prompt clip, no fine-tune); best CER of the three on the female test subset |
| `waxal_yor_5_female` | WAXAL Yoruba speaker 5, female, native Yoruba | zero-shot; strongest identity match, but its 3.8 s prompt has no pauses, so pauses are short and endings often clipped |

**Recommended:** `flash` with any of the three voices; it scores best overall (see **Flash results**), with one known weakness: with `anchor_male` it under-pauses in Igbo and Yoruba. Earlier listening verdict (before Flash): Phase 3 sounds best with the female voices, Phase 2 is fine for the male anchor. `infer.py` prints a note when you pick a pairing outside these. The female voices run with `phase2` but are untested there. Settings live in `settings.json`.

## Setup (single GPU)

```bash
# 1. CosyVoice code (the commit the model was built and tested on)
git clone --recursive https://github.com/FunAudioLLM/CosyVoice.git
cd CosyVoice && git checkout 074ca6dc9e80a2f424f1f74b48bdd7d3fea531cc && git submodule update --init --recursive && cd ..
# 2. This repo
huggingface-cli download Modularcomputing/dynamic-accent-preview --local-dir dynamic-accent-preview
# 3. Python deps (Python 3.10; on Colab keep the preinstalled torch/torchaudio)
pip install -r dynamic-accent-preview/requirements.txt
```

The stock base models download on first run and need no changes: `FunAudioLLM/Fun-CosyVoice3-0.5B-2512` to `pretrained/Fun-CosyVoice3-0.5B-2512` for `flash` (`--cv3_dir` / `CV3_DIR` to reuse a copy), and `FunAudioLLM/CosyVoice2-0.5B` to `pretrained/CosyVoice2-0.5B` for `phase2`/`phase3`. The same CosyVoice commit runs both (it has `CosyVoice3`). **`flash` needs a CUDA GPU** (tested on an H100, bf16). Point `--model_dir` (or `COSYVOICE2_DIR`) at an existing copy, and `--cosyvoice_dir` (or `COSYVOICE_REPO`) at your CosyVoice clone. A 16 GB GPU is enough for the CLI; the Gradio app keeps both versions loaded. Tested on an H100 (bf16). `infer.py` also runs on CPU in fp32, which is slow. T4-class GPUs fall back to fp16, which has not been tested.

## Usage

```bash
cd dynamic-accent-preview
# default: --version flash --voice anchor_male
python infer.py --cosyvoice_dir ../CosyVoice --text "<eng>this is dynamic accent,</eng> <pcm>e dey sound nice abi?</pcm>" --out out.wav
# Flash with the female Igbo voice
python infer.py --cosyvoice_dir ../CosyVoice --version flash --voice waxal_ibo_5_female \
  --text "<ibo>Kedu ka ị mere?</ibo> <pcm>Abeg no vex, I go call you for evening.</pcm>" --out ibo_f.wav
# plain Yoruba without tone marks: Flash's front-end restores them before synthesis
python infer.py --cosyvoice_dir ../CosyVoice --version flash --voice waxal_yor_5_female --frontend native --frontend_lang yor \
  --text "E kaaaro, se daadaa ni gbogbo yin wa?" --out yor.wav
# A/B: the same text, seed and voice through several versions -> cmp_phase2.wav, cmp_phase3.wav, cmp_flash.wav
python infer.py --cosyvoice_dir ../CosyVoice --compare --compare_versions phase2,phase3,flash --voice anchor_male --text "..." --out cmp.wav
python infer.py --list_voices     # voices, versions, recommended pairing
```

| flag | what it does |
|---|---|
| `--text` | the text: tagged (below) or plain with `--auto_tag` |
| `--version flash\|phase3\|phase2` | model version (default `flash`) |
| `--voice anchor_male\|waxal_ibo_5_female\|waxal_yor_5_female` | voice (default `anchor_male`); old ids `anchor`, `waxal_ibo_5`, `waxal_yor_5` still work |
| `--list_voices` | list voices, versions and the recommended pairing |
| `--compare`, `--compare_versions phase2,phase3,flash` | render with several versions → `<out>_<version>.wav` |
| `--frontend codeswitch\|native\|accent` | `flash` text front-end: `codeswitch` (default; tagged text, or untagged text in `--frontend_lang`), `native` (whole text in `--frontend_lang`, diacritics restored), `accent` (words as written, read with the `--frontend_lang` accent) |
| `--frontend_lang eng\|pcm\|ibo\|yor\|hau` | language for untagged text / `native` / `accent` |
| `--no_diacritics` | `flash`: do not restore missing Yoruba/Igbo tone marks and underdots |
| `--frontend translate --experimental_translate` | **experimental, off by default**: translate English/Pidgin into `--frontend_lang` (ibo/yor/hau) with MADLAD-400-3B-MT (Apache-2.0, ~12 GB download) before synthesis. MADLAD often loops (repeated words or dots) on longer inputs and translates Pidgin poorly, so check the printed text |
| `--cv3_dir` | `flash`: path of the Fun-CosyVoice3-0.5B-2512 snapshot |
| `--rate default\|none\|very slow\|slow\|normal\|fast\|very fast` | Phase 3 / Flash speaking-rate instruction to the LM (default: the version's setting, `normal`). It is a learned instruction, not time-stretching. `phase2` ignores it |
| `--auto_tag` | tag untagged mixed text automatically (see below), print the tagged text, then synthesise |
| `--tag_only` | with `--auto_tag`: print the tagged text and stop, so you can edit it and pass it back as `--text` |
| `--force_lang eng\|pcm\|ibo\|yor\|hau` | read the whole text in one language (tags are ignored) |
| `--lang_map "omo=pcm,sha=pcm"` | per-word language overrides for the tagger |
| `--lid none\|glotlid` | with `--auto_tag`: add [GlotLID v3](https://huggingface.co/cis-lmu/glotlid) clause scores (Apache-2.0 plus notices, 1.7 GB, `pip install fasttext`) |
| `--seed N` | sampling seed (default 0). Try 2–3 seeds and keep the take you like |
| `--out file.wav` | output path (24 kHz WAV) |
| `--dtype auto\|bf16\|fp16\|fp32` | LM autocast dtype on GPU (samples: bf16). CPU always runs fp32 |
| `--stitch` | `phase3`: sentence by sentence instead of one pass (not recommended) |
| `--prompt_clip 0-3` | which anchor prompt clip (released systems: 0) |
| `--model_dir`, `--cosyvoice_dir`, `--lm`, `--flow`, `--prompts`, `--voices` | paths and checkpoint overrides |
| `--pause_mode clause`, `--speed` | `phase2` only: the experimental "prosody v2" (fixed gaps plus WSOLA speed-up). A listening test rejected it; the defaults (`model`, `1.0`) are the real Phase 2 system |

Generation on GPU is not bit-reproducible: bf16 sampling can pick a different token on near-ties, so the same seed can give a slightly different take.

### Controlling code-switching

**1. Explicit tags (most control).** Wrap each span in `<eng>`, `<pcm>`, `<ibo>`, `<yor>` or `<hau>`. Keep punctuation inside the span it belongs to and put a space between spans. Untagged text is read as English.

```text
<eng>this is dynamic accent,</eng> <pcm>e dey sound nice abi?</pcm>
<pcm>omo,</pcm> <eng>I'm very excited to announce this new model,</eng> <ibo>ọ na-eme m obi ụtọ,</ibo> <pcm>I no go lie give you</pcm>
<yor>Ẹ káàárọ̀ o, ṣé dáadáa ni?</yor> <eng>I will send you the report this afternoon.</eng>
<hau>Ina kwana, yaya aiki?</hau> <eng>The meeting has been moved to Friday morning.</eng>
<ibo>Ndewo nụ, ụmụnna m.</ibo> <pcm>Una well done o, I know say the road no easy.</pcm> <eng>We have three things to discuss this evening.</eng>
<eng>Please tell everyone that</eng> <yor>a ó pàdé ní ọ̀la.</yor>
```

A switch can fall mid-sentence (`<eng>Please tell everyone that</eng> <yor>a ó pàdé ní ọ̀la.</yor>`). The tag sets the pronunciation and accent of the words inside it; the voice stays the same.

**2. `--auto_tag` for untagged text.** `tagger.py` does word- and span-level language ID: Yoruba/Igbo/Hausa letters (ẹ ṣ / ị ụ ṅ / ɓ ɗ ƙ ƴ), small lexicons of frequent words, Pidgin markers ("dey", "abi", "omo", "wetin", "abeg", "una", "no go", "e don", "know say", pronoun + "go" + verb …) and clause context for ambiguous words ("na", "o", "ka", "ya"). It prints the tagged text so you can check and edit it:
```bash
python infer.py --auto_tag --tag_only --text "omo, I'm very excited to announce this new model, ọ na-eme m obi ụtọ, I no go lie give you"
# [tagged] <pcm>omo,</pcm> <eng>I'm very excited to announce this new model,</eng> <ibo>ọ na-eme m obi ụtọ,</ibo> <pcm>I no go lie give you</pcm>
python infer.py --auto_tag --text "..." --out out.wav            # tag, print, synthesise
python tagger.py "Kedu ka ị mere? Abeg no vex, I go call you for evening."   # tagger on its own
```
Write Yoruba and Igbo with their diacritics where you can; text without diacritics is much harder to identify. See the tagger accuracy below.
**3. `--force_lang`** reads everything in one language, e.g. `--force_lang ibo`.

**4. `--lang_map`** fixes words the tagger gets wrong, e.g. `--lang_map "omo=pcm,sha=pcm,oga=pcm"`.
### Gradio app (local)
```bash
python app.py --cosyvoice_dir ../CosyVoice      # http://127.0.0.1:7860  (--share for a public link)
```
It has a text box, an auto-tag toggle with an editable tagged-text preview, word overrides, a force-language option, version, voice and rate dropdowns, a seed, and a **Compare** button that plays Phase 3 and Flash side by side. The app launch has not been smoke-tested yet.
### From Python
```python
import types, soundfile as sf, infer
a = types.SimpleNamespace(model_dir="pretrained/CosyVoice2-0.5B", cosyvoice_dir="../CosyVoice", lm="", flow="",
                          prompts="prompts/prompts.json", voices="prompts/voices.json", dtype="auto", stitch=False,
                          rate="default", voice="waxal_ibo_5_female")
tts = infer.build("phase3", a)      # for "flash" also set frontend="codeswitch", frontend_lang="eng", no_diacritics=False,
                                     # experimental_translate=False, cv3_dir="pretrained/Fun-CosyVoice3-0.5B-2512"
wav = tts.say("<ibo>Kedu ka ị mere?</ibo> <eng>The meeting has been moved to Friday morning.</eng>", seed=0)
sf.write("out.wav", wav, tts.sr)
```
## Samples
### Flash (`--version flash`, Flash-DPO)
Seed 0, model pauses, speed 1.0, no post-processing. Inputs are written with diacritics unless noted.
| sample | voice | text |
|---|---|---|
| [igbo_pure_female](samples/flash/igbo_pure_female.mp3) | waxal_ibo_5_female | Chei, obi dị m ụtọ nke ukwuu ịkpọsa ụdị ọhụrụ a, ọ na-eme m obi ụtọ, agaghị m agha gị ụgha. |
| [igbo_para_female](samples/flash/igbo_para_female.mp3) | waxal_ibo_5_female | Ndewo nu, ụmụnna m. Taa bụ ụbọchị pụrụ iche n'obodo anyị. Anyị ga-ezukọ n'ama egwuregwu, rie nri, ma kparịta ụka maka ọdịnihu ụmụaka anyị. |
| [yoruba_female](samples/flash/yoruba_female.mp3) / [yoruba_male](samples/flash/yoruba_male.mp3) | waxal_yor_5_female / anchor_male | Ẹ káàsán o, ẹ̀yin ará. Inú mi dùn láti rí gbogbo yín lónìí. Ẹ jẹ́ ká jọ ṣiṣẹ́ pọ̀ fún ìlọsíwájú ìlú wa. |
| [hausa_male](samples/flash/hausa_male.mp3) | anchor_male | Barka da yamma, 'yan uwa. Muna farin cikin ganin ku a yau. Za mu fara taron nan ba da daɗewa ba, don haka ku zauna lafiya. |
| [pidgin_female](samples/flash/pidgin_female.mp3) | waxal_ibo_5_female | Omo, this traffic no be here o. Abeg make una no vex, I go reach there small time. Wetin dey happen for your side? |
| [codeswitch_ibo_pcm_eng](samples/flash/codeswitch_ibo_pcm_eng.mp3) | waxal_ibo_5_female | `<ibo>Nne m, ọ dị mma, ka anyị gaa ahịa taa.</ibo> <pcm>Abeg make we no late o,</pcm> <eng>the shop closes at 6pm.</eng>` |
| [codeswitch_yor_eng](samples/flash/codeswitch_yor_eng.mp3) | waxal_yor_5_female | `<yor>Ẹ jọ̀ọ́, ẹ dúró díẹ̀,</yor> <eng>I will send you the report this afternoon.</eng> <yor>Ẹ ṣé púpọ̀.</yor>` |
| [codeswitch_dynacc](samples/flash/codeswitch_dynacc.mp3) | waxal_ibo_5_female | `<eng>this is dynamic accent,</eng> <pcm>e dey sound nice abi?</pcm>` |
| [numbers_igbo](samples/flash/numbers_igbo.mp3) | waxal_ibo_5_female | Ọnụ ahịa ya bụ ₦2,500, anyị ga-ahụ gị na 21/10/2026 n'elekere 4:30. (numbers verbalised by the front-end) |
| [diacritics_yoruba_from_plain](samples/flash/diacritics_yoruba_from_plain.mp3) | waxal_yor_5_female | input without marks: "E kaaaro, se daadaa ni gbogbo yin wa? Oju ojo dara pupo loni." → front-end: Ẹ káàárọ̀, ṣé dáadáa ni gbogbo yín wà? Ojú ọjọ́ dára púpọ̀ lónì. |
### Phase 2 and Phase 3
Seed 0, rendered with `infer.py`. Inputs: `para_ibo_pcm_eng`, `para_yor_eng`, `para_hau_eng_pcm` (code-switched paragraphs), `ibo_000` (Igbo), `custom_dynacc` ("this is dynamic accent, e dey sound nice abi?") and `omo_announce` ("omo, I'm very excited to announce this new model, ọ na-eme m obi ụtọ, I no go lie give you").
| input | Phase 2, anchor_male | Phase 3, anchor_male | Phase 3, female |
|---|---|---|---|
| para_ibo_pcm_eng | [mp3](samples/phase3/phase2/para_ibo_pcm_eng.mp3) | [mp3](samples/phase3/phase3/para_ibo_pcm_eng.mp3) | [waxal_ibo_5_female](samples/phase3/phase3_female/para_ibo_pcm_eng__waxal_ibo_5.mp3) |
| para_yor_eng | [mp3](samples/phase3/phase2/para_yor_eng.mp3) | [mp3](samples/phase3/phase3/para_yor_eng.mp3) | [waxal_yor_5_female](samples/phase3/phase3_female/para_yor_eng__waxal_yor_5.mp3) |
| para_hau_eng_pcm | [mp3](samples/phase3/phase2/para_hau_eng_pcm.mp3) | [mp3](samples/phase3/phase3/para_hau_eng_pcm.mp3) | – |
| ibo_000 | [mp3](samples/phase3/phase2/ibo_000.mp3) | [mp3](samples/phase3/phase3/ibo_000.mp3) | [waxal_ibo_5_female](samples/phase3/phase3_female/ibo_000__waxal_ibo_5.mp3) |
| custom_dynacc | [mp3](samples/phase3/phase2/custom_dynacc.mp3) | [mp3](samples/phase3/phase3/custom_dynacc.mp3) | [waxal_yor_5_female](samples/phase3/phase3_female/custom_dynacc__waxal_yor_5.mp3) |
| omo_announce | [mp3](samples/phase3/phase2/omo_announce.mp3) | [mp3](samples/phase3/phase3/omo_announce.mp3) | [waxal_ibo_5_female](samples/phase3/phase3_female/omo_announce__waxal_ibo_5.mp3), [waxal_yor_5_female](samples/phase3/phase3_female/omo_announce__waxal_yor_5.mp3) |
Older renders: `samples/pcm_eng_dynamic_accent_seed{0,1,2}.mp3` (Phase 2, three seeds) and `samples/prosody_v2/` (the rejected prosody-v2 post-processing, kept for reference).
## Flash results
Same eval set and judges as the Phase 2 vs Phase 3 tables below (183 items plus code-switched and paragraph groups), seed 0, all groups pooled. **Pace** and **pauses** are ratios to native speakers reading the same texts (1.00 = native). **Clipped endings** = share of renders whose trailing silence is under 0.1 s (native recordings: 23.8%). SIM is ECAPA cosine to the voice's own real clips.

| voice | system | CER | CER no tones | SIM | pace × native | pauses/s × native | clipped endings |
|---|---|---|---|---|---|---|---|
| anchor_male | Phase 3 | 0.174 | 0.146 | 0.815 | 0.92 | 1.11 | 9.1% |
| anchor_male | **Flash** | **0.162** | **0.135** | 0.784 | **0.99** | 0.75 | **7.1%** |
| waxal_ibo_5_female | Phase 3 | 0.164 | 0.132 | 0.683 | 1.00 | 0.74 | 22.8% |
| waxal_ibo_5_female | **Flash** | 0.165 | 0.134 | **0.700** | 1.03 | **0.95** | **10.7%** |

- Flash lowers the anchor's CER (0.174 → 0.162) and clipped endings (9.1% → 7.1%), brings its pace to about native (0.92× → 0.99×), and more than halves clipped endings on the female Igbo voice (22.8% → 10.7%) at about native pace and pause rate.
- Per language with the anchor, Flash CER: Igbo 0.189 (Phase 3: 0.210), Yoruba 0.230 (0.243), Hausa 0.123 (0.146), English 0.026 (0.039), paragraphs 0.131 (0.168), code-switch 0.208 (0.212); Pidgin 0.227 (0.209) is worse, and the Pidgin judge is weak anyway.
- **Known weakness:** with `anchor_male`, Flash **under-pauses in Igbo (0.40× native pauses per second) and Yoruba (0.30×)**. Long Igbo or Yoruba passages with the male voice can sound rushed; use a female voice or `--compare` with `phase3` there.
- Speaker similarity of the anchor drops slightly (0.815 → 0.784), because Flash uses the stock CV3 flow zero-shot instead of the anchor-SFT flow.
- **UTMOS caveat:** UTMOS22 rates real native speech (3.53) below every synthetic system (3.8–4.3), so it rewards clean studio-like synthesis rather than nativeness. It was not used to choose the model (Phase 3 has the highest UTMOS22: 4.15 anchor / 4.27 female; Flash 3.89 / 4.16; Flash UTMOSv2 3.17 / 3.51). The choice rests on CER, pace and pauses relative to native speech, and clipped endings. Native-speaker MOS / blind A/B is still pending.
- Text front-end: diacritic restoration word accuracy Yoruba 0.862 (stripped input: 0.171), Igbo 0.968 (0.640).

## Evaluation: Phase 2 vs Phase 3

183-item eval set (30 per language, 10 code-switched, 3 hand-written + 20 native long-form paragraphs), anchor voice, seed 0. CER is ASR round-trip: MMS-1b-all for Igbo/Hausa/Yoruba, Whisper-large-v3-turbo for English and Pidgin, the lower of the two for code-switched items and paragraphs. "No tones" (in brackets) strips diacritics. SIM is ECAPA cosine to the anchor (ceiling about 0.84 on real held-out anchor clips). The Pidgin judge is weak: real Pidgin recordings score about 0.43.

| system | ibo | hau | yor | pcm | eng | CS | para (hand / native) | SIM |
|---|---|---|---|---|---|---|---|---|
| `phase2` as shipped (sentence stitching) | 0.215 (0.171) | 0.194 | 0.234 (0.149) | 0.196 | 0.035 | 0.163 | 0.192 / 0.154 | 0.785 |
| Phase 2 LM, Phase 3 decoding | 0.243 (0.189) | 0.169 | 0.275 (0.175) | 0.195 | 0.046 | 0.185 | 0.195 / 0.158 | 0.787 |
| **`phase3`** | **0.219 (0.173)** | **0.151** | **0.233 (0.145)** | 0.202 | 0.043 | **0.151** | **0.170** / 0.155 | **0.788** |

Same decoding, Phase 3 LM vs Phase 2 LM: lower CER in every language except Pidgin, same speaker similarity.

| system | comma pause s | sentence pause s | pauses / s | pause-length W1 vs anchor | trailing silence s | clipped endings (< 0.1 s) | articulation syl/s |
|---|---|---|---|---|---|---|---|
| real anchor recordings | 0.417 | 0.543 | 0.624 | 0 | 0.16 | 15% | 4.43 |
| Phase 2 LM, Phase 3 decoding | 0.403 | 0.389 | 0.329 | 0.194 | 0.08 | 63% | 4.36 |
| **`phase3`** | **0.422** | **0.453** | 0.319 | **0.161** | **0.17** | **11%** | 4.44 |

Female voices (29-item subset, `phase3`):

| voice | CER | no-tone CER | SIM to the speaker's own real clips | clipped endings |
|---|---|---|---|---|
| `anchor_male` (same items) | 0.182 | 0.141 | 0.12–0.13 (it is a different speaker) | 17% |
| `waxal_ibo_5_female` | **0.149** | **0.101** | 0.64 | 21% |
| `waxal_yor_5_female` | 0.171 | 0.134 | **0.76** | 62% |

Naturalness has not been rated by native speakers (no MOS or blind A/B yet). The numbers above are automatic proxies.

### Auto-tagger accuracy

Word-level accuracy of `tagger.py` against hand-tagged text (no GlotLID / with `--lid glotlid`):

| test set | words | rule-based | + GlotLID | exact sentences (rule-based) |
|---|---|---|---|---|
| the 6 sample inputs above | 145 | 1.000 | 1.000 | 6/6 |
| 10 new mixed sentences, written before the lexicons | 113 | 0.991 | 0.991 | 9/10 |
| 12 code-switched eval items (overlaps the samples) | 206 | 0.995 | 1.000 | 11/12 |
| 150 single-language eval sentences (real WAXAL/FLEURS text, never looked at while building the lexicons) | 1,697 | **0.898** | **0.915** | 115/150 |

On the held-out single-language set, per language (rule-based / + GlotLID): English 0.99 / 1.00, Igbo 1.00 / 1.00, Yoruba 1.00 / 1.00, Hausa 0.89 / 0.94, **Pidgin 0.63 / 0.65**. Pidgin written without markers looks like English and gets tagged `eng`. The mixed test sentences were written and gold-tagged by the tagger's author, not native speakers, so their scores are optimistic. Check the printed tags, and use explicit tags or `--lang_map` when it matters.

## Limitations

- **Flash** (`flash`): the male anchor under-pauses in Igbo and Yoruba (0.40× / 0.30× native pause rate); Pidgin CER is slightly worse than Phase 3; CUDA only; the translate-first mode is experimental and off by default (MADLAD-400 loops on long inputs and is weak on Pidgin); diacritic restoration can still pick the wrong tone on ambiguous words, so write marks yourself where you can. The limitations below for Phase 3 (no laughter, Pidgin data, accent after a switch) apply to Flash too; the pace gap is largely closed for Flash.
- **Pace (`phase2`/`phase3`)**: speech is still 13–28% slower than native speakers (Igbo 4.60 vs 5.30 syl/s, Hausa 3.77 vs 5.26 syl/s), and there are about half as many pauses per second as in real speech. It can sound careful rather than fully native.
- **Pidgin is the weakest language**: only about 5 h of training data, no reliable ASR judge, and the auto-tagger's Pidgin/English split is its weakest point (~0.63 word accuracy on plain Pidgin).
- **No laughter or other expressive events**: the training data has effectively no laughter, so tags like `[laughter]` do nothing useful. `[pause]`/`[breath]` were learned from data, but there is no reliable user control over them yet.
- English after a Nigerian-language span keeps a strong local accent.
- Yoruba tone quality needs native listening; the automatic tone metric is saturated.
- Female voices are zero-shot; `waxal_yor_5_female` copies its short prompt (short pauses, clipped endings).
- Not evaluated for numbers, abbreviations or URLs; write numbers as words.
- Do not use it to impersonate real people. The voices are WAXAL Media Trust speakers, released under CC-BY-SA-4.0 for speech technology.

## What is in the repo

| path | what it is | licence |
|---|---|---|
| `checkpoints/flash/flash_dpo/` | Flash LM (`best.pt`, sha256 `f59a6fc0…7e3a307`, CosyVoice3 LM + language/speaker/rate embeddings), `config.json`, `spk2id.json`, DPO `args.json`, `log.jsonl` | CC-BY-SA-4.0 |
| `checkpoints/flash/diacritics_byt5/` | Yoruba/Igbo diacritic-restoration model (fine-tune of [google/byt5-small](https://huggingface.co/google/byt5-small), Apache-2.0) used by the Flash front-end | CC-BY-SA-4.0 |
| `checkpoints/phase3/p3_lm_full/` | Phase 3 LM (`best.pt`, val step 4,500), `config.json` (rate, sampler, one-pass, points at the shared Phase 2 flow), `spk2id.json`, `args.json`, `log.jsonl` | CC-BY-SA-4.0 |
| `checkpoints/phase2/p2_lm_B/` | Phase 2 LM (`best.pt`) and `spk2id.json` | CC-BY-SA-4.0 |
| `checkpoints/phase2/p2_flow_sft/best.pt` | anchor-SFT flow-matching decoder, used by both versions | CC-BY-SA-4.0 |
| `prompts/` | anchor prompt clips (`clip0..3`, `prompts.json` with transcripts and S3 tokens) and female voice prompts (`voice_*.wav`, `voices.json`), all from WAXAL | CC-BY-SA-4.0 |
| `samples/` | example renders (`samples/flash/` for Flash) | CC-BY-SA-4.0 |
| `settings.json` | versions, voices, recommended pairing | Apache-2.0 |
| `infer.py`, `tagger.py`, `app.py` | inference CLI, auto-tagger, Gradio app | Apache-2.0 |
| `flash_lm.py`, `frontend/` | Flash model and synthesis code (training code for reference; the data manifest is not shipped), Flash text front-end (normalisation, diacritics, experimental MT) | Apache-2.0 |
| `prompts/flash_voices_text.json` | transcripts of the voice clips, used as Flash LM prompts | CC-BY-SA-4.0 |
| `code/` | the original training and synthesis code, for reference | Apache-2.0 |
| `requirements.txt`, `cosyvoice_requirements.txt` | dependencies (`cosyvoice_requirements.txt` is CosyVoice's own file at the commit used) | – |
| `assets/` | banner | – |

## Training data and attribution

The model weights are trained on, and the voice prompts are taken from, the datasets below. Several are CC-BY-SA-4.0, so the weights, voice prompts and samples are released under **CC-BY-SA-4.0** (share-alike).

| dataset | used for | licence | attribution |
|---|---|---|---|
| [WAXAL](https://huggingface.co/datasets/google/WaxalNLP), Media Trust TTS subsets (Igbo, Yoruba, Hausa, Nigerian Pidgin) | main training data; the **anchor voice** (Yoruba speaker 4) and the **female prompt voices** (Igbo speaker 5, Yoruba speaker 5) | CC-BY-SA-4.0 (as listed on the dataset card for the Media Trust subsets) | Google and Media Trust, WAXAL (arXiv:2602.02734) |
| [Open.Bible Igbo](https://huggingface.co/datasets/AfriSpeech/open-bible-speech-african) (Biblica® Open Igbo Contemporary Bible™, Audio Edition) | Igbo (Phase 3) | CC-BY-SA-4.0 | Biblica, Inc. via open.bible; Biblica® and Open Igbo Contemporary Bible™ are trademarks of Biblica, Inc. |
| [NaijaS2ST](https://huggingface.co/datasets/McGill-NLP/NaijaS2ST) (Igbo) | Igbo, many native speakers (Phase 3) | CC-BY-4.0 | McGill-NLP, NaijaS2ST (arXiv:2604.16287) |
| [IgboAPI](https://huggingface.co/datasets/nkowaokwu/ibo-dict) (ibo-dict, ibo-dict-expansion) | Igbo sentences (Phase 3) | CC-BY-4.0 | Nkọwa okwu / IgboAPI contributors |
| [IgboSynCorp](https://doi.org/10.7910/dvn/yb9fwk) | natural Igbo speech (Phase 3); Flash adds pseudo-transcripts of its untranscribed audio | CC0-1.0 | University of Ibadan and Afe Babalola University (Lacuna Fund) |
| [FLEURS](https://huggingface.co/datasets/google/fleurs) (English, Hausa, Igbo, Yoruba) | multi-speaker data | CC-BY-4.0 | Google, FLEURS (Conneau et al., 2022) |
| [BibleTTS](https://masakhane-io.github.io/bibleTTS/) (Yoruba, Hausa) | Yoruba and Hausa | CC-BY-SA-4.0 | Meyer et al., BibleTTS (2022); Biblica / open.bible audio |
| [OpenSLR SLR70](https://openslr.org/70/) (Nigerian English) | English | CC-BY-SA-4.0 | Google, crowdsourced Nigerian English |
| [OpenSLR SLR86](https://openslr.org/86/) (Yoruba) | Yoruba | CC-BY-SA-4.0 | Google, crowdsourced Yoruba |
| code-switched pairs | about 9 h of synthetic code-switch training pairs | follows the source clips (CC-BY-SA-4.0 / CC-BY-4.0) | built by concatenating clips from the sets above, following the Google 2019 code-switching recipe (Zhang et al., 2019); no separate Google dataset |
| base model [FunAudioLLM/Fun-CosyVoice3-0.5B-2512](https://huggingface.co/FunAudioLLM/Fun-CosyVoice3-0.5B-2512) | Flash: LM initialisation, flow, vocoder, tokenizers | Apache-2.0 | FunAudioLLM / Alibaba Tongyi Lab, CosyVoice 3 |
| [google/byt5-small](https://huggingface.co/google/byt5-small) | base of the Flash diacritic-restoration model (fine-tuned on transcripts of the datasets above) | Apache-2.0 | Google, ByT5 (Xue et al., 2022) |
| base model [FunAudioLLM/CosyVoice2-0.5B](https://huggingface.co/FunAudioLLM/CosyVoice2-0.5B) | LM initialisation, flow, vocoder, tokenizers | Apache-2.0 | FunAudioLLM / Alibaba Tongyi Lab, CosyVoice 2 (Du et al., 2024) |
| optional [GlotLID](https://huggingface.co/cis-lmu/glotlid) (not bundled) | `--lid glotlid` in the tagger | Apache-2.0 plus notices | Kargaran et al., GlotLID (EMNLP 2023) |

Flash's training data is the licensed set above (the same sources as Phase 3, re-weighted with less Bible narration and slow read speech, plus IgboSynCorp pseudo-transcripts); its DPO pairs are the model's own renders of eval-style prompts, scored automatically. Excluded on licence grounds from every released checkpoint, Flash included: African Voices / WaZoBiaSpeech (its cards ban TTS use), NaijaVoices (non-commercial), and non-commercial TTS weights. The optional MADLAD-400-3B-MT translator (Apache-2.0) is not bundled; NLLB was avoided because it is non-commercial.

## Licence

- **Model weights, voice prompts and samples** (`checkpoints/`, `prompts/`, `samples/`): **[CC-BY-SA-4.0](LICENSE)**. You may use them commercially. You must credit Modular and the datasets listed above, and anything you derive from the weights (fine-tunes, merges) must be shared under CC-BY-SA-4.0 too.
- **Code** (`infer.py`, `tagger.py`, `app.py`, `flash_lm.py`, `frontend/`, `code/`, `settings.json`): **[Apache-2.0](LICENSE-CODE)**.
- The base models, CosyVoice2-0.5B and Fun-CosyVoice3-0.5B-2512 (and byt5-small for the diacritic model), are Apache-2.0; their notices still apply to the parts they contribute.

## Citation

```bibtex
@misc{dynamicaccent2026,
  title  = {Dynamic Accent: Code-switching TTS for Nigerian languages},
  author = {Modular},
  year   = {2026},
  url    = {https://huggingface.co/Modularcomputing/dynamic-accent-preview}
}
```