README.md

metadata

license: apache-2.0
language:
  - en
  - pcm
  - yo
  - ig
  - ha
base_model: FunAudioLLM/CosyVoice2-0.5B
pipeline_tag: text-to-speech
tags:
  - text-to-speech
  - code-switching
  - nigerian-languages
  - cosyvoice2
  - research-preview

Dynamic Accent (Phase 2, research preview)

Nigerian multilingual, code-switching text-to-speech in one consistent voice: English, Nigerian Pidgin, Yoruba, Igbo and Hausa, mixed freely inside one sentence.

It is built on FunAudioLLM/CosyVoice2-0.5B and adds:

• per-token language embeddings, a speaker token and an adversarial (GRL) speaker classifier on the LM, fine-tuned on about 120 h of Nigerian-language and code-switched speech with pause-normalised data (pause profile B);
• in-context anchor prompts at the LM and the flow, so prosody and pause style follow the anchor voice (WAXAL Yoruba speaker 4);
• a flow-matching decoder fine-tuned on the anchor voice (anchor SFT) for tone and pitch detail.

This repo is a private research preview for testing. It is not a production model.

Inference defaults to prosody v2: clause-level generation with anchor-matched pause gaps, and a 1.05 pitch-preserving speed-up. See the section below.

What is in the repo

path	what it is
checkpoints/p2_lm_B/best.pt	Dynamic Accent LM (CosyVoice2 Qwen2 LM + language/speaker embeddings), pause profile B, 4,000 steps, best validation CE
checkpoints/p2_lm_B/spk2id.json	speaker-token map (the anchor is waxal_yor_4)
checkpoints/p2_flow_sft/best.pt	anchor-SFT flow-matching decoder (replaces the CosyVoice2 flow weights)
prompts/clip{0..3}_*.wav, prompts/prompts.json	the 4 anchor prompt clips (24 kHz) with their transcripts and S3 speech tokens. The final system uses clip 0
infer.py	self-contained inference: the same path as the Phase 2 final samples
code/	the original training and synthesis code (train_b.py, train_b2.py, da/) for reference
cosyvoice_requirements.txt	CosyVoice's own requirements file, at the commit used
samples/	example renders

Everything else comes from the stock base model FunAudioLLM/CosyVoice2-0.5B (Qwen tokenizer CosyVoice-BlankEN, llm.pt, hift.pt vocoder, campplus.onnx, speech_tokenizer_v2.onnx, cosyvoice2.yaml). Nothing in the base needs changing; infer.py downloads it on first run.

Setup (single GPU, e.g. Colab T4 or A100)

# 1. CosyVoice code (the commit the model was built and tested on)
git clone --recursive https://github.com/FunAudioLLM/CosyVoice.git
cd CosyVoice && git checkout 074ca6dc9e80a2f424f1f74b48bdd7d3fea531cc && git submodule update --init --recursive && cd ..

# 2. This repo (private: log in with a token that can read Modularcomputing)
huggingface-cli login
huggingface-cli download Modularcomputing/dynamic-accent-preview --local-dir dynamic-accent-preview

# 3. Python deps (Python 3.10; on Colab keep the preinstalled torch/torchaudio)
pip install -r dynamic-accent-preview/requirements.txt

The base model downloads to dynamic-accent-preview/pretrained/CosyVoice2-0.5B the first time you run infer.py. You can also point --model_dir (or COSYVOICE2_DIR) at an existing copy. Point --cosyvoice_dir (or COSYVOICE_REPO) at your CosyVoice clone if it is not inside the repo folder.

The 0.5B model should fit comfortably on a 16 GB T4. This build was tested on an H100 only, not yet on Colab.

Usage

cd dynamic-accent-preview
python infer.py --cosyvoice_dir ../CosyVoice \
  --text "<eng>this is dynamic accent,</eng> <pcm>e dey sound nice abi?</pcm>" \
  --seed 0 --out out.wav                     # defaults: --pause_mode clause --speed 1.05
# Phase 2 final behaviour: add  --pause_mode model --speed 1.0

• Language tags: wrap each span in <eng>, <pcm>, <yor>, <ibo> or <hau>. Untagged text is read as English. Keep punctuation inside the span it belongs to.
• --seed: the final system is a single sample at seed 0. Try seeds 1 and 2 (or more) and keep the take you like. Best-of-N picking lowered CER by about a third in evaluation.
• --pause_mode clause (default, prosody v2): the text is cut at , ; : . ? ! and at language switches. Each chunk is generated separately and its edge silence is trimmed (energy VAD, 30 ms kept). Chunks are then joined with set gaps taken from the real anchor's pause statistics: 0.40 s ± 0.05 at commas and ;:, 0.60 s at sentence ends, and 0.12 s at an unpunctuated language switch, with 10 ms fades. A one-word chunk is merged into the next chunk of the same language.
• --speed (default 1.05): a pitch-preserving tempo factor (WSOLA via audiotsm) applied to the speech only, so the gaps stay as set. The allowed range is 1.0–1.2.
• --pause_mode model --speed 1.0 reproduces the Phase 2 final system: model-generated pauses, sentence-by-sentence generation, internal silences capped at 0.6 s, and 0.6 s between sentences. Both modes use anchor prompt clip 0 at the LM and flow (--prompt_clip 0).
• --dtype: the LM ran under bf16 autocast. T4 has no bf16, so auto uses fp16 there. fp16 has not been tested yet; if a T4 run sounds wrong, compare against an A100 (bf16) run.
• Determinism: generation is not bit-reproducible across runs on GPU. Sampling under bf16 can pick a different token on near-ties, so the same seed can give a slightly different take.

From Python:

from infer import DynamicAccent
tts = DynamicAccent("pretrained/CosyVoice2-0.5B", "../CosyVoice",
                    "checkpoints/p2_lm_B/best.pt", "checkpoints/p2_flow_sft/best.pt", "prompts/prompts.json")
wav = tts.say("<hau>Ina kwana, yaya aiki?</hau> <eng>The meeting has been moved to Friday morning.</eng>",
              seed=0, pause_mode="clause", speed=1.05)
import soundfile as sf; sf.write("hau_eng.wav", wav, tts.sr)

Example code-switched inputs

<eng>this is dynamic accent,</eng> <pcm>e dey sound nice abi?</pcm>
<yor>Ẹ káàárọ̀ o, ṣé dáadáa ni?</yor> <eng>I will send you the report this afternoon.</eng>
<ibo>Kedu ka ị mere?</ibo> <pcm>Abeg no vex, I go call you for evening.</pcm>
<hau>Ina kwana, yaya aiki?</hau> <eng>The meeting has been moved to Friday morning.</eng>
<ibo>Ndewo nụ, ụmụnna m.</ibo> <pcm>Una well done o, I know say the road no easy.</pcm> <eng>We have three things to discuss this evening.</eng>
<hau>Barka da yamma, yaya kuke?</hau> <eng>The weather report says it will rain tomorrow morning.</eng> <pcm>Abeg no forget o, the rain fit heavy well well.</pcm>

The Yoruba, Igbo and Hausa example lines are hand-written and have not been checked by native speakers.

Samples

• samples/pcm_eng_dynamic_accent_seed{0,1,2}.mp3: "this is dynamic accent, e dey sound nice abi?" (English then Pidgin), seeds 0, 1 and 2. These are Phase 2 final renders (--pause_mode model --speed 1.0).
• samples/prosody_v2/*.mp3: the same custom sentence plus para_ibo_pcm_eng, para_yor_eng, para_hau_eng_pcm, ibo_000 and yor_000, rendered with the defaults (--pause_mode clause --speed 1.05, seed 0).

Prosody v2 (inference-time pause and pace fix)

Speaking rate is articulation rate: syllables per second of speech, with pauses of 0.15 s or more excluded. It uses the same syllable counter for every source. Reference speech is 150 clips per source. The system rate comes from the Phase 2 final renders of the eval set.

	anchor (yor)	yor	ibo	hau	pcm	eng
reference speech (syl/s)	4.45	4.78 (WAXAL, other speakers)	4.73	5.71	3.87	4.53 (FLEURS)
Phase 2 final system (syl/s)	4.20 (yor)	4.20	4.50	3.98	3.63	4.77
reference ÷ system	1.06	1.14	1.05	1.43	1.06	0.95

The median ratio is 1.06, so the default --speed is 1.05 (rounded to 0.05 and clamped to 1.0–1.2). Hausa is the outlier. Its vowel-based syllable count is the roughest of the five.

Before and after on the six sample items (custom sentence, 3 paragraphs, ibo_000, yor_000), seed 0. Pauses are measured at forced-aligned word boundaries with the Phase 2 report harness. The anchor row is real WAXAL recordings. With only 6 items, read these as a direction, not a result.

	comma pause (s)	sentence pause (s)	unpunctuated pause rate (>0.25 s)	articulation rate (syl/s)	para CER	para CER (no tones)	all-6 CER	SIM
real anchor	0.417	0.543	0.137	4.45	–	–	–	ceiling ≈0.84
before: Phase 2 final (model, 1.0)	0.339	0.488	0.057	4.26	0.175	0.124	0.175	0.814
clause, speed 1.0	0.339	0.526	0.095	3.95	0.179	0.123	0.166	0.775
after: clause, speed 1.05 (default)	0.275 (≈0.37 *)	0.550	0.086	4.18	0.193	0.143	0.207	0.750
clause, speed 1.10	0.315	0.550	0.067	4.37	0.244	0.194	0.224	0.748
model pauses, speed 1.05	0.235	0.441	0.029	4.35	0.189	0.149	0.188	0.788

* At two commas that follow the one-letter Pidgin particle "o," the aligner moves the inserted gap to the boundary before "o". This reads as a 0.0 s comma plus one unpunctuated pause. The other comma gaps in clause mode measure 0.37–0.39 s, against the 0.40 s target.

• Pauses: clause mode moves comma and sentence pauses and the phrase-break rate toward the anchor's numbers. Sentence pauses go from 0.49 to 0.55 s against the anchor's 0.54 s.
• Pace: generating short chunks one at a time makes speech about 7% slower (4.26 to 3.95 syl/s). So 1.05 only brings the rate back to about where the final system was, and it is still below the anchor.
• Cost: WSOLA speed-up and chunking each lower ECAPA similarity (0.81 to 0.75–0.79). At 1.10, paragraph CER rises clearly (0.175 to 0.244). At 1.05 the rise is small (0.175 to 0.193), and mixed across paragraphs: Hausa got better, Igbo and Yoruba got worse. A blind listening check should decide between 1.0 and 1.05.

Evaluation (Phase 2 report)

CER is ASR round-trip: MMS-1b-all for ibo/hau/yor, Whisper-large-v3-turbo for eng and pcm, and min(MMS, Whisper) for code-switched (CS) items and paragraphs. "No tones" strips the diacritics. SIM is ECAPA cosine to the anchor centroid; the ceiling is about 0.84 on real held-out anchor clips. Pidgin CER is limited by the judge: real Pidgin recordings score 0.43. Full eval set: 30 items per language + 9 CS sentences + 3 paragraphs = 162 items, single sample (seed 0, anchor prompt).

system	ibo CER	hau CER	yor CER (no tones)	pcm CER	eng CER	CS CER	para CER	SIM to anchor (mean of 5 langs)	SIM spread	CS half-vs-half SIM
Phase 1 Arm B (smoke)	0.212	0.244	0.407 (0.294)	0.412	0.035	0.201	0.195	0.804	0.030	0.649
p2_lm (profile A) + SFT flow	0.199	0.183	0.223 (0.140)	0.190	0.038	0.184	0.176	0.769	0.069	0.634
p2_lm_B (profile B) + SFT flow (this release)	0.188	0.188	0.224 (0.137)	0.174	0.034	0.167	0.173	0.782	0.039	0.642

Pause and pitch (energy VAD on forced-aligned word boundaries):

system	int. pause p90 (s)	pauses / s	spurious pause rate	comma pause (s)	pause-length W1 vs anchor	para: sentence pause (s)	para: pause p90 (s)	F0 std (st)	F0 range (st)	yor tone-direction acc (pairs)
real anchor recordings (WAXAL val+test, raw)	1.071	0.624	0.137	0.417	0	0.543	–	3.152	10.173	0.750 (2990)
Phase 1 Arm B (smoke)	1.334	0.584	0.044	0.245	0.147	0.315	1.035	3.058	9.999	0.777 (251)
p2_lm (profile A) + SFT flow	0.414	0.328	0.008	0.263	0.330	0.509	0.547	3.038	10.069	0.752 (254)
p2_lm_B (profile B) + SFT flow (this release)	0.460	0.288	0.087	0.251	0.262	0.488	0.531	3.057	10.134	0.749 (255)

Zero-shot baselines on the 62-item stage subset (for context): CosyVoice2 zero-shot ibo/hau/yor CER 0.689 / 0.281 / 0.852, and CosyVoice3 zero-shot 0.629 / 0.197 / 0.717, against 0.173 / 0.220 / 0.226 for Phase 2 (p2_lm + SFT flow).

Known limitations

• In model mode, comma pauses are short (0.25 s on the full eval set, against the anchor's 0.42 s). clause mode sets them directly (see Prosody v2).
• Speaker similarity is 0.78 against Phase 1's 0.80. English and Pidgin drift the most from the anchor.
• English after a Nigerian-language span keeps a strong local accent.
• Pidgin has only about 6 h of training data and no reliable ASR judge.
• Yoruba tone quality needs human listening; the automatic tone metric is saturated.
• Naturalness has not been rated by native speakers yet (no MOS / A/B).

Training data

The training data includes CC-BY-SA sources. Datasets used:

• WAXAL (Yoruba, Igbo, Hausa, Nigerian Pidgin; includes the anchor voice, Yoruba speaker 4), CC-BY-SA-4.0
• BibleTTS (Yoruba and Hausa), CC-BY-SA
• FLEURS (English, Hausa, Igbo, Yoruba)
• OpenSLR SLR70 (Nigerian English)
• OpenSLR SLR86 (Yoruba)
• 4,000 code-switched pairs built from the sets above

Licence

Apache-2.0 (see LICENSE). The base model, CosyVoice2-0.5B, is Apache-2.0.