waxal-benchmarking/voxcpm2-multilang28-full

WAXAL VoxCPM2 — 28-Language African TTS

A full fine-tune of VoxCPM2 (2B params, tokenizer-free, diffusion-AR TTS with voice cloning) on 28 African languages, produced by the WAXAL project at LyngualLabs.

This is a research checkpoint. It is not production-polished — see Known Limitations below before using it for anything beyond experimentation.

Languages

VoxCPM2 has no built-in mechanism for specifying language at inference — the base model infers pronunciation purely from text orthography. We added one: a dedicated <lang:XX> tag per language, registered as a real token in the tokenizer and resized into the text embedding table (mean-initialized, then learned during fine-tuning). The tag is mandatory — always prepend it to the text you want synthesized, e.g.:

<lang:yo> Ẹ káàárọ̀! Báwo ni oorun rí lálẹ́ àná?

Language	Tag	Raw hours	Hours after resampling
Yoruba	<lang:yo>	1038.7	602.4
Swahili	<lang:sw>	1004.1	592.3
Kinyarwanda	<lang:rw>	989.9	588.1
Igbo	<lang:ig>	930.2	570.1
Hausa	<lang:ha>	682.6	488.3
Wolaytta	<lang:wal>	196.6	262.1
Dholuo	<lang:luo>	191.5	258.7
Amharic	<lang:am>	189.2	257.1
Oromo	<lang:om>	189.2	257.1
Sidama	<lang:sid>	188.2	256.4
Tigrinya	<lang:tig>	180.0	250.8
Kikuyu	<lang:kik>	185.8	254.8
Kalenjin	<lang:kal>	153.2	231.3
Malagasy	<lang:mlg>	179.8	250.6
Luganda	<lang:lug>	111.4	197.3
Dagaare	<lang:dagr>	104.3	190.9
Somali	<lang:som>	103.2	189.9
Ewe	<lang:ewe>	103.1	189.8
Ikposo	<lang:ikp>	102.6	189.3
Wolof	<lang:wol>	101.3	188.1
Dagbani	<lang:dagb>	96.3	183.4
Maasai	<lang:maa>	83.7	171.0
Shona	<lang:sna>	83.7	171.0
Akan	<lang:aka>	75.4	162.3
Nyankole	<lang:nya>	48.8	130.6
Masaaba	<lang:mas>	42.7	122.1
Lusoga	<lang:lus>	39.0	116.7
Acholi	<lang:ach>	29.7	101.9

Source data: the AfroNet speech collection (Professor/<language>-speech-data), 7,424 raw hours total.

Training

• Base: openbmb/VoxCPM2, with 28 new <lang:XX> tokens added and the text embedding table resized (mean-initialized rows) before any fine-tuning started — this is an independent run from scratch, not built on top of any earlier narrower fine-tune.
• Mode: full fine-tuning (all parameters trainable except the frozen AudioVAE codec).
• Data balancing: the 28 languages range 35x in size (1038.7h Yoruba vs. 29.7h Acholi). Naive concatenation would starve the smallest languages; full equalization would cause heavy repetition/overfitting on them. We used sqrt-temperature resampling (weight ∝ √hours, renormalized to preserve total volume) — a standard middle-ground technique from multilingual ASR/TTS training. This downsamples the biggest languages to ~58–97% of their raw hours and upsamples the smallest to ~1.4–3.4x, via a combination of subsampling and controlled repetition.
• Scale: 2,929,563 training rows (clips) after resampling, 9,450 held-out validation rows.
• Steps: 44,000 steps (~0.99 epoch over the resampled training set), batch size 3 × grad-accum 11 × 2 GPUs (effective batch 66).
• Hardware: 2× NVIDIA H200, via torchrun DDP.
• Learning rate: 1e-5, 100 warmup steps.

Usage

from huggingface_hub import snapshot_download
from voxcpm import VoxCPM
import soundfile as sf

model_dir = snapshot_download("waxal-benchmarking/voxcpm2-multilang28-full")
model = VoxCPM.from_pretrained(model_dir, load_denoiser=False, optimize=False)

wav = model.generate(
    text="<lang:yo> Ẹ káàárọ̀! Báwo ni oorun rí lálẹ́ àná?",
    cfg_value=2.0,
    inference_timesteps=22,
)
sf.write("out.wav", wav, model.tts_model.sample_rate)

optimize=False is required — torch.compile (the default optimization path) conflicts with some inference environments (e.g. HF ZeroGPU Spaces); disabling it costs some speed but avoids crashes.

Voice cloning works the same way as base VoxCPM2 — pass prompt_wav_path and prompt_text (the literal transcript of what's spoken in the reference clip) alongside the tagged text.

Try it live: WAXAL VoxCPM Multilingual Demo Space.

Evaluation

We built an automated evaluation pipeline rather than relying on casual listening, using:

• ASR-based CER (Character Error Rate) via Meta's Omnilingual ASR (omniASR_CTC_7B, ~1,600-language coverage, verified against this model's own supported-language table — not assumed), comparing transcriptions of generated speech against the ground-truth input text.
• The same ASR run on real human recordings of the same held-out text, as a per-language noise floor — several of these languages get nontrivial CER even on real speech (e.g. Fante in an earlier AfroNet pilot scored CER 0.49–0.57 on real recordings), so a raw TTS CER number is meaningless without that baseline.
• DNSMOS (no-reference naturalness proxy). Caveat: DNSMOS is trained on English/high-resource-language listener ratings, so treat these scores as relative (comparing our own languages/checkpoints) rather than absolute.

Pilot results (5 held-out validation clips per language, no reference voice):

Metric	Result
DNSMOS (naturalness)	2.78–3.41 across all 28 languages, no outliers
Generated-speech CER	Usually lower than real-recording CER for the same ASR (e.g. Swahili: 0.028 generated vs. 0.013 real; Hausa: 0.044 vs. 0.113) — generated audio is clean and consistent, so it's often at least as ASR-decodable as noisy real recordings
Non-terminating generation	A measurable failure mode: in a 5-clip-per-language sample, Malagasy, Sidama, and Somali hit an apparent generation-length ceiling in 3/5 clips; Shona and Swahili in 2/5; several others in 1/5. Confirmed as genuine overrun (not just naturally long sentences) by comparing generated duration against the real recording's duration for the same text.

The non-terminating-generation issue doesn't cleanly correlate with raw data size — Swahili (one of the largest languages here) is affected while several much smaller languages (Hausa, Acholi, Dagbani, Wolof) show none. It's an open issue, not yet root-caused.

Known Limitations

• Voice-design and cross-lingual flexibility are likely reduced. Earlier narrow single-language full fine-tunes on this same base model showed that full-FT on read-speech-only data degrades the base model's text-prompted style/voice-design control and non-trained-language generation. This 28-language run is far more linguistically diverse than those single-language runs, which should help, but it is still exclusively plain narration-style audio — we have not verified how much voice-design capability survives.
• Non-terminating generation (see Evaluation) affects a meaningful minority of generations, worse for some languages (Malagasy, Sidama, Somali) than others.
• Translation/transcript quality varies by language in our own test tooling — some of the lowest-resource languages here (Wolaytta, Sidama in particular) have had machine-translated test sentences that looked suspect on inspection; this is a tooling caveat about our test sentences, not necessarily the model.
• Language coverage via Omnilingual ASR for evaluation purposes was spot-checked, not independently verified for linguistic accuracy of every transcription.

Related

• waxal-benchmarking/voxcpm2-multilang10-full — an earlier, independent fine-tune on just the 10 highest-resource languages from the same collection (Yoruba, Swahili, Kinyarwanda, Igbo, Hausa, Wolaytta, Dholuo, Amharic, Oromo, Sidama), trained the same way.
• AfroNet speech collection — the source training data.