ILRDF/chatterbox-formosan-all-finetune

Chatterbox Formosan (all dialects) finetune

A finetune of ResembleAI/chatterbox (multilingual T3 t3_mtl23ls_v3) for zero-shot voice-cloning TTS in 42 Formosan dialects of Taiwan.

• Only T3 was finetuned; S3Gen and the VoiceEncoder are the original Chatterbox weights (included for convenience).
• Input: IPA converted with g2p.csv, plus one added tag per dialect.
• Voice cloned from a reference recording (no transcript needed); 24 kHz output.

Files

File	Description
t3_formosan.safetensors	Finetuned T3 (text vocabulary 2454 → 2496: 42 dialect tags)
s3gen.pt, ve.pt	S3Gen and VoiceEncoder, original Chatterbox weights
grapheme_mtl_merged_expanded_v1.json	Chatterbox tokenizer with the 42 dialect tags
dialect_tags.json	lang_code → dialect tag, token id, dialect name
g2p.csv	Grapheme → IPA table per lang_code (used for the training data)
formosan_tts.py	Inference (FormosanTTS)

Usage

pip install git+https://github.com/resemble-ai/chatterbox.git@5de7a54aa4e5e2baadb0182dde554908b48b85c2

import sys

import soundfile as sf
from huggingface_hub import snapshot_download

repo_dir = snapshot_download("ILRDF/chatterbox-formosan-all-finetune")
sys.path.insert(0, repo_dir)
from formosan_tts import FormosanTTS

tts = FormosanTTS(repo_dir, device="cuda")
wav, sr = tts.generate("Nga'ay ho!", "ami-x-skl", "ref.wav")  # 秀姑巒阿美語
sf.write("out.wav", wav, sr)

print(tts.text_to_ipa("Nga'ay ho!", "ami-x-skl"))  # ŋaʡaj ħo!  (the model's input)

• ref_audio: a clean 6–10 s recording of the target voice.
• Characters without a G2P rule (e.g. digits) raise ValueError. Case matters only in Saisiyat (S/s) and Kavalan (R).
• The stock ChatterboxMultilingualTTS.generate() does not work with this model (it accepts only its 23 languages and its text normalization differs).

42 dialects and their tags

lang_code	Dialect	Tag (token id)
ami-x-frng	馬蘭阿美語	[fm_amis_malan] (2456)
ami-x-iams	南勢阿美語	[fm_amis_southern] (2455)
ami-x-pld	恆春阿美語	[fm_amis_hengchun] (2454)
ami-x-pswl	海岸阿美語	[fm_amis_coastal] (2457)
ami-x-skl	秀姑巒阿美語	[fm_amis_xiuguluan] (2458)
bnn-x-bkh	卡群布農語	[fm_bunun_kaqun] (2468)
bnn-x-bnz	巒群布農語	[fm_bunun_luanqun] (2466)
bnn-x-isbk	郡群布農語	[fm_bunun_junqun] (2471)
bnn-x-td	卓群布農語	[fm_bunun_zhuoqun] (2467)
bnn-x-vtn	丹群布農語	[fm_bunun_tanqun] (2465)
ckv	噶瑪蘭語	[fm_kavalan] (2473)
dru-x-kgdv	多納魯凱語	[fm_rukai_dona] (2478)
dru-x-lbw	大武魯凱語	[fm_rukai_dawu] (2480)
dru-x-ngdr	霧臺魯凱語	[fm_rukai_wutai] (2490)
dru-x-opnh	萬山魯凱語	[fm_rukai_wanshan] (2484)
dru-x-tldr	茂林魯凱語	[fm_rukai_maolin] (2481)
dru-x-trmk	東魯凱語	[fm_rukai_eastern] (2482)
pwn-x-kcdsn	東排灣語	[fm_paiwan_eastern] (2470)
pwn-x-pnvn	中排灣語	[fm_paiwan_central] (2469)
pwn-x-vnrn	北排灣語	[fm_paiwan_northern] (2477)
pwn-x-ynvl	南排灣語	[fm_paiwan_southern] (2475)
pyu-x-ksvk	建和卑南語	[fm_puyuma_jianhe] (2474)
pyu-x-ktrp	知本卑南語	[fm_puyuma_zhiben] (2479)
pyu-x-mkzy	西群卑南語	[fm_puyuma_xiqun] (2476)
pyu-x-pym	南王卑南語	[fm_puyuma_nanwang] (2483)
ssf	邵語	[fm_thao] (2492)
sxr	拉阿魯哇語	[fm_saaroa] (2486)
szy	撒奇萊雅語	[fm_sakizaya] (2489)
tao	雅美語	[fm_yami] (2494)
tay-x-cql	四季泰雅語	[fm_atayal_fourseasons] (2459)
tay-x-kls	宜蘭澤敖利泰雅語	[fm_atayal_yilanzeaol] (2463)
tay-x-mtuw	汶水泰雅語	[fm_atayal_wenshui] (2462)
tay-x-plngw	萬大泰雅語	[fm_atayal_wanda] (2461)
tay-x-sql	賽考利克泰雅語	[fm_atayal_sekolik] (2460)
tay-x-sul	澤敖利泰雅語	[fm_atayal_zeaol] (2464)
trv-x-td	都達賽德克語	[fm_seediq_duda] (2488)
trv-x-tgdy	德固達雅賽德克語	[fm_seediq_tegudaya] (2491)
trv-x-trk	德鹿谷賽德克語	[fm_seediq_deluvalley] (2485)
trv-x-truku	太魯閣語	[fm_truku] (2495)
tsu	鄒語	[fm_tsou] (2493)
xnb	卡那卡那富語	[fm_kanakanavu] (2472)
xsy	賽夏語	[fm_saisiyat] (2487)

Training

Data: train splits of seven FormoSpeech datasets; 518,060 utterances / 740.9 h used in training.

Dataset	Revision	Utterances	Hours
formospeech/klokah	85d5ac7e	320,276	466.7
formospeech/ilrdf_dicts	6f33ec5e	89,549	118.1
formospeech/common_voice_formosan	8b831b11	45,554	50.2
formospeech/nchu_pangcah	10885872	24,724	34.7
formospeech/ithuan_formosan	ec3c2cba	15,495	29.6
formospeech/nchc_formosan	3d92966b	11,150	25.4
formospeech/ntu_formosan_corpus	26245016	11,312	16.2

• Removed: utterances ≤ 1.5 s, with < 1 or > 18 phones/s, or without a Mandarin translation (not applied to Common Voice, which has none); Common Voice sentences that also appear in the test set; utterances over 24 s (600 speech tokens) or 200 characters of IPA (4,065 utterances, 29.6 h).
• Training audio was denoised (except ithuan_formosan). The voice prompt is another utterance of the same speaker (by speaker label or voice similarity), redrawn every epoch, with its original (noisy) audio half of the time.
• 4× RTX A5000, 2 epochs (8,096 steps), batch 128, lr 5e-5 (linear, 300 warmup steps), AdamW, bf16. The 42 tag embeddings were initialized to the mean of the existing ones.

Evaluation

• Test set: the eval splits of formospeech/klokah and formospeech/ithuan_formosan (15,167 utterances, 24.8 h, not used in training). The reference is another test utterance of the same speaker.
• CER / WER: ILRDF/whisper-large-v2-formosan-lang-tokens, lower-cased, .,!?; removed, averaged over the 42 dialects. SIM-o: WavLM-large ECAPA-TDNN. UTMOS: utmos22_strong.
• Baseline: ILRDF/f5-tts-formosan-all-finetune-v3. Its G2P cannot convert 473 test sentences, so the tables use the remaining 14,694.

Model	CER	WER	SIM-o	UTMOS
Ground truth (real recordings)	1.88%	7.09%	0.673	2.39
This model	1.83%	9.56%	0.665	2.88
F5-TTS Formosan v3	5.16%	13.99%	0.672	2.27

Per dialect, this model has the lower CER in 40 of 42 dialects of formospeech/klokah and 2 of 6 of formospeech/ithuan_formosan (about 20 utterances each), and the higher UTMOS in all of them. On all 15,167 utterances: CER 1.88%, WER 9.75%, SIM-o 0.666, UTMOS 2.88.

Per-dialect results on formospeech/klokah (CER in %)

lang_code	Dialect	n	CER ours	CER F5	CER GT	SIM-o ours	SIM-o F5	SIM-o GT	UTMOS ours	UTMOS F5	UTMOS GT
ami-x-frng	馬蘭阿美語	348	0.75	2.66	1.64	0.657	0.687	0.678	2.75	2.00	2.25
ami-x-iams	南勢阿美語	293	1.39	7.22	1.31	0.670	0.658	0.668	2.90	2.21	2.28
ami-x-pld	恆春阿美語	277	1.10	4.70	0.99	0.651	0.655	0.657	2.88	2.29	2.35
ami-x-pswl	海岸阿美語	425	1.42	6.60	0.47	0.676	0.679	0.685	2.74	1.99	2.26
ami-x-skl	秀姑巒阿美語	305	1.14	2.71	0.78	0.652	0.649	0.649	2.80	2.21	2.38
bnn-x-bkh	卡群布農語	272	3.11	3.09	2.83	0.627	0.635	0.626	3.03	2.38	2.53
bnn-x-bnz	巒群布農語	236	1.67	2.19	0.95	0.638	0.655	0.661	3.13	2.63	2.67
bnn-x-isbk	郡群布農語	531	1.25	5.51	1.55	0.694	0.703	0.709	2.85	2.12	2.45
bnn-x-td	卓群布農語	307	2.18	3.62	1.98	0.666	0.687	0.684	3.17	2.51	2.77
bnn-x-vtn	丹群布農語	277	2.76	7.20	3.42	0.647	0.668	0.679	2.81	2.27	2.35
ckv	噶瑪蘭語	391	1.30	3.92	0.82	0.690	0.702	0.687	2.72	2.07	2.34
dru-x-kgdv	多納魯凱語	263	3.57	3.57	2.71	0.665	0.671	0.686	3.01	2.58	2.50
dru-x-lbw	大武魯凱語	303	1.90	3.16	2.60	0.680	0.693	0.691	2.91	2.34	2.64
dru-x-ngdr	霧臺魯凱語	454	2.11	9.76	1.27	0.650	0.633	0.654	3.16	2.38	2.68
dru-x-opnh	萬山魯凱語	240	2.52	2.84	1.95	0.630	0.630	0.619	2.62	2.23	2.23
dru-x-tldr	茂林魯凱語	243	6.95	9.86	7.91	0.622	0.627	0.608	3.06	2.51	2.51
dru-x-trmk	東魯凱語	296	2.72	5.06	2.81	0.675	0.674	0.656	2.96	2.40	2.43
pwn-x-kcdsn	東排灣語	280	2.09	5.42	1.83	0.688	0.687	0.697	3.12	2.51	2.63
pwn-x-pnvn	中排灣語	298	1.92	3.40	2.01	0.675	0.686	0.665	2.72	2.18	2.20
pwn-x-vnrn	北排灣語	517	2.08	7.41	2.11	0.664	0.688	0.691	2.91	2.28	2.34
pwn-x-ynvl	南排灣語	298	2.69	6.03	2.55	0.671	0.681	0.677	2.70	2.19	2.30
pyu-x-ksvk	建和卑南語	256	1.25	5.48	0.89	0.694	0.700	0.677	2.72	2.14	2.15
pyu-x-ktrp	知本卑南語	260	1.03	6.08	0.71	0.687	0.708	0.702	2.70	1.94	2.16
pyu-x-mkzy	西群卑南語	257	1.66	4.62	1.08	0.694	0.678	0.703	2.81	2.13	2.27
pyu-x-pym	南王卑南語	447	0.87	4.64	0.93	0.708	0.717	0.722	2.54	1.84	1.91
ssf	邵語	367	1.35	3.23	1.51	0.672	0.671	0.678	3.33	2.84	2.83
sxr	拉阿魯哇語	430	2.05	9.09	1.41	0.680	0.668	0.690	2.95	2.26	2.49
szy	撒奇萊雅語	510	0.65	4.87	1.13	0.689	0.717	0.692	2.79	2.16	2.24
tao	雅美語	453	2.71	5.77	2.28	0.676	0.680	0.658	2.98	2.32	2.49
tay-x-cql	四季泰雅語	277	1.36	4.18	1.16	0.663	0.654	0.689	2.92	2.31	2.38
tay-x-kls	宜蘭澤敖利泰雅語	304	0.93	4.65	3.08	0.644	0.638	0.636	2.95	2.41	2.41
tay-x-mtuw	汶水泰雅語	286	1.96	4.58	1.83	0.639	0.669	0.679	2.58	2.09	2.21
tay-x-plngw	萬大泰雅語	283	1.93	3.07	2.53	0.644	0.655	0.659	3.02	2.25	2.28
tay-x-sql	賽考利克泰雅語	535	1.71	5.12	1.20	0.651	0.659	0.662	2.94	2.39	2.51
tay-x-sul	澤敖利泰雅語	267	1.36	2.93	2.01	0.677	0.685	0.695	2.82	2.26	2.30
trv-x-td	都達賽德克語	270	1.38	4.61	0.77	0.677	0.667	0.685	2.90	2.11	2.31
trv-x-tgdy	德固達雅賽德克語	434	1.08	6.86	1.38	0.660	0.680	0.688	2.91	2.34	2.31
trv-x-trk	德鹿谷賽德克語	273	0.80	7.62	2.07	0.657	0.646	0.630	2.68	2.13	2.24
trv-x-truku	太魯閣語	446	0.84	9.97	0.69	0.647	0.633	0.648	2.62	2.18	2.23
tsu	鄒語	522	2.15	6.65	3.20	0.658	0.675	0.675	3.00	2.50	2.51
xnb	卡那卡那富語	421	0.79	3.56	1.29	0.671	0.681	0.708	2.67	1.96	2.36
xsy	賽夏語	428	1.61	3.57	2.06	0.641	0.675	0.649	2.86	2.31	2.21

Per-dialect results on formospeech/ithuan_formosan (CER in %)

lang_code	Dialect	n	CER ours	CER F5	CER GT	SIM-o ours	SIM-o F5	SIM-o GT	UTMOS ours	UTMOS F5	UTMOS GT
ami-x-skl	秀姑巒阿美語	20	1.36	0.99	1.08	0.798	0.799	0.755	3.75	3.08	3.79
dru-x-lbw	大武魯凱語	20	6.54	9.81	16.60	0.609	0.570	0.668	3.52	3.12	3.56
dru-x-opnh	萬山魯凱語	20	9.16	6.15	5.24	0.700	0.625	0.718	3.98	3.59	4.03
pyu-x-pym	南王卑南語	15	1.58	1.09	3.17	0.812	0.682	0.779	4.04	3.39	4.02
trv-x-tgdy	德固達雅賽德克語	20	1.42	1.64	1.53	0.760	0.782	0.757	3.62	3.31	3.51
trv-x-truku	太魯閣語	19	0.75	0.54	0.43	0.748	0.746	0.726	3.82	3.40	3.32

Gender stability

F5-TTS v3 sometimes switches to the other gender's voice, mostly with question references and short texts. Share of outputs whose median F0 is below 0.70 × the reference's:

Reference	Target text	Utterances	F5-TTS v3	This model
Statement	> 3 words	10,580	1.03%	0.26%
Statement	≤ 3 words	2,051	9.31%	0.93%
Question	> 3 words	1,600	2.69%	0.12%
Question	≤ 3 words	415	12.77%	0.72%

Limitations

• Training utterances are at most 24 s, so long outputs may not stop properly. Synthesize long text sentence by sentence.
• Repeated identical short sentences in one call (e.g. Hai. Hai. Hai. …) may be spoken the wrong number of times.
• SIM-o is slightly below F5-TTS; it is limited by the unchanged S3Gen.
• On formospeech/klokah, CER is highest for dru-x-tldr (6.95%), then dru-x-kgdv and bnn-x-bkh (3.1–3.6%).