Confucius4-TTS: a Multilingual and Cross-Lingual Zero-Shot TTS Engine

One voice. Any language.

This repository is an aggregate distribution prepared for vLLM-Omni. It includes Confucius4-TTS and its W2V-BERT 2.0, CAMPPlus, and BigVGAN runtime dependencies. No single license covers the entire directory.

Bundled vLLM-Omni Distribution

The dependency weights are co-located so the package can be published and downloaded as one model directory:

Confucius4-TTS/
├── t2s_model.safetensors
├── s2a_model.pt
├── wav2vec2bert_stats.pt
├── tokenizer.*
├── pretrained/
│   ├── w2v-bert-2.0/
│   └── campplus/
└── bigvgan/

Component	Pinned upstream revision	License
Confucius4-TTS	`36b10dd6ff5026e333123bda5f6366de27c8b7a1`	Apache-2.0
W2V-BERT 2.0	`da985ba0987f70aaeb84a80f2851cfac8c697a7b`	MIT
CAMPPlus	`e4b6ede7ce16997aff4ae69fbca1f0175e2afede`	Apache-2.0
BigVGAN	`633ff708ed5b74903e86ff1298cf4a98e921c513`	MIT, with Apache-2.0-derived source files

Release and compliance metadata:

Verify a downloaded copy with:

sha256sum -c SHA256SUMS

When publishing, exclude generated directories such as pretrained/campplus/.cache/; they are local download caches and are not part of this release.

Responsible Use

Voice cloning can reproduce a person's identity and speaking characteristics. Use reference audio only with the speaker's consent or another valid legal basis. Clearly disclose synthetic audio where appropriate, protect reference audio and generated voice data, and do not use this package for impersonation, fraud, harassment, or other harmful activity.

Confucius4-TTS is an advanced LLM-based text-to-speech (TTS) system designed for multilingual and cross-lingual speech synthesis. Built on a speech encoder + large language model (LLM) architecture, Confucius4-TTS enables high-quality speech generation while preserving speaker identity across languages. You can try our online demo at https://confucius4-tts.youdao.com/gradio.

✨ Key Features

14 Languages Supported: Chinese, English, Japanese, Korean, German, French, Spanish, Indonesian, Italian, Thai, Portuguese, Russian, Malay and Vietnamese (more coming soon)
Unconstrained Voice Cloning: No reference transcript required
Cross-Lingual Voice Transfer: Unaccented speech synthesis across 14 languages
Zero-Shot Voice Transfer: Clone voices without additional training
Seamless Emotion Transfer: Clone the feeling, not just the voice
Robust Generalization: Stable performance in real-world multilingual scenarios

With strong cross-lingual generalization, Confucius4-TTS allows users to seamlessly switch languages while keeping the same voice, delivering fluent, natural, and expressive speech.

Installation
Inference
Training
Performance
Citation

🛠 Installation

Requirements

Python 3.10
CUDA 12.6

Setup

Clone the repository:

git clone https://github.com/netease-youdao/Confucius4-TTS.git
cd Confucius4-TTS

Create and activate a conda environment:

conda create -n confuciustts python=3.10 -y
conda activate confuciustts

Install dependencies:

pip install -r requirements.txt

🚀 Inference

Use the provided example.py script for zero-shot TTS synthesis:

python example.py \
    --prompt_wav path/to/reference.wav \
    --text "Your text to synthesize" \
    --lang en \
    --out output.wav \
    --config config/inference_config.yaml

You can also use the Python API directly:

import torch
import torchaudio
from confuciustts.cli.inference import ConfuciusTTS

model = ConfuciusTTS(
    config_path="config/inference_config.yaml",
    device="cuda" if torch.cuda.is_available() else "cpu",
)

audio = model.generate(
    text="Hello, welcome to Confucius4-TTS.",
    lang="en",
    prompt_wav="path/to/reference.wav",
    verbose=True,
)

torchaudio.save("output.wav", audio.cpu(), model.sample_rate)

🚀 Fine-Tuning

Confucius4-TTS follows a "speech encoder + LLM" architecture. The training pipeline covers two modules:

Text2Semantic (T2S): generates semantic token sequences from text and speaker conditioning.
Semantic2Acoustic (S2A): a flow-matching model that converts semantic tokens into mel spectrograms.

1. Prepare Pretrained Models

Download the two external models:

# Wav2Vec2-BERT (speaker conditioning & semantic feature extraction)
huggingface-cli download facebook/w2v-bert-2.0 \
    --local-dir pretrained/w2v-bert-2.0

# Amphion MaskGCT (semantic codec implementation)
git clone https://github.com/open-mmlab/Amphion.git external/Amphion

After downloading, your directory should look like:

checkpoints/
├── t2s_model.safetensors        # pretrained T2S weights
├── s2a_model.pt                 # pretrained S2A weights
├── wav2vec2bert_stats.pt        # semantic feature normalization statistics
├── special_tokens_map.json      # tokenizer files
├── tokenizer.json
├── tokenizer.model
└── tokenizer_config.json
pretrained/
├── w2v-bert-2.0/                # Wav2Vec2-BERT model
└── campplus/
    └── campplus_cn_common.bin   # CAMPPlus speaker encoder checkpoint
external/
└── Amphion/                     # MaskGCT semantic codec implementation

2. Prepare Training Data

Training data is provided as TSV files (tab-separated, no header) with the following 5 columns:

Column	Description
`lang`	Language code (e.g. `zh`, `en`, `ja`)
`wav_path`	Path to the target audio
`norm_text`	Normalized text
`semantic_ids_path`	Pre-extracted semantic tokens (`.npy` file path)
`ref_audio_paths`	Reference audio path(s), comma-separated for multiple

Configure the train/validation paths in config/train_t2s.yaml:

data:
  train_data_path:
    - data/train.tsv
  val_data_path:
    - data/val.tsv

3. Launch T2S Training

Set the pretrained T2S checkpoint path in config/train_t2s.yaml:

paths:
  t2s_checkpoint: checkpoints/t2s_model.safetensors

Single-node training:

python -m confuciustts.cli.train_t2s -c config/train_t2s.yaml

4. Launch S2A Training

Set the checkpoint paths in config/train_s2a.yaml. t2s_checkpoint points to the frozen T2S backbone; s2a_checkpoint is optional and can be used to resume from a pretrained S2A model:

paths:
  t2s_checkpoint: checkpoints/t2s_model.safetensors
  s2a_checkpoint: checkpoints/s2a_model.pt   # optional: resume from pretrained S2A

Single-node training:

python -m confuciustts.cli.train_s2a -c config/train_s2a.yaml

During S2A training, the T2S model, speaker encoder (Wav2Vec2-BERT), and style encoder (CAMPPlus) are all frozen. Only the flow-matching S2A model is trained.

📊 Performance

Confucius4-TTS achieves competitive results on multilingual and cross-lingual zero-shot TTS benchmarks, with strong intelligibility and speaker similarity across multiple languages.

Lower is better for WER/CER (↓), and higher is better for SIM (↑).

CV3-eval Cross-lingual

CV3-eval Cross-lingual Results (click to expand)

Direction	Metric	Confucius4-TTS	CosyVoice2†	CosyVoice3-0.5B†	CosyVoice3-1.5B†	OmniVoice†	VoxCPM2
en→zh	CER↓	6.16	13.50	8.48	8.01	6.53	6.29
ja→zh	CER↓	4.87	48.10	6.86	6.78	52.64	4.20
ko→zh	CER↓	1.28	7.70	5.24	3.30	1.71	1.20
zh→en	WER↓	3.19	17.10	6.83	5.39	3.72	3.84
ja→en	WER↓	3.44	11.20	5.86	5.94	5.25	4.10
ko→en	WER↓	3.42	13.10	18.30	13.70	3.91	5.69

† Requires reference text.

X-Voice Benchmark

X-Voice Cross-lingual Results (click to expand)

Direction	Metric	Confucius4-TTS	X-Voice	IndexTTS2	OmniVoice†	VoxCPM2
de→zh	CER↓	2.86	3.07	3.46	7.79	3.62
en→zh	CER↓	3.21	3.06	3.78	3.30	3.35
fr→zh	CER↓	2.70	3.01	3.53	8.16	3.75
ja→zh	CER↓	3.50	3.39	4.11	60.88	4.53
ko→zh	CER↓	2.86	3.13	2.90	7.35	6.33
th→zh	CER↓	2.82	2.79	3.08	2.85	5.96
vi→zh	CER↓	2.75	2.78	2.98	6.59	3.65

† Requires reference text.

Seed-TTS-eval

Seed-TTS-eval English & Chinese Zero-shot Results (click to expand)

System	English WER↓	English SIM↑	Chinese CER↓	Chinese SIM↑
Confucius4-TTS	1.49	0.700	0.94	0.765
Confucius4-TTS (Continuation)†	1.68	0.715	1.15	0.766
Seed-TTS†	2.25	0.762	1.12	0.796
Qwen3-TTS†	1.24	0.714	0.77	0.770
FishAudio S2†	1.79	0.643	0.98	0.737
OmniVoice†	1.62	0.740	0.87	0.777
VoxCPM2†	1.70	0.752	0.97	0.793
X-Voice	1.91	0.627	1.47	0.746

† Requires reference text.

MiniMax-MLS-Test

MiniMax-MLS-Test Results (click to expand)

Language	Metric	Confucius4-TTS	Confucius4-TTS (Continuation)†	MiniMax-Speech	ElevenLabs	Qwen3-TTS†	FishAudio S2†	OmniVoice†	VoxCPM2†
German	WER↓	0.47	0.68	1.91	0.57	1.24	0.55	0.80	1.12
	SIM↑	0.775	0.777	0.733	0.614	0.768	0.706	0.804	0.805
French	WER↓	3.66	4.87	4.10	5.22	2.86	3.90	3.58	3.42
	SIM↑	0.723	0.755	0.628	0.535	0.716	0.658	0.776	0.738
Indonesian	WER↓	1.12	1.41	1.24	1.06	–	2.93	1.34	1.17
	SIM↑	0.765	0.767	0.729	0.660	–	0.736	0.777	0.795
Korean	CER↓	1.84	2.50	1.75	1.87	1.76	1.62	2.66	3.34
	SIM↑	0.812	0.824	0.776	0.700	0.790	0.742	0.831	0.837
Thai	WER↓	1.56	2.47	2.70	73.94	–	6.66	2.93	2.19
	SIM↑	0.773	0.807	0.800	0.588	–	0.749	0.847	0.841
Japanese	CER↓	4.14	4.05	3.52	10.65	3.82	3.52	3.59	3.51
	SIM↑	0.788	0.806	0.776	0.738	0.771	0.753	0.821	0.825
Vietnamese	WER↓	1.61	1.59	0.88	73.42	–	14.11	0.95	4.19
	SIM↑	0.751	0.753	0.743	0.369	–	0.693	0.775	0.793
Italian	WER↓	1.30	3.26	1.54	1.74	0.95	1.49	1.20	1.34
	SIM↑	0.787	0.791	0.699	0.579	0.752	0.764	0.813	0.779
Portuguese	WER↓	2.48	3.91	1.88	1.33	1.53	1.57	1.83	1.71
	SIM↑	0.796	0.801	0.805	0.711	0.805	0.777	0.866	0.842
Spanish	WER↓	1.02	1.65	1.03	1.08	1.13	0.95	0.81	1.32
	SIM↑	0.778	0.794	0.762	0.615	0.814	0.734	0.814	0.829
Russian	WER↓	4.64	5.42	4.28	3.88	3.21	4.24	4.63	4.53
	SIM↑	0.787	0.796	0.761	0.675	0.784	0.768	0.784	0.807

† Requires reference text.

Acknowledgements

Confucius4-TTS builds on the following open-source projects:

Qwen3-TTS — Speaker encoder (ECAPA-TDNN) and text embedding projector architectures
CosyVoice — Text normalization pipeline
Amphion / MaskGCT — Semantic codec implementation
w2v-BERT 2.0 — Semantic feature extraction and speaker conditioning
Seed-VC — Flow matching architecture reference
BigVGAN — High-fidelity neural vocoder for mel-spectrogram to waveform synthesis

Citation

If you find Confucius4-TTS useful in your research or project, please consider citing:

@misc{wang2026confucius4tts,
  title         = {Confucius4-TTS: Transcript-Free Cross-Lingual Zero-Shot TTS with a Learnable Speaker Encoder},
  author        = {Huaxuan Wang and Huimin Wang and Ruiyu Zhang and Yingjie Li and Yitao Duan},
  year          = {2026},
  eprint        = {2608.11650},
  archivePrefix = {arXiv},
  primaryClass  = {cs.SD},
  url           = {https://arxiv.org/abs/2608.11650}
}

Downloads last month: -; Downloads are not tracked for this model. How to track

Paper for linchaoying/Confucius4-TTS

Confucius4-TTS: Transcript-Free Cross-Lingual Zero-Shot TTS with a Learnable Speaker Encoder

Paper • 2608.11650 • Published Aug 12 • 1

linchaoying
/

Confucius4-TTS

Confucius4-TTS: a Multilingual and Cross-Lingual Zero-Shot TTS Engine

Bundled vLLM-Omni Distribution

Responsible Use

Contents

🛠 Installation

Requirements

Setup

🚀 Inference

🚀 Fine-Tuning

1. Prepare Pretrained Models

2. Prepare Training Data

3. Launch T2S Training

4. Launch S2A Training

📊 Performance

CV3-eval Cross-lingual

X-Voice Benchmark

Seed-TTS-eval

MiniMax-MLS-Test

Acknowledgements

Citation

Paper for linchaoying/Confucius4-TTS

Confucius4-TTS: Transcript-Free Cross-Lingual Zero-Shot TTS with a Learnable Speaker Encoder