netease-youdao/Confucius4-TTS

Confucius4-TTS: a Multilingual and Cross-Lingual Zero-Shot TTS Engine

One voice. Any language.

                                  

Confucius4-TTS is an advanced LLM-based text-to-speech (TTS) system designed for multilingual and cross-lingual speech synthesis. Built on a speech encoder + large language model (LLM) architecture, Confucius4-TTS enables high-quality speech generation while preserving speaker identity across languages. You can try our online demo at https://confucius4-tts.youdao.com/gradio.

✨ Key Features

• 14 Languages Supported: Chinese, English, Japanese, Korean, German, French, Spanish, Indonesian, Italian, Thai, Portuguese, Russian, Malay and Vietnamese (more coming soon)
• Unconstrained Voice Cloning: No reference transcript required
• Cross-Lingual Voice Transfer: Unaccented speech synthesis across 14 languages
• Zero-Shot Voice Transfer: Clone voices without additional training
• Seamless Emotion Transfer: Clone the feeling, not just the voice
• Robust Generalization: Stable performance in real-world multilingual scenarios

With strong cross-lingual generalization, Confucius4-TTS allows users to seamlessly switch languages while keeping the same voice, delivering fluent, natural, and expressive speech.

Contents

• Installation
• Inference
• Training
• Performance
• Citation

🛠 Installation

Requirements

• Python 3.10
• CUDA 12.6

Setup

1. Clone the repository:

git clone https://github.com/netease-youdao/Confucius4-TTS.git
cd Confucius4-TTS

2. Create and activate a conda environment:

conda create -n confuciustts python=3.10 -y
conda activate confuciustts

3. Install dependencies:

pip install -r requirements.txt

🚀 Inference

For environments with restricted access to HuggingFace, set a mirror endpoint before running:

export HF_ENDPOINT=https://hf-mirror.com

Basic Usage

Use the provided example.py script for zero-shot TTS synthesis:

python example.py \
    --prompt_wav path/to/reference.wav \
    --text "Hello, this is a test of zero-shot voice cloning." \
    --lang en \
    --out output.wav \
    --config config/inference_config.yaml

You can also use the Python API directly:

import torch
import torchaudio
from confuciustts.cli.inference import ConfuciusTTS

model = ConfuciusTTS(
    config_path="config/inference_config.yaml",
    device="cuda" if torch.cuda.is_available() else "cpu",
)

audio = model.generate(
    text="Hello, welcome to Confucius4-TTS.",
    lang="en",
    prompt_wav="path/to/reference.wav",
    verbose=True,
)

torchaudio.save("output.wav", audio.cpu(), model.sample_rate)

vLLM Usage

The basic path above runs the Text2Semantic (T2S) autoregressive stage with HuggingFace Transformers. For faster T2S generation you can switch to the vLLM backend (example_vllm.py), which accelerates the LLM stage with PagedAttention.

We currently support vLLM 0.16.0 (V1 engine). Older vLLM versions have not been tested and may not work, because the model registration and GPUModelRunner monkey-patches target the v1 engine internals.

⚠️ vllm has specific requirements on GPU architecture / driver / CUDA. It may not run on your hardware, and installing it into the base env can break other packages. We recommend creating a separate conda environment cloned from the basic one, then installing only the vLLM add-on dependencies.

# 1. Clone a dedicated env from the basic env (so confuciustts deps are inherited)
conda create -n confuciustts_vllm --clone confuciustts
conda activate confuciustts_vllm

# 2. Install the vLLM add-on dependencies
pip install -r requirements_vllm_add.txt

# 3. Run the vLLM example (models auto-download on first run)
python example_vllm.py \
    --prompt_wav path/to/reference.wav \
    --text "要合成的文本" \
    --lang zh \
    --out output_vllm.wav

Both non-streaming and streaming generation are supported via the --stream flag:

# Non-streaming: synthesize the whole utterance and save one .wav
python example_vllm.py \
    --prompt_wav path/to/reference.wav \
    --text "要合成的文本" \
    --lang zh \
    --out output_vllm.wav

# Streaming: generate chunk-by-chunk (model.generate_stream), concatenated into one .wav here
python example_vllm.py \
    --prompt_wav path/to/reference.wav \
    --text "要合成的文本" \
    --lang zh \
    --out output_vllm_stream.wav \
    --stream

Web Demo

A Gradio web UI is provided for interactive zero-shot voice cloning in the browser. Upload a reference audio, type the text, pick a language, and click Generate.

python webui.py --port 7860

Then open http://<server-ip>:7860 in your browser. The reference audio is uploaded to the server via HTTP, so the client browser does not need to share a filesystem with the server.

Online Server

For programmatic access (e.g. calling TTS from another service/machine), use the FastAPI server. It exposes both non-streaming and streaming endpoints and accepts the reference audio as an HTTP file upload, so the client and server do not need to be on the same host.

python server.py --port 8000

Request form fields: text, lang (e.g. zh, en, ja), reference (audio file upload).

# Non-streaming → out.wav
curl -F "text=要合成的文本" -F "lang=zh" -F "reference=@path/to/reference.wav" \
     http://localhost:8000/api/tts -o out.wav

# Streaming → raw int16 PCM (22050 Hz, mono), playable per X-Sample-Rate header
curl -F "text=要合成的文本" -F "lang=zh" -F "reference=@path/to/reference.wav" \
     http://localhost:8000/api/tts/stream -o out.pcm

🚀 Fine-Tuning

Confucius4-TTS follows a "speech encoder + LLM" architecture. The training pipeline covers two modules:

• Text2Semantic (T2S): generates semantic token sequences from text and speaker conditioning.
• Semantic2Acoustic (S2A): a flow-matching model that converts semantic tokens into mel spectrograms.

1. Prepare Pretrained Models

Download the two external models:

# Wav2Vec2-BERT (speaker conditioning & semantic feature extraction)
huggingface-cli download facebook/w2v-bert-2.0 \
    --local-dir pretrained/w2v-bert-2.0

# Amphion MaskGCT (semantic codec implementation)
git clone https://github.com/open-mmlab/Amphion.git external/Amphion

After downloading, your directory should look like:

checkpoints/
├── t2s_model.safetensors        # pretrained T2S weights
├── s2a_model.pt                 # pretrained S2A weights
├── wav2vec2bert_stats.pt        # semantic feature normalization statistics
├── special_tokens_map.json      # tokenizer files
├── tokenizer.json
├── tokenizer.model
└── tokenizer_config.json
pretrained/
├── w2v-bert-2.0/                # Wav2Vec2-BERT model
└── campplus/
    └── campplus_cn_common.bin   # CAMPPlus speaker encoder checkpoint
external/
└── Amphion/                     # MaskGCT semantic codec implementation

2. Prepare Training Data

Training data is provided as TSV files (tab-separated, no header) with the following 5 columns:

Configure the train/validation paths in config/train_t2s.yaml:

data:
  train_data_path:
    - data/train.tsv
  val_data_path:
    - data/val.tsv

3. Launch T2S Training

Set the pretrained T2S checkpoint path in config/train_t2s.yaml:

paths:
  t2s_checkpoint: checkpoints/t2s_model.safetensors

Single-node training:

python -m confuciustts.cli.train_t2s -c config/train_t2s.yaml

4. Launch S2A Training

Set the checkpoint paths in config/train_s2a.yaml. t2s_checkpoint points to the frozen T2S backbone; s2a_checkpoint is optional and can be used to resume from a pretrained S2A model:

paths:
  t2s_checkpoint: checkpoints/t2s_model.safetensors
  s2a_checkpoint: checkpoints/s2a_model.pt   # optional: resume from pretrained S2A

Single-node training:

python -m confuciustts.cli.train_s2a -c config/train_s2a.yaml

During S2A training, the T2S model, speaker encoder (Wav2Vec2-BERT), and style encoder (CAMPPlus) are all frozen. Only the flow-matching S2A model is trained.

📊 Performance

Confucius4-TTS achieves competitive results on multilingual and cross-lingual zero-shot TTS benchmarks, with strong intelligibility and speaker similarity across multiple languages.

Lower is better for WER/CER (↓), and higher is better for SIM (↑).

CV3-eval Cross-lingual

X-Voice Benchmark

Seed-TTS-eval

MiniMax-MLS-Test

Acknowledgements

Confucius4-TTS builds on the following open-source projects:

• Qwen3-TTS — Speaker encoder (ECAPA-TDNN) and text embedding projector architectures
• CosyVoice — Text normalization pipeline
• Amphion / MaskGCT — Semantic codec implementation
• w2v-BERT 2.0 — Semantic feature extraction and speaker conditioning
• Seed-VC — Flow matching architecture reference
• BigVGAN — High-fidelity neural vocoder for mel-spectrogram to waveform synthesis

Citation

If you find Confucius4-TTS useful in your research or project, please consider citing:

@misc{wang2026confucius4tts,
  title         = {Confucius4-TTS: Transcript-Free Cross-Lingual Zero-Shot TTS with a Learnable Speaker Encoder},
  author        = {Huaxuan Wang and Huimin Wang and Ruiyu Zhang and Yingjie Li and Yitao Duan},
  year          = {2026},
  eprint        = {2608.11650},
  archivePrefix = {arXiv},
  primaryClass  = {cs.SD},
  url           = {https://arxiv.org/abs/2608.11650}
}