**Confucius4-TTS** is a multilingual and cross-lingual **zero-shot text-to-speech engine** released by **NetEase Youdao** under **Apache 2.0**. Its stated goal is compact: *one voice, any language* — clone a speaker from a reference clip, then have that same voice speak any of 14 supported languages without an accent carried over from the source. The GitHub repository holds roughly **734 stars and 71 forks**, with the last push on **27 July 2026**. Weights are mirrored on Hugging Face and ModelScope, and Youdao hosts a public Gradio demo. ## Architecture Confucius4-TTS follows a **speech encoder + LLM** design split into two trainable stages: | Stage | Role | |---|---| | **Text2Semantic (T2S)** | Autoregressive LLM generating semantic token sequences from text plus speaker conditioning | | **Semantic2Acoustic (S2A)** | Flow-matching model converting semantic tokens into mel spectrograms | Three external components are pulled in rather than reimplemented: **Wav2Vec2-BERT** (`facebook/w2v-bert-2.0`) for speaker conditioning and semantic feature extraction, **Amphion's MaskGCT** as the semantic codec, and **CAMPPlus** as the style/speaker encoder. During S2A training the T2S backbone, Wav2Vec2-BERT, and CAMPPlus are all frozen — only the flow-matching model is updated, which keeps the fine-tuning cost of the second stage well below a full retrain. The single most practically important design choice is **unconstrained voice cloning: no reference transcript is required**. In the comparison tables, several competing systems — FishAudio S2, OmniVoice, VoxCPM2, CosyVoice3 — are explicitly marked as needing the reference text alongside the reference audio. Removing that requirement means an arbitrary voice clip can be used directly. ## Benchmarks The README reports four evaluation suites. Lower WER/CER is better; higher SIM is better. **CV3-eval cross-lingual (WER):** | Direction | Confucius4-TTS | F5-TTS† | Spark-TTS | CosyVoice2† | CosyVoice3-0.5B† | |---|---:|---:|---:|---:|---:| | en→zh | **6.71** | 11.60 | 12.40 | 13.50 | 8.48 | | zh→en | **3.19** | 5.57 | 7.36 | 17.10 | 6.83 | | ja→en | **3.44** | – | – | 11.20 | 5.86 | | ko→en | **3.42** | – | – | 13.10 | 18.30 | | ja→zh | 4.93 | – | – | 48.10 | 6.86 | † Requires reference text. The cross-lingual margins are the strongest part of the release. Against CosyVoice2 the gaps are not incremental — 48.10 versus 4.93 WER on ja→zh is a difference in kind, not degree. **MiniMax-Multilingual-Test** shows a similar pattern on lower-resource languages: **Thai WER 1.56 against ElevenLabs' 73.94**, and **Vietnamese 1.61 against 73.42**. Those ElevenLabs figures indicate the commercial system essentially failing to produce intelligible speech on those language pairs rather than merely scoring worse. German lands at **0.47 WER, best in table**. On monolingual **Seed-TTS-eval**, the picture flips: English WER **1.49** trails FishAudio S2 at 0.99 and Qwen3-TTS at 1.24; Chinese CER **0.94** trails FishAudio S2 at 0.54. Confucius4-TTS is a cross-lingual specialist, not a monolingual leader. **Speaker similarity is the consistent weak spot.** Across X-Voice, Seed-TTS-eval and MiniMax, SIM scores sit below OmniVoice and VoxCPM2 in nearly every row — Seed-TTS English SIM 0.70 against VoxCPM2's 0.753, Chinese 0.765 against 0.795. Both of those comparators require reference text, so the trade is legible: Confucius4-TTS gives up some timbre fidelity to drop the transcript requirement. ## Deployment Inference requires **Python 3.10 and CUDA 12.6**. The default path runs the T2S stage through HuggingFace Transformers; a **vLLM backend** accelerates it with PagedAttention and supports both batch and streaming generation via a `--stream` flag. Four entry points ship: `example.py` for CLI synthesis, a Python API, `webui.py` for a Gradio interface, and `server.py` for a FastAPI service exposing `/api/tts` (returns a complete WAV) and `/api/tts/stream` (raw int16-LE PCM chunks, sample rate in the `X-Sample-Rate` header). All accept the reference audio as an HTTP file upload, so client and server need not share a filesystem. The full training pipeline for both T2S and S2A is published, with data supplied as 5-column TSV files. ## Caveats **The vLLM path is brittle.** Only **vLLM 0.16.0 (V1 engine)** is supported, because model registration and `GPUModelRunner` **monkey-patches target v1 engine internals**. The README recommends a separate cloned conda environment specifically because installing vLLM into the base environment can break other packages. Any vLLM upgrade is likely to break this integration. **Environment pins are narrow.** Python 3.10 and CUDA 12.6 are stated requirements, not suggestions. **Speaker similarity trails the field**, as above — for applications where timbre fidelity matters more than transcript-free operation, the reference-text models remain ahead. **Adoption is thin so far.** The Hugging Face repository shows only **14 likes**, and the model card carries no license field even though the GitHub LICENSE file is Apache 2.0. Independent third-party evaluations are not yet available; every number here is vendor-reported. ## Verdict Confucius4-TTS is a focused answer to a specific problem: keeping one speaker's identity intact while switching languages, without needing a transcript of the reference clip. On cross-lingual WER it clears the open field convincingly and, on Thai and Vietnamese, beats a commercial incumbent by margins that suggest the incumbent does not really support those pairs. Teams building multilingual dubbing, localization, or voice-consistent assistants across Asian and European languages should evaluate it directly — the Apache 2.0 license and published training pipeline make adaptation realistic. Teams whose priority is maximum speaker similarity in a single language, or who need a stable high-throughput serving stack today, will find the SIM gap and the vLLM version pin harder to accept.