Open Source
Explore the latest AI open-source projects from GitHub and HuggingFace.
Explore the latest AI open-source projects from GitHub and HuggingFace.
Confucius4-TTS is a multilingual and cross-lingual zero-shot text-to-speech engine released by NetEase Youdao under Apache 2.0. Its stated goal is compact: one voice, any language — clone a speaker from a reference clip, then have that same voice speak any of 14 supported languages without an accent carried over from the source.
The GitHub repository holds roughly 734 stars and 71 forks, with the last push on 27 July 2026. Weights are mirrored on Hugging Face and ModelScope, and Youdao hosts a public Gradio demo.
Confucius4-TTS follows a speech encoder + LLM design split into two trainable stages:
| Stage | Role |
|---|---|
| Text2Semantic (T2S) | Autoregressive LLM generating semantic token sequences from text plus speaker conditioning |
| Semantic2Acoustic (S2A) | Flow-matching model converting semantic tokens into mel spectrograms |
Three external components are pulled in rather than reimplemented: Wav2Vec2-BERT (facebook/w2v-bert-2.0) for speaker conditioning and semantic feature extraction, Amphion's MaskGCT as the semantic codec, and CAMPPlus as the style/speaker encoder. During S2A training the T2S backbone, Wav2Vec2-BERT, and CAMPPlus are all frozen — only the flow-matching model is updated, which keeps the fine-tuning cost of the second stage well below a full retrain.
The single most practically important design choice is unconstrained voice cloning: no reference transcript is required. In the comparison tables, several competing systems — FishAudio S2, OmniVoice, VoxCPM2, CosyVoice3 — are explicitly marked as needing the reference text alongside the reference audio. Removing that requirement means an arbitrary voice clip can be used directly.
The README reports four evaluation suites. Lower WER/CER is better; higher SIM is better.
CV3-eval cross-lingual (WER):
| Direction | Confucius4-TTS | F5-TTS† | Spark-TTS | CosyVoice2† | CosyVoice3-0.5B† |
|---|---|---|---|---|---|
| en→zh | 6.71 | 11.60 | 12.40 | 13.50 | 8.48 |
| zh→en | 3.19 | 5.57 | 7.36 | 17.10 | 6.83 |
| ja→en | 3.44 | – | – | 11.20 | 5.86 |
| ko→en | 3.42 | – | – | 13.10 | 18.30 |
| ja→zh | 4.93 | – | – | 48.10 | 6.86 |
† Requires reference text.
The cross-lingual margins are the strongest part of the release. Against CosyVoice2 the gaps are not incremental — 48.10 versus 4.93 WER on ja→zh is a difference in kind, not degree.
MiniMax-Multilingual-Test shows a similar pattern on lower-resource languages: Thai WER 1.56 against ElevenLabs' 73.94, and Vietnamese 1.61 against 73.42. Those ElevenLabs figures indicate the commercial system essentially failing to produce intelligible speech on those language pairs rather than merely scoring worse. German lands at 0.47 WER, best in table.
On monolingual Seed-TTS-eval, the picture flips: English WER 1.49 trails FishAudio S2 at 0.99 and Qwen3-TTS at 1.24; Chinese CER 0.94 trails FishAudio S2 at 0.54. Confucius4-TTS is a cross-lingual specialist, not a monolingual leader.
Speaker similarity is the consistent weak spot. Across X-Voice, Seed-TTS-eval and MiniMax, SIM scores sit below OmniVoice and VoxCPM2 in nearly every row — Seed-TTS English SIM 0.70 against VoxCPM2's 0.753, Chinese 0.765 against 0.795. Both of those comparators require reference text, so the trade is legible: Confucius4-TTS gives up some timbre fidelity to drop the transcript requirement.
Inference requires Python 3.10 and CUDA 12.6. The default path runs the T2S stage through HuggingFace Transformers; a vLLM backend accelerates it with PagedAttention and supports both batch and streaming generation via a --stream flag.
Four entry points ship: example.py for CLI synthesis, a Python API, webui.py for a Gradio interface, and server.py for a FastAPI service exposing /api/tts (returns a complete WAV) and /api/tts/stream (raw int16-LE PCM chunks, sample rate in the X-Sample-Rate header). All accept the reference audio as an HTTP file upload, so client and server need not share a filesystem.
The full training pipeline for both T2S and S2A is published, with data supplied as 5-column TSV files.
The vLLM path is brittle. Only vLLM 0.16.0 (V1 engine) is supported, because model registration and GPUModelRunner monkey-patches target v1 engine internals. The README recommends a separate cloned conda environment specifically because installing vLLM into the base environment can break other packages. Any vLLM upgrade is likely to break this integration.
Environment pins are narrow. Python 3.10 and CUDA 12.6 are stated requirements, not suggestions.
Speaker similarity trails the field, as above — for applications where timbre fidelity matters more than transcript-free operation, the reference-text models remain ahead.
Adoption is thin so far. The Hugging Face repository shows only 14 likes, and the model card carries no license field even though the GitHub LICENSE file is Apache 2.0. Independent third-party evaluations are not yet available; every number here is vendor-reported.
Confucius4-TTS is a focused answer to a specific problem: keeping one speaker's identity intact while switching languages, without needing a transcript of the reference clip. On cross-lingual WER it clears the open field convincingly and, on Thai and Vietnamese, beats a commercial incumbent by margins that suggest the incumbent does not really support those pairs.
Teams building multilingual dubbing, localization, or voice-consistent assistants across Asian and European languages should evaluate it directly — the Apache 2.0 license and published training pipeline make adaptation realistic. Teams whose priority is maximum speaker similarity in a single language, or who need a stable high-throughput serving stack today, will find the SIM gap and the vLLM version pin harder to accept.