Open Source
Explore the latest AI open-source projects from GitHub and HuggingFace.
Explore the latest AI open-source projects from GitHub and HuggingFace.
**ZONOS2** is Zyphra's open-weight text-to-speech model, built on a mixture-of-experts backbone and trained on more than **6 million hours** of multilingual speech. Its focus is high-fidelity zero-shot voice cloning across 34 languages, served through a local inference stack rather than an API. The repository is **MIT licensed** with weights published under Apache-2.0 on Hugging Face, where the model has collected **138 likes**. The GitHub repo sits at roughly **293 stars** and **29 forks**. ## Architecture The inference path is unusual enough to be worth stating plainly: text is normalized with NeMo TN and consumed as **raw UTF-8 bytes** (a 519-entry vocabulary — no BPE tokenizer), speaker identity arrives as an **ECAPA-TDNN embedding**, and the MoE backbone autoregressively generates **DAC audio tokens** across 9 codebooks of size 1024. | Component | Configuration | |---|---| | Layers | 28, model dim 2048, head dim 128, 4 KV heads | | Experts | 16, **top-1 routing**, MoE from layer 3 | | Audio tokens | 9 DAC codebooks × 1024 entries | | Context | 6144 tokens | | Checkpoint | 15.3 GB bfloat16 (≈7.7B total parameters) | | Output | float32 PCM, 44.1 kHz mono | Top-1 routing over 16 experts is the load-bearing efficiency choice: total capacity stays large while per-token compute stays near a single dense FFN, which is what makes the low-latency claim plausible for a model of this checkpoint size. Language support is tiered — English, Mandarin and Japanese at tier 1; Korean, Russian, Italian, Portuguese, French, Spanish, Vietnamese, German, Hebrew and Dutch at tier 2; and 21 further languages at tier 3. ## Control Surface Where ZONOS2 separates itself from most open TTS releases is the amount of conditioning exposed at generation time. **Emotion without identity drift.** Emotion is applied as additive *direction vectors* on the speaker conditioning rather than through fine-tuning or separate checkpoints, so timbre is preserved while prosody shifts. Sliders cover happy, sad, angry and surprised, plus valence and arousal axes. Shipped directions include a `calibration.json` so `emotion_strength: 1.0` is already a per-speaker sensible default, and `scripts/build_emotion_directions.py` lets you derive your own set from an emotion-labelled corpus such as ESD. **Accurate versus expressive mode.** A single flag trades voice-match fidelity against expressive range — with the documentation noting that strong emotion effects want expressive mode plus `emotion_cfg_scale` near 1.5, at roughly double the compute. **Speaking rate and recording quality as conditioning.** Rate can be set as an OpenAI-style `speed` multiplier, as cleaned bytes-per-second, or as an exact model bucket. Quality conditioning runs over six measured features — LUFS, estimated SNR, max pause, bandlimit, and leading/trailing silence — bucketed so you can ask for, say, a specific trailing-silence profile instead of trimming afterwards. **Serving.** The reference server is built on Mini-SGLang, streams by default on port 1919, and exposes both a full-featured `/tts/generate` endpoint and an OpenAI-compatible `/v1/audio/speech`. A Python API (`TTSLLM`) runs offline at documented parity with the server. For CPU and cross-platform use, Zyphra maintains a separate **ggml implementation, zonos2.cpp**. ## Caveats **No published numbers.** The claim is quality "on par with — or even surpassing — top TTS providers." There is no benchmark table, no MOS or WER evaluation, and no comparison baseline in the repository. The cited technical report appears only as a bare `@misc` entry with no link or arXiv identifier. Prospective users are being asked to judge by listening to blog samples. **The reference server is Linux x86_64 with NVIDIA CUDA only.** No macOS, no Windows, no AMD. zonos2.cpp is the escape hatch, and it is a separate repository with its own maturity question. **Text normalization covers far fewer languages than the model.** The `language` parameter accepts ten values (`en_us`, `en_gb`, `fr_fr`, `de`, `es`, `it`, `pt_br`, `ja`, `cmn`, `ko`). The model claims 34. For the other 24, numbers, dates and currency reach the model unverbalized unless you normalize them yourself upstream. **A 15.3 GB bf16 checkpoint is not a laptop model.** Despite top-1 routing keeping active compute low, the full expert set must be resident. Combined with the CUDA-only server, the practical floor is a substantial NVIDIA GPU. **Project activity is thin.** Two contributors, and the last push was a month before this writing. That is normal for a model-release repository, but it is not an actively co-developed codebase. **Zero-shot cloning quality is also a misuse surface.** High-fidelity cloning from a short reference clip, released under a permissive license with no gating, carries the impersonation risk that every capable open TTS model now carries. The repository ships no watermarking or consent mechanism. ## Verdict ZONOS2 is best read as an engineering release rather than a research one. The MoE-plus-DAC-tokens design, the byte-level text path, and above all the conditioning surface — emotion directions that leave timbre intact, recording-quality buckets, rate control in three units — are the parts that matter, and they are the parts most open TTS releases do not bother to expose. The gap is evidence. Without evaluation numbers, "on par with top providers" is a claim you have to test yourself, and the CUDA-only Linux requirement makes that test more expensive than it should be. For teams building a self-hosted voice product on Linux GPUs who need per-generation prosody control and multilingual cloning, it is a strong candidate worth benchmarking against the alternatives. For anyone needing macOS or CPU deployment, zonos2.cpp is the path — and a separate evaluation.