## Introduction Fish Speech is a state-of-the-art open-source text-to-speech system developed by Fish Audio that has rapidly emerged as one of the most capable multilingual TTS solutions available. With over 26,000 GitHub stars and training on more than 10 million hours of audio across approximately 50 languages, Fish Speech represents a significant leap in open-source speech synthesis quality and versatility. What distinguishes Fish Speech from other TTS systems is its Dual-Autoregressive architecture, which structurally mirrors large language models. This design choice enables it to inherit LLM-native serving optimizations, delivering production-grade streaming performance with a real-time factor of 0.195 on H200 GPUs. The project is actively maintained and has reached version 1.5.1, demonstrating consistent development momentum. ## Architecture and Design Fish Speech employs a novel Dual-Autoregressive architecture consisting of two complementary models: a 4-billion parameter model responsible for semantic codebook generation, and a 400-million parameter model that handles acoustic detail refinement. This two-stage approach separates high-level linguistic understanding from fine-grained audio production, resulting in speech that captures both semantic accuracy and acoustic naturalness. The system incorporates GRPO-based reinforcement learning alignment, a technique borrowed from the LLM training paradigm, to further refine output quality. This alignment process helps the model produce speech that better matches human preferences for naturalness, clarity, and emotional expressiveness. | Specification | Detail | |---------------|--------| | Semantic Model | 4B parameters | | Acoustic Model | 400M parameters | | Training Data | 10M+ hours of audio | | Languages | ~50 languages | | Alignment | GRPO reinforcement learning | | Serving | SGLang integration | | RTF (H200) | 0.195 | Benchmark results on the Seed-TTS evaluation set demonstrate Fish Speech's competitive edge: 0.54% word error rate on Chinese and 0.99% on English, surpassing several closed-source alternatives including Qwen3-TTS and MiniMax Speech-02. ## Key Capabilities Fish Speech offers a comprehensive feature set that spans both research and production use cases: **Fine-Grained Inline Control**: Unlike many TTS systems that offer only global style parameters, Fish Speech supports natural language instructions embedded directly in the text. Tags like `[whisper]`, `[super happy]`, `[sad]`, and `[shouting]` allow precise control over speaking style at the phrase level, enabling complex emotional narratives within a single generation. **Voice Cloning**: With just 10-30 seconds of reference audio, Fish Speech can clone a speaker's voice with high fidelity. The cloned voice retains the original speaker's timbre, accent, and speaking characteristics, making it suitable for personalized TTS applications and content localization. **Native Multi-Speaker Support**: The model handles multiple speakers and multi-turn conversations natively, generating distinct voices for different speakers without requiring separate model instances or post-processing concatenation. **Zero-Shot Multilingual Generation**: Fish Speech processes text in approximately 50 languages without requiring language-specific phoneme preprocessing. This eliminates the traditional TTS pipeline complexity of maintaining separate grapheme-to-phoneme converters for each language. **Production Streaming**: Integration with SGLang enables efficient serving with batched inference, making Fish Speech viable for real-time applications at scale. The 0.195 RTF on H200 means the system generates speech roughly five times faster than real-time. ## Developer Integration Fish Speech provides multiple deployment options. The Python SDK offers straightforward programmatic access: ```python from fish_speech import FishSpeech model = FishSpeech.from_pretrained("fishaudio/fish-speech-1.5") audio = model.generate( text="Hello, welcome to Fish Speech.", reference_audio="reference.wav" ) ``` For production deployments, the SGLang-based serving infrastructure supports concurrent requests, streaming output, and batched processing. Docker containers are available for containerized deployment, and a Gradio-based web interface enables quick prototyping and demonstration. The project also provides fine-tuning scripts for domain adaptation, allowing users to specialize the model on specific speakers, languages, or acoustic environments with relatively small datasets. ## Limitations Fish Speech operates under the Fish Audio Research License, which imposes restrictions on commercial use without explicit authorization. While the model supports approximately 50 languages, quality varies significantly across less-represented languages. GPU acceleration is required for practical inference speeds, and the 4B parameter semantic model demands substantial VRAM for local deployment. Voice cloning quality depends heavily on the quality and length of reference audio, with noisy or very short samples producing degraded results. ## Who Should Use This Fish Speech is ideal for developers building multilingual voice applications, content creators producing localized audio content, researchers exploring expressive speech synthesis, and teams needing production-grade TTS with voice cloning capabilities. Its LLM-like architecture makes it particularly interesting for teams already familiar with large language model serving infrastructure. The fine-grained inline control system appeals to creative applications where emotional and stylistic variation within a single passage is essential.