Speech To Speech is Hugging Face's open-source toolkit for building voice agents from swappable, open models. It assembles a classic conversational pipeline — voice activity detection, speech-to-text, a language model, and text-to-speech — into a single low-latency service, then exposes it through an OpenAI Realtime-compatible WebSocket API. That combination is the project's core pitch: you get a fully open, self-hostable voice stack that any client written for OpenAI's Realtime API can talk to without modification. Licensed under Apache 2.0 and already running in production as the conversation backend for thousands of Reachy Mini robots, it climbed GitHub's weekly trending list as teams looked for an open alternative to closed real-time voice services. ## A Modular Voice-Agent Pipeline The architecture is a cascade of four stages, each running in its own thread and connected by queues: Silero VAD v5 detects when someone starts and stops speaking, an STT model transcribes the turn (with optional live partial transcripts), a language model streams the response and any tool calls, and a TTS model synthesizes audio back to the caller. The default configuration is opinionated but sensible — Parakeet TDT for local speech recognition, an OpenAI-compatible endpoint for the LLM, and Qwen3-TTS for local speech output. Because every stage is a separate, interchangeable backend selected by a CLI flag, you can reshape the pipeline for your latency, quality, and hardware constraints without rewriting the plumbing. ## OpenAI Realtime-Compatible, Swappable Everywhere The design decision that makes the project immediately useful is protocol compatibility. Running `speech-to-speech` starts a server at `ws://localhost:8765/v1/realtime` that speaks the OpenAI Realtime protocol, so existing Realtime clients connect to it as a drop-in endpoint — the README even demonstrates switching a client from hosted OpenAI to a self-hosted server by changing a single URL. The LLM slot is equally flexible: because it speaks OpenAI-compatible protocols, you can point it at a hosted provider, at Hugging Face Inference Providers, or at a local vLLM or llama.cpp server. Serving Gemma 4 through llama.cpp and pointing the pipeline at it yields a fully local, fully open voice agent. ## Local-First or Hosted The toolkit leans toward running on your own hardware while leaving the escape hatches open. A broad set of optional backends installs through pip extras — Kokoro-82M, Pocket TTS, ChatTTS, and Facebook MMS for speech synthesis; Faster Whisper, Lightning Whisper MLX, and Paraformer for recognition — and platform-specific dependencies resolve automatically, using GGML on Linux and mlx-audio on Apple Silicon. That range lets you trade off model size, language coverage, and speed per component, and the emphasis on models available through Transformers and the Hugging Face Hub keeps the whole stack inside a familiar, open ecosystem rather than behind a proprietary API. ## Trade-offs and Limitations A cascaded VAD -> STT -> LLM -> TTS design is modular and debuggable, but it is not a single end-to-end speech model, so each stage adds its own latency and potential error, and truly natural turn-taking remains hard. Setup is more involved than a hosted service: some backends have particular requirements — the Qwen3-TTS GGML backend, for instance, ships a wheel targeting a specific CUDA version and may need a matching build, and DeepFilterNet's `numpy<2` requirement conflicts with Pocket TTS's `numpy>=2`. Getting good latency generally implies a capable GPU, and quality depends entirely on which component models you choose. As with any fast-moving repository, some implementations have already been deprecated into an archive folder. ## Who Should Use This Speech To Speech is aimed at developers and teams who want a real-time voice agent they can host, inspect, and customize rather than rent. Its OpenAI Realtime compatibility makes it a natural landing spot for anyone already building against that API who wants an open backend, and its modular design suits robotics, on-device assistants, and privacy-sensitive deployments where sending audio to a third party is a non-starter. Users seeking a zero-configuration hosted voice service will find the setup demanding, but for building an open, self-hosted conversational stack, this Apache-2.0 toolkit is a well-engineered, production-tested starting point.