LiveKit Agents is a Python framework for building realtime voice AI — programmable participants that run on servers and can hear, see, and speak in a live session. Rather than shipping its own speech models, it defines the session loop around them: audio arrives over WebRTC, a voice activity detector and turn detector decide when the user has finished speaking, STT transcribes, an LLM reasons and calls tools, and TTS speaks the answer back with barge-in handled.

The integration surface is deliberately open. Any combination of STT, LLM, TTS, or a realtime speech-to-speech API can be composed inside an AgentSession, with plugins for Deepgram, OpenAI, Cartesia, and many other providers — or LiveKit Inference, a unified hosted API, when you would rather not manage provider keys. Swapping a TTS vendor is a one-line change, which is the practical argument for the abstraction.

The piece hardest to build yourself is semantic turn detection. Fixed silence thresholds either cut users off mid-thought or leave dead air; LiveKit runs a transformer model over the conversation to judge whether a pause is a real end-of-turn, which is where most of the perceived latency and interruption quality in a voice agent lives. Alongside it sit integrated job scheduling with dispatch APIs to route users to agents, and telephony via LiveKit's SIP stack, so an agent can place and receive ordinary phone calls without a separate bridge.

The framework also ships things voice projects usually improvise: a built-in test framework with LLM judges for asserting an agent behaves as intended across turns, native MCP support that registers tools from an MCP server in one line, and RPC and data APIs for exchanging structured state with clients. Client SDKs cover the major platforms, and a companion AgentsJS library serves JavaScript and TypeScript teams.

The caveat is architectural rather than a defect: the stack is designed around LiveKit's own WebRTC media server. That server is open source and self-hostable, so nothing here is locked behind the hosted product, but adopting Agents means adopting LiveKit's media plane, and the smoothest path — LiveKit Inference and managed dispatch — runs through LiveKit Cloud. Both the framework and the server are Apache-2.0, and the project releases frequently, with livekit-agents 1.6.9 on 2026-08-07.