LiveKit Agents: Realtime Voice AI Framework
/agents
LiveKit Agents is a Python framework for building realtime voice AI agents that run as long-lived server processes instead of one-shot API calls. Reach for it if you need to swap STT, LLM, and TTS providers per deployment and eventually take phone calls over telephony; skip it if a single request/response LLM call already covers what you're building — the AgentServer/AgentSession model is more infrastructure than that needs.
What is the LiveKit Agents Framework?
LiveKit Agents is an open-source Python framework for building realtime, programmable voice participants that run on servers — conversational, multi-modal agents that can see, hear, and understand, per the README. It centers on three objects: an Agent holding instructions and tools, an AgentSession managing the user interaction, and an AgentServer that schedules jobs and launches an agent per session.
Key Features for Conversational AI
- ✓Provider-agnostic session config: an AgentSession takes any combination of VAD, STT, LLM, and TTS, or a single Realtime API — the README's own example swaps between LiveKit Inference and direct Deepgram/OpenAI/Cartesia plugins with a few commented-out lines.
- ✓Job scheduling and dispatch APIs route end users to a running agent instance, instead of a hand-built queue and routing layer.
- ✓Telephony integration through LiveKit's own SIP stack, so an agent can place or receive phone calls without a separate vendor.
- ✓Semantic turn detection through a transformer model that judges when a user has actually finished speaking, rather than a fixed silence timeout.
- ✓Native MCP support: a tool from an MCP server plugs into an agent with one line of code, per the README.
- ✓A built-in test framework — pytest-based session tests plus LLM 'judges' that check an assistant reply against a stated intent.
- ✓Data exchange with connected clients via RPCs and other LiveKit Data APIs, on top of the voice/video session itself.
- ✓Client-side reach through LiveKit's WebRTC SDK ecosystem, so the same agent can be reached from browser, mobile, or desktop clients.
Practical Examples and Use Cases
- •A phone-based ordering or reservations agent — the repo's own examples include a full restaurant ordering/reservations flow.
- •An outbound caller agent that dials out over the phone, reusing the same calling integration.
- •A vision-capable assistant, e.g. the README's Gemini Live vision example (with an iOS app) where the agent can see through a client's camera.
- •A multi-agent handoff flow — one agent gathers user details via a function_tool, then hands the session to a second, specialized agent.
- •A text-only integration that reuses the same agent code without voice, for chat-first surfaces.
Installation and Setup
Install the core package with provider plugins as extras: pip install "livekit-agents[openai,deepgram,cartesia]" pulls in the framework plus the named STT/LLM/TTS plugins. Documentation lives at docs.livekit.io/agents; install/usage beyond that page is not detailed further in the README. Building with an AI coding agent, the README recommends adding the LiveKit Docs MCP server for current API context, plus the LiveKit Agent Skill package — installed via npx skills add livekit/agent-skills --skill livekit-agents — for architectural guidance on workflows, handoffs, tasks, and testing.
Building and Running Your Agent
An AgentServer wraps the app; you register an entrypoint with @server.rtc_session(), build an AgentSession with your chosen vad/stt/llm/tts (or LiveKit's own unified Inference API), attach an Agent with instructions and @function_tool-decorated tools, then call session.start() and session.generate_reply(). A session needs three environment variables: LIVEKIT_URL, LIVEKIT_API_KEY, and LIVEKIT_API_SECRET. Three run modes cover the work cycle: python myagent.py console runs locally with terminal audio I/O and no external server, for quick testing; python myagent.py dev starts the server with hot reload and connects to LiveKit Cloud or a self-hosted server, testable through the Agents Playground or any LiveKit client SDK; python myagent.py start runs it "with production-ready optimizations," per the README. For automated checks, the framework ships native pytest integration — a test can run a session against a canned user_input and assert on the resulting events, including an LLM-based judge() call that checks the assistant's reply against a stated intent.
Strengths
- ✓Provider-agnostic sessions: the README's own example swaps between LiveKit's unified Inference API and direct Deepgram/OpenAI/Cartesia plugins with a few commented-out lines, not a rewrite.
- ✓Semantic turn detection runs on an actual transformer model instead of a fixed silence timeout — a real answer to an agent talking over the user.
- ✓Placing and receiving phone calls is a first-class integration through LiveKit's own SIP stack, not a bolt-on — the README lists a dedicated outbound-caller example.
- ✓Native MCP support means an existing MCP toolserver plugs into an agent with one line of code, instead of hand-rolled tool-calling glue.
- ✓Ships a real test framework: pytest-based session tests plus LLM 'judges' that check an assistant's reply against a stated intent, for a category of software where correctness is otherwise hard to assert.
Considerations When Using LiveKit Agents
- △AgentServer, AgentSession, and the entrypoint pattern are a specific mental model to learn — this isn't a one-line chat-completion call, and there's real scaffolding before your first response.
- △Full phone-calling and Realtime API paths depend on LiveKit Cloud or a self-hosted LiveKit server plus SIP configuration — not something pip install alone gives you.
- △The Apache-2.0 license covers the framework code, but the bundled turn detection models ship separately under the LiveKit Model License, per the repo's LICENSE and MODEL_LICENSE files.
- △Python is the documented, primary implementation here — JS/TS support lives in the separate AgentsJS repository, so a Node.js team is tracking two codebases, not one.
Alternative AI Agent Frameworks
Frequently Asked Questions
LiveKit Agents (the framework code) is licensed under Apache-2.0. The bundled turn detection models are licensed separately, under the LiveKit Model License, per the repo's LICENSE and MODEL_LICENSE files.
LiveKit Agents is written in Python, matching the repository's listed primary language. A separate JS/TS implementation, AgentsJS, exists as its own repository for teams building on Node.js.
LiveKit Agents supports mixing and matching STT, LLM, TTS, or a Realtime API per session through provider plugins — the README's own install example alone bundles openai, deepgram, and cartesia as optional extras.
LiveKit Agents uses a transformer model for semantic turn detection, judging when a user has actually finished speaking rather than relying on a fixed silence timeout, which the README says helps reduce interruptions.
LiveKit Agents includes a start command that the README describes as running the agent "with production-ready optimizations," separate from the console and dev commands used for local testing and hot-reload development.
LiveKit Agents integrates with LiveKit's own telephony stack so an agent can place or receive phone calls, and the project's own examples include a dedicated outbound-caller agent.
The problem it solves
Building a voice AI agent well means solving several problems outside a single LLM API call at once: judging when a caller is actually done talking (not just pausing) instead of cutting them off, keeping a session alive across a phone call or WebRTC room, swapping the STT/LLM/TTS combo per deployment without rewriting the app, and testing the result when the LLM in the loop is non-deterministic. LiveKit Agents packages those specific pieces — semantic turn detection, an AgentSession abstraction, pluggable provider integrations, and a native pytest-based test framework with judges — instead of leaving a developer to wire a VAD library, a provider SDK, and a call-routing script together by hand.
Who should try it — and who should skip
Reach for LiveKit Agents if you're building a voice agent that needs to run continuously as a server process, plug in different STT/LLM/TTS providers per use case, and eventually take phone calls or ship to a real WebRTC client. Skip it if you want a single request/response LLM call from a script or a serverless function — the AgentServer/AgentSession model, and the LiveKit Cloud or self-hosted server it expects underneath, is more infrastructure than a one-off text chatbot needs.
Related repositories
Still deciding about agents?
One click hands the question to an AI along with this page — see what it says about agents.