Building Real-Time AI Voice Agents: What LiveKit and OpenAI Realtime Actually Take

Everyone can wire up a chatbot in an afternoon. Almost nobody has actually shipped a real-time voice agent that holds a live video call, listens while someone is still talking, and responds with the latency and personality of an actual conversation. Having built AI interviewers that do exactly this — for hiring platforms and interview practice tools — the gap between "AI chat demo" and "production real-time voice agent" is bigger than most teams expect going in.

Text chat and voice agents are different engineering problems

A text-based AI assistant is fundamentally a request-response loop: user sends a message, model replies, done. A real-time voice agent has to solve problems that simply don't exist in that model:

Turn-taking. A human conversation has interruptions, pauses, and overlapping speech. The agent needs to know when the other person has actually finished talking versus just pausing to think — get this wrong and the agent either interrupts rudely or sits in awkward silence.

Streaming, not batching. You can't wait for someone to finish a 30-second answer, transcribe it, run it through a model, and then respond — that's an unusable multi-second dead zone. Audio has to stream in and responses have to start generating before the person has even finished speaking.

Session isolation at scale. If you're running interviews for multiple candidates simultaneously, each session's audio, transcript, and model state need to stay completely isolated. Cross-session bleed isn't a minor bug here; it's a data integrity failure.

The stack that actually handles this

For the real-time video/audio layer, I've settled on LiveKit — it handles WebRTC connection management, room/session isolation, and audio routing, which is the plumbing you don't want to build yourself. On top of that, OpenAI's Realtime API handles the actual conversational intelligence: it's built specifically for streaming audio in and audio out with low latency, rather than the batch-oriented completions API most people reach for first.

The architecture pattern that's worked well:

1. LiveKit Agents run server-side, joining each session as a participant — this is where the AI "lives" during the call.
2. Audio streams directly between the candidate's browser and the agent over WebRTC, avoiding the extra hop of uploading audio to a server and downloading a response.
3. The Realtime API handles speech-to-text, response generation, and text-to-speech in one continuous stream, rather than three separate services you have to glue together and debug independently.
4. A conventional backend (FastAPI in my case) handles everything that isn't real-time — session configuration, candidate profiles, storing the final transcript and structured feedback once the call ends.

Where it actually breaks in practice

The demo-to-production gap shows up in a few predictable places:

Latency compounds. Each hop — network to LiveKit, LiveKit to the model, model back through LiveKit, back to the browser — adds milliseconds that feel negligible individually but add up to a noticeably laggy conversation if you're not deliberate about minimizing hops.

Personality and configuration have to be prompt-engineered, not hardcoded. A generic "AI interviewer" persona sounds obviously robotic. Getting an agent to interview at a specific difficulty level, in a specific tone, for a specific role, requires careful meta-prompting — instructions about how the model should behave, not just what it should say.

Concurrent sessions need genuine isolation, not just separate database rows. Two candidates talking to the agent at the same time need fully separate LiveKit rooms, fully separate model context windows, and fully separate state — sharing any of that, even accidentally through a shared connection pool, causes context bleed that's hard to debug because it's intermittent.

Is this the right tool for your problem?

Real-time voice AI is genuinely powerful for anything that currently requires a human to be live on a call doing a repeatable, structured conversation — screening interviews, practice/coaching sessions, structured intake calls. It's overkill for anything that could just as easily be a form or an async chat; the added latency, cost, and complexity of a live voice pipeline only pays for itself when the liveness of the conversation is actually part of the value, not just a novelty.