Service · S·04

AI Voice Agent Development

Real-time phone and web voice agents on the OpenAI Realtime API — engineered to a strict latency budget, with barge-in handling, mid-call tool calls, and human handoff built in from the first design session.

Last reviewed: 24 July 2026

01 · Definition

What is an AI voice agent?

An AI voice agent is a software system that holds a real-time spoken conversation with a caller — over the phone or in a browser — understands intent, takes actions mid-call, and hands the call to a human when it should. It is built either on a speech-to-speech model such as the OpenAI Realtime API or on a pipeline of speech recognition, a language model, and speech synthesis. A production voice agent must do four things a chatbot never has to: respond fast enough to feel conversational, with a total-to-first-audio budget of roughly one second; stop speaking the instant the caller interrupts; call tools such as calendars and CRMs while the line is live; and transfer to a person with full context so the caller never repeats themselves. Rohan Jalil builds voice agents to these budgets, with guardrails, escalation rules, and full transcripts as standard.

Voice is not a skin you put on a chatbot. The conversation is synchronous, the caller cannot scroll back, and every hundred milliseconds of delay is audible. That changes the architecture from the first line of code.

02 · Latency

What makes voice harder than chat?

Latency, mostly. In text, a two-second pause is invisible. On a call, it reads as a dropped line. A voice agent is engineered against a millisecond budget, stage by stage.

Stage	What happens	Engineering target
Turn detection & ASR finalisation	Detecting that the caller has stopped speaking and finalising the transcript of their turn	~100–300 ms
Model response start	Time until the language model begins producing its reply	~200–500 ms
TTS first byte	Time until synthesised speech starts streaming back to the caller	~100–200 ms
Transport & telephony	WebRTC or SIP overhead in both directions	~50–150 ms
Total to first audio	The gap the caller actually hears before the agent starts speaking	Under ~1 s target; ~1.5 s ceiling

These figures are design budgets, not guarantees: network conditions, telephony provider, model choice, and how much work the agent does mid-turn all move them. The point of the budget is that every stage is measured in production, so when total latency drifts you know exactly which stage to fix.

The second hard problem is barge-in: callers interrupt, and the agent must stop talking within a fraction of a second, discard its queued audio, and listen. Getting turn-taking wrong — talking over the caller, or leaving dead air — is the fastest way to make a voice agent feel like a bad IVR menu.

03 · Scope

What does a production voice agent include?

The demo is the easy part. These six subsystems are what separate a weekend prototype from an agent you can put on a public phone number.

Telephony & web audio	Inbound and outbound calls over SIP or a provider such as Twilio, plus in-browser voice over WebRTC — with codec handling and echo treatment sorted before any prompt work
Turn-taking & barge-in	Voice-activity-based turn detection tuned per use case; when the caller interrupts, playback is cancelled and queued audio flushed so the agent yields immediately
Tool calls mid-call	Calendar lookups, CRM reads and writes, and order-status checks executed while the line is live, with a spoken acknowledgement covering any tool that takes longer than a beat
Guardrails & escalation	Scoped topics, explicit refusal behaviour for anything out of bounds, and escalation triggers: a request for a human, repeated misunderstanding, or signs of frustration
Human handoff with context	Warm transfer that carries a conversation summary, the caller's intent, and every detail already collected — the caller never starts the story over
Monitoring & transcripts	Structured transcripts for every call, per-stage latency dashboards, failure alerts, and recordings where lawful — so quality is reviewed from evidence, not anecdotes

04 · Fit

Where do voice agents work well?

Voice agents earn their keep on high-volume, well-bounded calls — and fail expensively everywhere else. Part of this service is saying no to the wrong use case.

FIT·01

Reception & routing

Answer every call, identify the reason for calling, and route to the right person or queue — no hold music, no phone tree.

FIT·02

Lead qualification

Ask the qualifying questions, capture structured answers straight into the CRM, and book qualified leads onto a sales calendar.

FIT·03

Bookings & scheduling

Appointments, reschedules, cancellations, and reminder calls — checked against the live calendar mid-conversation.

FIT·04

Tier-1 support

Order status, account basics, and questions a knowledge base can answer — grounded in retrieval, escalated the moment it gets hard.

NOT·A·FIT

Where a voice agent is the wrong tool

Medical, legal, or financial advice; complaints and other emotionally charged calls; negotiation; any conversation where a wrong answer is costly and no human can intervene. Callers who ask for a person get one — immediately.

05 · Process

How does a voice agent build run?

One narrow call type first, tuned against the latency budget on real audio, then rolled out in stages with humans listening.

Step 01

Call design & scope

Pick one call type worth automating. Map the happy path, every escape hatch, and the escalation rules before any code — including what the agent must refuse to discuss.

Step 02

Prototype on real audio

A browser-based prototype on the OpenAI Realtime API, tested with real voices, accents, and background noise — not just the founder's clean desk microphone.

Step 03

Latency & barge-in tuning

Measure every stage against the budget table above; tune turn detection, playback cancellation, and response strategy until interruptions feel natural.

Step 04

Tools & guardrails

Wire the mid-call tool calls — calendars, CRM, order systems — plus scoped topics, refusal behaviour, and every handoff trigger, each with its own tests.

Step 05

Telephony & staged rollout

Connect the phone number and go live on after-hours or overflow calls first, with transcripts reviewed daily before the agent takes front-line volume.

Step 06

Handover

Latency dashboards, a transcript review workflow, and a regression call suite your team can run whenever the prompt, tools, or model version changes.

06 · FAQ

AI voice agent questions, answered

What latency can I expect?

A well-built voice agent should begin speaking within roughly one second of the caller finishing — that is the engineering budget this service designs and measures against, not a guaranteed constant. Network conditions, the telephony provider, and how much mid-turn work the agent does all move the number, and anything much beyond 1.5 seconds starts to feel broken. Every stage is instrumented in production so drift is caught at the stage that caused it. The same measure-everything discipline runs through the documented builds in the case studies.

Can it hand off to a human?

Yes — human handoff is designed in from day one, not bolted on. The agent warm-transfers the call and passes context with it: a summary of the conversation, the caller's intent, and every detail already collected, so the caller never starts over. Handoff fires on an explicit request for a person, repeated misunderstanding, signs of frustration, or any topic outside the agent's scope.

Which stack — OpenAI Realtime or pipeline ASR+LLM+TTS?

The OpenAI Realtime API is the default: as a speech-to-speech model it delivers the lowest latency and the most natural barge-in behaviour with the least glue code. A pipeline of separate speech recognition, language model, and speech synthesis wins when you need a specific branded voice, a language the Realtime API handles poorly, tighter per-stage cost control, or recognition that must run inside your own infrastructure. The honest answer is decided per project, against the latency budget and the requirements — not by loyalty to either architecture.

How do you test a voice agent?

With a scripted call suite run against recorded and live audio, then a staged rollout with humans reviewing transcripts. The suite covers the happy path, mid-sentence interruptions, accents and background noise, out-of-scope requests, and deliberately failed tool calls. Before full launch the agent takes only after-hours or overflow traffic while transcripts are reviewed daily, and the same suite re-runs on every prompt, tool, or model change.

07 · Related

Related services & evidence

S·01

08 · Next step

Put a voice agent on the line — safely.

Hire on Upwork → Email Rohan