You're on hold after work, phone warm against your ear, while a recorded menu asks you to press 1 for appointments and 2 for billing. The thing you need sits somewhere between those buttons. That gap is why AI voice agents are showing up on clinic lines, shop counters, and contact centres. They aren't a prettier menu. They're software that has to finish a job inside a live call, on a clock you can hear, and stop when it shouldn't guess.
What an AI Voice Agent Is in Plain Terms
Say it's 7 pm and you call a clinic to move Thursday's appointment. A natural voice answers, checks the calendar, offers two open slots, and texts a confirmation before you hang up. Nobody typed a ticket afterward. The booking changed while you were still on the line.
An AI voice agent is software that listens to speech, understands the request, takes actions in other systems, and answers in speech within one live call. Think of a receptionist who can hear you, look something up, and book it, not a recording that only routes button presses.

Three things people mix up
A touch-tone IVR follows a fixed menu tree. You press a key, it plays the next prompt, and it never remembers why you called. A smartphone assistant handles a single command, such as setting a timer or reading the weather, then waits for the next one. A voice agent holds context across several turns, completes a workflow, and hands the call to a person when it can't finish safely.
Those three jobs look similar because all of them use a voice. They fail in different places. The IVR fails when your reason doesn't match a button. The assistant fails when the task needs your account, your calendar, and a confirmation. The agent fails when it sounds confident and still books the wrong day.
What it has to combine
Four capabilities sit inside that one reply.
- Speech recognition: turns the caller's audio into text the rest of the system can use.
- Language understanding: decides what the person wants, including a change of mind mid-call.
- Speech synthesis: speaks the reply so the caller hears an answer, not a screen.
- Tool use: looks up or updates a calendar, CRM, or billing record before it speaks.
A smooth voice doesn't prove any of those four worked. The quick test for a product demo is simple. Can it finish the task, and does it know when to stop and transfer? Judge the agent on whether the call ends correctly and the handoff is clean, not on how human it sounds.
Practical rule: If the demo never shows a transfer, you haven't seen the product. You've seen the easy path.
How the Voice Pipeline Turns Speech Into a Reply
The wait you notice isn't "the model thinking." It's a chain. End-to-end latency runs from the end of the caller's speech to the first sound of the agent's reply. Model inference time alone hides the rest of that path.

In a cascaded stack, the stages run in call order. Airbyte's breakdown of production voice agents puts representative median times at about 400 ms for streaming speech recognition, about 450 ms for large-language-model time to first token, and about 220 ms for text-to-speech time to first byte. After network overhead, the complete path to first audio lands near 950 ms.
The stages in order
- The caller stops, and the system has to decide the turn is over. A pause while someone looks for a date isn't the same as "your turn."
- Streaming speech recognition writes the audio into text, about 400 ms at the median when it doesn't wait for a finished file.
- A language model reads that transcript and picks a reply or a tool, about 450 ms to the first token.
- Text-to-speech starts audio from that text, about 220 ms to the first byte.
- Playback reaches the ear. Network and orchestration bring the full first-audio path to about 950 ms.
Waiting for a final transcript, then a complete model answer, then a finished audio file stacks those stages end to end. Streaming partial transcripts, and passing model output to speech synthesis in fragments, lets the stages overlap. The caller hears the start of a sentence while the rest is still being written.
| Stage | What It Does | Reference Latency |
|---|---|---|
| Speech recognition (streaming) | Turns caller audio into text | About 400 ms median |
| Language model | Interprets the request and picks a reply or tool | About 450 ms to first token |
| Text-to-speech | Converts the reply into audio | About 220 ms to first byte |
| Network and orchestration | Moves audio and text between services | Brings the total to about 950 ms first audio |
| Backend tool call | Looks up or updates CRM, billing or knowledge base | Often the longest tail, so budget p95 and p99 plus a timeout |
The lookup that blows the budget
A CRM, billing system, or knowledge base can take longer than every speech stage combined. Median speech numbers look fine in a lab and still feel broken on a Tuesday afternoon when the account service is slow. A latency budget has to include p95 and p99 backend timing, a timeout, and a fallback line such as "one moment while I check that." Without the fallback, silence reads as a dropped call.
Measure end-of-speech to first audio on your own stack, plus time to a complete response, recovery after an interruption, and the same path with a real tool call. Inference time on a benchmark card doesn't tell you what the caller hears.
This short walkthrough shows how those stages line up when audio, text, and a reply have to stay in sync.
<iframe width="100%" style="aspect-ratio: 16 / 9;" src="https://www.youtube.com/embed/SPB2T-eLrOg" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>How Agents Hold a Conversation Without Talking Over You
A skilled phone operator stops mid-sentence when you cut in, and writes the order number down before reading it back. A voice agent has to do the same four things, or the call feels rude even when the facts are right.
It has to infer whether a pause is the end of a turn or a thinking gap. It has to keep listening while it speaks, and cancel its own audio the moment you barge in. It has to preserve dialogue state, so "the other one" still means the Thursday slot from two turns ago. It also has to ground tool arguments in what you said, not in a nearby guess.
Why text tests lie
A recent benchmark of naturalistic disfluencies and multi-step tool use, covered in an arXiv evaluation of full-duplex voice agents, shows the gap. The strongest system reached a tool-selection F1 of 0.876, argument accuracy of 0.680, response quality of 0.792, and Pass@1 task success of 0.600, with a 13.5% interruption rate.
Picking the right tool most of the time is not the same as finishing the task. Argument accuracy at 0.680 means the details inside the tool call are wrong often enough to book the wrong slot or read the wrong account. An interruption rate of 13.5% means the agent talks over the caller on a noticeable share of turns. Text chat never has to yield the floor.
Silence is not a turn boundary. People pause to find a policy number, and a system that jumps in then trains callers to talk over it.
Local mock APIs can isolate reasoning. Production tests have to repeat the same scripts with real service delay, failed logins, messy speech, barge-ins, and bad audio. Those conditions change both whether the task completes and whether the caller trusts the next sentence.
What to require before you sign
Ask any vendor to show these behaviours on a recorded trial, not a slide.
- Confirm before acting: read back the date, amount, or account change and wait for a yes.
- Yield on barge-in: stop playback immediately, keep the earlier context, and answer the interruption.
- Transfer after trouble: move the call after repeated misrecognition, or as soon as the caller asks for a person.
Word-error rate can look excellent while all three of those fail. Put pause handling, interruption rate, turn-end latency, tool-selection score, argument accuracy, and task completion on the same scorecard.
Answering and Screening Injury Calls With Lighthouse Legal
An injury call can arrive after the office closes, while the caller is still deciding whether to seek help. An AI agent can answer, collect information, and route the call according to the firm's intake rules. Lighthouse Legal offers customizable AI voice agents for this work. The agent supports the intake team; attorneys and staff retain responsibility for judgment and legal advice.

What the call actually does
The agent can answer calls outside staffed hours and during busy periods. It asks questions based on the firm's intake criteria, records relevant details, and routes inquiries for review. A caller who meets the firm's rules, has a high-priority issue, or asks to speak with a person can be transferred to the firm.
The call produces a recording, transcript, and report. The firm retains ownership of the call data, giving its team material to review against the written screening criteria. Setup involves defining routing rules and assigning a new phone number or forwarding the firm's existing line.
Pricing is shared during a demonstration. Firms considering AI voice agents for law firms should ask how the agent handles delays, unclear answers, and requests for a person. The right test is how many calls it completes and whether it captures useful information and routes callers appropriately.
This approach may suit firms that miss after-hours calls or need more consistent intake notes. It is a poor fit if the firm expects an agent to advise callers or decide a case's value.
Where Voice Agents Work Today and Where They Stop
A high share of calls that never reach a person is not a success score. It can mean the agent closed the wrong bookings all afternoon. The useful question is where the work already fits, and where a person has to take over before the agent is allowed to continue.
Customer-service and contact-centre lines are the base case, because the calls are repetitive and the answer can be checked against a system of record. A market report on voice AI cites Grand View Research valuing the global AI voice-agent market at about USD 2.5 billion in 2025, with a projection of USD 35.2 billion by 2033. North America held 38.1% of the 2025 share, and inbound voice agents represented 52.1% of global revenue. Research firms don't define "voice AI" the same way, so treat those figures as direction, not a budget line.
Write the exit before the scope
Decide the handoff rule first, then set what the agent may automate. Every live use case needs a defined exit.
| Use Case | What the Agent Handles | When to Hand to a Human |
|---|---|---|
| Appointment booking and changes | Checks availability, books and confirms | Complex scheduling conflicts or clinical questions |
| Order and account status | Authenticates caller and reads status | Disputes or refunds above policy limits |
| Call routing and triage | Collects reason for call and routes to queue | Urgent or distressed callers |
| Information collection | Gathers intake details and identifiers | Requests outside approved scope or sensitive exceptions |
| Outbound reminders and follow-ups | Confirms appointments and payment dates | Any objection or request to speak to someone |
Booking a haircut and rescheduling a procedure are not the same job, even if both touch a calendar. Status reads are safe when the agent only repeats what the system shows. They stop being safe when the caller disputes a charge. Outbound reminders stay narrow on purpose. The moment someone objects, or asks for a person, the script ends.
Run three questions on any proposed use case. Is the task repetitive? Can the answer be verified against approved data? Does a mistake cost little enough to undo? A use case that fails any of the three should start as a human-led workflow, or as a narrower slice of the same job. Automation scope is what remains after those exits are written down.
How to Test a Voice Agent Before It Takes Live Calls
Containment rate counts calls that never reach a person. It doesn't count whether those calls were right. A finished call with the wrong policy is worse than a transfer, because the caller leaves believing the business said something it didn't.
In a 2025 customer-service trends survey from PolyAI, 81% of customer-experience leaders named incorrect information as their biggest concern. 59% worried about damage to the customer experience, and 38% about brand damage. 53% said their technology stack wasn't ready for implementation. Only 15% of surveyed businesses relied on voice AI alone, while 85% combined it with human agents. The hybrid pattern isn't a lack of ambition. It's how teams keep a wrong answer from becoming the official one.
Scenarios a demo never runs
Script these before anyone talks about launch. A polished happy path won't surface them.
- Accents and code-switching: the same request in the ways your callers speak.
- Interruptions and silence: barge-in, long pauses, and a caller who starts again.
- Emotional or ambiguous callers: distress, vague asks, and two possible intents in one sentence.
- Authentication failures: wrong details, locked accounts, and a caller who can't pass the check.
- Hallucinated policies: questions whose true answer isn't in the approved source.
- Tool outages: CRM or calendar down, slow, or returning an error.
- Human escalation: an explicit request for a person, plus a transfer after repeated misses.
Read the metrics in this order
Verified resolution and error severity outrank containment. A high automation rate with unchecked errors is a bad week you haven't measured yet.
| Metric | What It Reveals | How to Measure |
|---|---|---|
| Verified resolution rate | Calls actually solved, checked against the system of record | Compare sampled transcripts with account records |
| Error severity | How harmful each wrong answer would be to the caller | Grade errors as minor, costly or harmful |
| Containment rate | Calls that never reach a person | Read only alongside verified resolution |
| Escalation quality | Whether the human receives the caller's details and reason | Review the handoff summary on sampled calls |
| Caller trust signals | Repeat requests for a person, hang-ups and CSAT | Track by call type and caller group |
| Degraded-condition pass rate | Performance during outages, poor audio and authentication failures | Run scripted failure tests before launch |
A call the agent "contained" and got wrong should count as a failure, not as automation.
Run the same scenario set again with live service latency, real authentication, and a forced tool outage. A green demo on mock APIs doesn't predict the tail your callers will hit. Don't make a launch decision on the mock run.
Voice Rights, Disclosure and the Risks Beyond Accuracy
The voice on the line is a rights question, not only a model setting. Someone's recorded speech became an asset, and that person may have little say in where it shows up next.

A 2025 ACM Fairness, Accountability, and Transparency study found that voice actors described low transparency and limited control in established industry relationships, including concern that their voices could be altered or used without informed consent. Privacy, security, safety, and provenance stay live risks even with the EU AI Act, Illinois biometric-privacy protections, and California privacy law on the books. Those laws set duties. They don't tell you whether the voice in your demo was licensed for this use.
Customer consent is not performer consent
Customer consent covers the call. The caller agrees that you may process what they say so you can book, route, or answer. Performer consent covers reuse of a worker's voice as a synthetic asset. Authorizing a company to handle a phone call doesn't authorize indefinite reuse of someone's voice in other products, ads, or cloned lines.
Telling the caller they are speaking with AI is necessary and still not enough. Disclosure can be perfect while the underlying voice has no clear provenance, no revocation path, and no owner when an impersonation claim arrives.
Questions to put in the vendor review
Ask these in writing, and keep the answers with the contract.
- License: Is this voice licensed for synthetic use in your industry and region, and can you see the grant?
- Retention: How long are recordings and derived voice data kept, and who can delete them?
- Processing location: Where is biometric or voice data processed and stored?
- Withdrawal: How does a performer or a caller revoke consent, and how fast does that take effect?
- Impersonation: Who investigates a claim that the voice was copied, and what indemnity applies?
Audit logs, a retention limit, and a named incident owner matter more than a clause that says "AI disclosure provided." If the vendor can't separate the caller's permission from the speaker's permission, don't put that voice on a live line.
Decide Build or Buy and Run a Pilot First
Pick one call type and refuse to widen it until the numbers from live calls say the exit rules hold. A pilot is a test with a stop condition, not a soft launch.
- Write the handoff before you shop. Take the exit from the use-case list, such as "any clinical question transfers," and make a product owner sign it. The pass threshold is a written rule a stranger could apply to a transcript. Looking at vendors first invites a demo to set your scope.
- Choose build or buy on ownership, not on a feature grid. Ask who owns the telephony integration, who controls the voice license, and who carries compliance duties. Buying usually fits standard contact-centre flows. Building fits a proprietary workflow with strict data rules. Name an engineering owner for the path you pick, and a date when the other path is formally dropped.
- Demand a trial, not a tour. The vendor, or your build team, shows end-of-speech to first audio, verified resolution, and interruption handling on your calls, including degraded audio and a tool outage. A product manager owns the pass mark before the trial starts. A smooth demo with a mock calendar doesn't count. Our directory lists 98 AI tools, which is useful for a first shortlist and useless as a substitute for that trial.
- Pilot one call type for two to four weeks. Sample calls for review and track error severity, not just how many calls stayed with the agent. An operations owner publishes the sample rate and the weekly cutoff. If harmful errors show up, the pilot stops even if containment looks strong.
- Put rights and failure in the contract. Cover voice licensing, recording retention, audit logs, incident response, and escalation to a person. Legal owns the clause list, and the pilot can't expand until those clauses are signed. A handshake on "we'll add disclosure later" isn't a control.
The go or no-go rests on verified resolution and error severity for that one call type. Containment is a footnote.
Start there. One call type, a fixed scenario set, live latency, and a written decision. If the pilot fails the threshold, keep the calls with people and narrow the job. That is a result, not a delay.
If you want a short daily pass over what changed in AI, without sitting through another product demo, Zeniteq's daily briefing is free, lands at 7 AM in your timezone, and you can unsubscribe any time. Each note links out when you want the longer piece.
Sources (7)
- 01airbyte.comAirbyte's breakdown of production voice agents (opens in a new tab)
- 02arxiv.orgarXiv evaluation of full-duplex voice agents (opens in a new tab)
- 03lighthouselegal.aiLighthouse Legal (opens in a new tab)
- 04lighthouselegal.aiAI voice agents for law firms (opens in a new tab)
- 05growthmarketreports.commarket report on voice AI (opens in a new tab)
- 06poly.ai2025 customer-service trends survey from PolyAI (opens in a new tab)
- 07dl.acm.org2025 ACM Fairness, Accountability, and Transparency study (opens in a new tab)