Voice AI for customer and staff calls: what works, what is hard

Where voice AI works well

A voice agent listens to a caller, turns the speech into text, decides what to say or do, and speaks the reply. It works best on calls that are routine and structured: the caller wants one of a small number of things, the information needed is known, and the call ends in a defined action.

Typical examples are booking or moving an appointment, checking the status of an order or a claim, confirming a delivery slot, collecting details before a person takes over, and answering questions about opening hours and policies. Internal lines qualify too, such as a staff IT helpdesk. Out-of-hours cover is a common first use, because the alternative is voicemail.

Voice is a poor first choice for calls that are emotional, sensitive or open-ended, such as complaints, disputes and complex sales. Leave those with people, and let the agent route them quickly.

• A small set of caller intents
• Known information to collect
• A defined action at the end
• High volume or out-of-hours demand

What makes voice harder than chat

Hearing the caller is the first difficulty. Speech recognition is weaker on accents, dialects and names it has met less often, and on your own product names and reference codes. Callers ring from cars, shop floors and streets, on poor lines and speakerphones. Test with recordings of your real callers, add your vocabulary to the recogniser where it allows, and have the agent confirm what it heard without becoming tiresome.

Timing is the second. People talk over the agent, correct themselves mid-sentence and answer before the question ends, so the agent must stop speaking when the caller starts and keep what was said. It must also judge when the caller has finished. If it cuts in too early, it talks over someone reading out a number. If it waits too long, the call drags. And a pause that would pass unnoticed in chat feels broken on a call. The delay is the sum of recognition, the model, any lookups and speech synthesis, so every component counts, and a slow lookup needs a spoken holding line.

Design the conversation for the ear. Keep replies to a sentence or two, offer two or three options at most, and read back anything critical, such as a date, an amount or a reference number, before acting on it.

Ground answers in company knowledge

A voice agent that answers from the model's general knowledge will sound confident and be wrong about your prices, policies and hours. Ground it. Factual answers should come from your own documents and knowledge base through retrieval, and account-specific answers from a live lookup in your systems.

Spoken answers need tighter sources than written ones. A caller cannot skim, and cannot open a citation. Prepare short, current, approved answers for the common questions, and instruct the agent to say it does not know and offer a person when the sources are silent. Verify the caller's identity before disclosing anything about an account, and keep personal information out of what the model is given unless the call needs it.

This is the work we do in voice and language AI: voice AI and NLP systems built on custom RAG pipelines, so that answers come from the client's own material.

Hand over to a person

Every voice agent needs an exit. Callers should be able to ask for a person at any point, in plain words, and get one. The agent should also hand over on its own initiative when it has failed to understand twice, when the caller sounds distressed or angry, when the request is outside its scope, or when an action needs authority it does not have.

Pass the context with the call: who is calling, what they asked, what has been collected and checked. An adviser who starts with that on screen saves the caller from repeating it. Decide what happens when nobody is available, such as a callback within a stated window, and say so plainly on the call. Tell callers at the start that they are speaking to an automated assistant.

• A spoken route to a person at any time
• Automatic handover after repeated misunderstanding
• Context passed to the adviser
• A defined fallback when no one is available

Languages

Language support is uneven. Recognition and synthesis quality differ between languages, and between regional varieties of the same language. A system that performs well in one can be noticeably weaker in another, so ask for evidence language by language and test each one with native speakers.

Plan for mixing. Many callers switch between two languages inside a sentence, or use English words for products and amounts within another language. Decide how the agent detects the language, or if the caller chooses it, and what happens on a switch. Check the synthetic voice with native listeners too: names, numbers, dates and addresses are where it most often sounds wrong. Launch in the languages you have tested, and route the rest to people.

Measure success by resolved calls

The measure that matters is the share of calls the agent resolved: the caller got what they rang for, without a person and without ringing back. Containment alone misleads, because a caller who hangs up in frustration also counts as contained.

Define resolution for each call type, and verify it against your systems, for example a booking that exists or a ticket that closed. Listen to a sample of recordings every week. The numbers tell you how often something failed, and the recordings tell you why.

• Resolution rate per call type
• Repeat calls on the same issue
• Handover rate and reasons
• Abandoned calls and where they dropped
• Response delay as the caller hears it

How to pilot

Choose one call type with high volume and a clear outcome. Gather recordings or transcripts of real calls of that type, handled within your privacy and compliance requirements, and use them to write the conversation and to test recognition. Agree the resolution measure and today's baseline before launch.

Test internally first, with staff calling from mobiles, in noise and in each language. Then send a small share of live calls, or out-of-hours calls only, to the agent, with handover always available. Review recordings daily in the first weeks. Expand the share when resolution holds, and add the next call type only after the first is stable.

What to do next

Pull last month's call reasons and rank them by volume. Pick the top routine one, listen to twenty recordings, and write down what the caller needed, what the adviser looked up and what was done. That is your pilot specification. To discuss a build, contact our team.