“My surname is Tschanz.”
“Thank you, Mr Schanz.”
“No, Tschanz. With a T.”
“Got it, Mr Schanz.”
By the third attempt, the assistant does not need to demonstrate patience. It needs to recognise that its current repair method has failed.
Newer voice models can make interaction much more fluid. OpenAI describes GPT-Live as full-duplex and reports human evaluations covering turn-taking, interruptions and conversational flow. That does not remove production conditions: Swiss German, French place names, Bluetooth audio, road noise, unusual surnames, licence plates and unstable connections.
The real reliability question is not whether Voice AI ever misunderstands. It is how the workflow responds after the first misunderstanding.
A Voice AI repair loop needs states, not endless repetition
A weak bot has one fallback sentence: “Sorry, I didn’t understand. Could you repeat that?” If it asks the same thing three times, the conversation has not become safer or clearer.
The repair action should change.
On the first uncertainty, confirm a hypothesis: “Did I hear Tschanz?” On the second, reduce ambiguity: “Could you spell the surname?” Or switch to a different identifier where that is appropriate.
If the critical field remains unresolved, move to human handoff or callback.
Recommendation: do not repeat the same repair tactic three times. For a critical entity, two unresolved attempts are a reasonable pilot threshold before changing strategy or handing over. It is a design starting point, not a universal research-backed number.
Confirmed fact: voice quality includes timing, turn-taking and interruption behaviour; model improvements do not eliminate every language and audio failure mode.
Give errors different weights
Mishearing a filler word is not the same as mishearing Tuesday as Thursday during a booking.
Mark critical entities for each workflow: customer name, phone, date, time, address, amount, licence plate, room type or requested service. Those fields often deserve explicit confirmation before execution.
“I have Tuesday 25 August at 14:00. Is that correct?” is useful because an error has a business consequence.
Contradiction is another strong signal. The customer says, “No, not Bern — Thun,” and the agent later repeats Bern. That user correction is more meaningful than a generic confidence number.
Behaviour is a third signal: repeated “no”, “that’s wrong”, “I already said…”, or the caller starting to explain the same detail differently. Use such patterns to trigger repair or escalation without pretending the system can diagnose the caller’s emotions perfectly.
A repair should reduce or change the task
Use different tools instead of asking for the same audio again.
- Confirm alternatives. “Was that Meier or Meyer?”
- Chunk structured data. Read phone numbers back in small groups.
- Offer another input mode. A text link or keypad entry can be easier for codes and long numbers.
- Switch identifiers. Customer number may be easier than a surname if the workflow allows it.
- Reset the disputed field. Do not keep contaminating context with the value the customer has already rejected.
- Hand over with context. Once the field remains unresolved, move to a person or callback.
The handoff payload matters. Give the human the purpose of the call, confirmed fields, disputed field and repair attempts. The customer should not have to tell the entire story again.
Swiss German and code-switching can fool rigid thresholds
A Swiss call may move from Standard German into dialect, then contain a French surname and an Italian street. A single speech-confidence threshold can generate unnecessary transfers.
Combine signals. Is the entity business-critical? Did the caller explicitly correct the assistant? Has the same value failed more than once? Did language change? Can another input method solve the problem?
Handover also needs an operational mode. During opening hours, live transfer may be best. After hours, create a callback task with priority and preserved context.
Tell the customer what is happening: “I’m not capturing your surname reliably, so I’ll pass this to our team. I’ve already saved the phone number and appointment request we confirmed.”
That admission creates less frustration than pretending the fourth repeat will be different.
Minimum pilot: test names your demo normally avoids
Build an evaluation set with Swiss surnames, place names, licence plates, numbers, background noise, speakerphone and code-switching. Ask testers to correct the agent deliberately and verify whether the corrected value actually replaces the wrong one in downstream state.
Track first-pass success on critical entities, cases resolved after one repair, average repair attempts per call, handover rate caused by understanding failures, calls with three or more repeats of one field, and handovers that include sufficient context for the human.
Do not optimise for the lowest possible handover rate. An assistant that never transfers may simply trap customers in longer failed calls.
The practical conclusion is that misunderstanding deserves its own workflow. A short repair loop, a changed tactic and a respectful handoff protect customer experience better than assuming a stronger voice model will make the edge cases disappear.
Sensitive changes also need caller verification. Our AI phone assistant is designed around bounded actions and explicit handoff.
Sources
FAQ
How many times should a voice assistant ask again?
There is no universal number. Two unresolved attempts on the same critical field is a useful pilot trigger for a different strategy or handoff.
Should every field be read back?
No. Prioritise fields where a mistake changes the person, time, amount or real-world action.
Is low speech confidence enough to transfer?
No. Combine confidence with explicit corrections, repeated instability and business criticality.
What should the human receive at handoff?
Call purpose, confirmed data, disputed field, repair attempts and the requested next step.
What about after-hours handoff?
Create an owned callback or ticket with preserved context and a realistic response expectation.