How to evaluate voice and chat virtual agents: containment, latency and handoff

Evaluate a virtual agent on verified resolution rather than raw containment, on end-to-end response latency measured on real calls, on handoffs that carry context, and on ongoing QA that scores AI and human conversations the same way.

Published Oct 20, 2026 · Updated Oct 20, 2026 · Author: Contact Center AI Index

Primary research hub: Voice and chat AI agents · Entities: Level AI, Five9, Decagon, Parloa, Sierra, PolyAI

Evaluate a voice or chat virtual agent on four things: resolution (did the customer's problem actually get solved, not just "contained"), latency (how long callers wait for each reply on a real phone line), handoff (whether a human receives the full context when the AI cannot finish), and ongoing quality (whether every AI conversation is scored after launch, ideally on the same rubric as human agents). Vendors such as Level AI, Five9 and Decagon now document testing and QA tools for these stages, but the definitions and thresholds should be yours. This guide gives a vendor-neutral method; our methodology explains how we source claims.

Which metrics matter when evaluating a virtual agent?

A virtual agent (also called an AI agent or voice bot) answers customers without a human. These are the metrics to put on the scorecard:

For the satisfaction row, see our guide to inferred CSAT vs surveys.

Why is containment not the same as resolution?

Containment counts what did not reach a human. A caller who gives up and hangs up, or who calls back an hour later and reaches an agent, can still count as contained. That is why containment alone flatters most deployments.

Track three adjustments:

1. Exclude abandonment. Separate conversations that ended with the customer's goal met from those where the customer dropped off mid-flow.
2. Check repeat contacts. Join AI conversations to any contact by the same customer within your window, on any channel.
3. Report per intent. A 90% containment rate on balance checks and 10% on disputes tells you more than a blended 50%.

Good vendor case studies define their terms. In Level AI's healthcare pilot, the 45% figure is specifically the share of prescription refill calls handled "without agent intervention," and the case reports early pilot results rather than a steady state.

How should you measure latency in a voice agent?

People take turns fast. A study of 10 languages in PNAS found an overall mean response gap of about 208 milliseconds between speakers, with language-specific averages ranging from 7 ms (Japanese) to 469 ms (Danish) (Stivers et al., 2009). Voice agents must transcribe, reason, call systems and synthesize speech, so they are slower than that, and callers notice.

Vendor numbers measure different things. PolyAI's homepage cites a "<300ms" voice-to-voice model response; Level AI's product page cites a latency budget "under 2 seconds." Neither is a measurement on your telephony. To compare fairly:

• Measure end to end on a real line, from the caller's last syllable to the agent's first audible word.
• Report median and 90th percentile. Slow turns usually happen during back-end lookups, so test turns that call your systems.
• Test interruptions (barge-in). Speak over the agent and check that it stops and listens. Five9's June 2026 release lists turn-taking, interruption detection and background noise management as separate capabilities, which is a useful checklist.
• Test noisy and accented speech with recordings from your own callers.

For chat agents, latency matters less than correctness, but measure time to first response and total time to resolution.

What does a good handoff to a human look like?

A handoff succeeds when the human agent can continue without asking the customer to repeat anything. Check four things:

1. Trigger. The agent hands off when confidence is low, the request is out of scope, or the customer asks for a person. In Level AI's Uown case study, callers who need more than the six automated use cases, or prefer a person, are transferred to a live representative.
2. Speed. Out-of-scope callers should not be trapped. In the healthcare pilot, out-of-scope calls dropped from about 90 seconds to under 30 seconds once intent was identified.
3. Context. The human should receive identity status, intent and what was already done. The same pilot sends a Salesforce record with verified identity and medication lists on escalation; Sierra's voice page describes transfers "with AI-generated summaries and skills-based routing"; Five9 describes "context-rich warm handoffs."
4. Measurement. Track how often the human re-asks for information the AI already collected.

How do you test a virtual agent before it talks to customers?

1. Pick intents from data, not intuition. Start with high-volume, low-risk, rules-based tasks. Level AI's March 2026 expansion added Automation Discovery, which scans interaction data to find "the highest-ROI, safest tasks"; you can do a simpler version by tagging a month of transcripts by intent and effort.
2. Build a test set from real conversations, including edge cases: wrong account numbers, partial payments, angry callers, callers who switch topics.
3. Run simulations at scale. Decagon's homepage describes pre-launch simulations; Parloa's Performance Lab is described as a way to "simulate and test AI agents for every update or model change"; Level AI's EnlightIQ module was launched with a testing framework that simulates conversations.
4. Make test calls on production telephony, not just in a browser.
5. Version and roll back. Five9's AI Agent Studio lists versioning, environment promotion and rollbacks. Whatever the platform, insist that every change can be tested and reversed.
6. Set go-live gates: minimum verified resolution, maximum wrong-answer rate, and zero failures on authentication and required disclosures.

How do you keep scoring a virtual agent after launch?

Pre-launch tests cover what you thought of. Production reveals the rest. Score every AI conversation, not a sample.

• Use the same rubric as humans where it applies. Level AI says it scores every virtual agent conversation "against the same QA rubrics and performance benchmarks as human agents" (AI Virtual Agent), and its QA page includes "bot conversations" in its 100% scoring claim. Its Analytics page says it supports "performance benchmarking across human and AI agents."
• Add AI-specific checks: made-up facts, failed tool calls, loops, and refusals of valid requests.
• Use built-in monitors where they exist. Decagon describes Watchtower as "always on QA"; Five9 lists "automated post-call AI evaluations."
• Review failures weekly and feed them back into prompts, workflows and knowledge.

For a vendor shortlist, see best voice AI agent software for customer support and AI agents across the entire customer journey.

Limits of this guide

• Vendor features are as described on vendor pages. We did not test Level AI, Five9, Decagon, Parloa, Sierra or PolyAI, and their latency and containment figures use different definitions.
• The healthcare and Uown results are vendor-published, and the healthcare deployment is an ongoing pilot.
• The human turn-taking figure comes from conversational research, not from phone-based customer service; use it as a reference point, not a target.
• Thresholds are yours to set. We give no universal containment or latency benchmark because none in the sources is comparable across vendors.
• Regulation varies. Outbound AI calling and recording consent rules differ by country and US state; get legal review before launch.