AI-native customer service agent
A previous-generation chatbot produced low-quality answers and avoidable human handoffs. I led a prototype targeting more than one million customers, beginning with twenty conversations and a targeted evaluation framework.
Self-service resolution
User interviews
Target customers
01 The problem
The client's previous-generation chatbot often ended with "I couldn't help," sending customers to a person after an extra failed step. More than twenty interviews with customers and service agents made the priority clear: improve answer quality and reduce avoidable human handoffs before expanding the feature set.
I would ask about my policy in ten different ways, and it would still tell me it couldn't help. Then I'd need to speak to an agent anyway.
Customer
02 What I built
I led more than ten engineers and designers to prototype a retrieval-grounded agent targeting more than one million customers. I built a targeted LLM evaluation framework to diagnose failure modes, then redesigned the query system with RAG and specialist sub-agent routing.
Sanitized screen example to protect client confidentiality
03 Key decisions
Decision 01
Specialist sub-agents over one general agent
A router that hands each question to a sub-agent scoped to one domain.
One general-purpose agent carrying a much broader context across every product domain.
Specialists scored higher on real-question evaluations and user tests, and failures were easier to diagnose. The tradeoff was more routing logic and prompts to maintain.
Decision 02
Fixing retrieval before adding capability
Spending the early sprints on retrieval quality against real customer questions.
Expanding into more use cases and giving the agent more actions to perform.
Interviews prioritized dependable answers, while evaluations identified retrieval as the main failure point. Together, the evidence aligned the team around fixing the foundation first.
Decision 03
Escalating uncertainty instead of forcing an answer
Handing the conversation to a person when confidence fell below a threshold or the answer could not be grounded in an approved source.
Attempting to answer every question in order to maximize apparent automation.
A transparent handoff protected trust and made the system's limits explicit. An uncertain case was safer with a person than with a forced answer.
04 Outcome
Prototype testing with representative users increased self-service resolution by ten percentage points versus the previous-generation chatbot.
The work produced an executive-endorsed multi-year roadmap with eight prioritized capabilities for a product targeting more than one million customers. In hindsight, I would perfect fewer domains before expanding coverage.
Confidentiality note
This project was built within McKinsey & Company. Specific client names, proprietary architectures, and internal tooling details are confidential. What's shared here reflects the publicly describable scope of the work; product screens shown are generic, illustrative recreations, not the shipped product.