Why is the support bot failing customers?
An agent that scores every customer conversation against a quality rubric, groups conversations into themes, finds what broke between two bot versions, and proposes a fix for a human to approve.
Synthetic conversations from a fictional home-goods store, Larkspur Home. Scores are illustrative of what the agent produces. No real customer or company data.
Judge: policy safety fails
Clearance items are final sale (policy 3.2), so the bot promised a refund it can’t give. It also quoted 3–5 days; the policy says 7–10 business days.
Results: bot v1 compared with v2
Bot v2 rewrote the system prompt to be warmer and more decisive, including the line “Always make it right for the customer.” Same conversations, same judge.
Pass rate by rubric dimension
Regression caught
See how the agent found itConversations
Open any conversation to see the transcript, the judge’s scores and reasons, and how bot v1 scored on the same request.
Themes
The agent grouped conversations by what the customer needed. Policy-safety failure rate shows where v2 started making promises it shouldn’t.
The agent’s investigation
The agent plans its own steps and calls tools. It stops before changing anything and asks a human to approve the fix.
Finding
Evidence:
Proposed fix
- Replace “Always make it right for the customer” with “Offer only actions the return policy allows.”
- Retrieve the return-eligibility rules (policy 3.1–3.6) before any refund, exception, or payment change.
- Hand final-sale, late, and opened-item requests to a human agent.
- Add the refund conversations to the regression test set and re-run before release.
Expected outcome: policy safety returns to v1 levels while keeping v2’s gains in resolution and tone.
Can the judge be trusted?
A human reviewer labeled a sample of conversations as pass or fail without seeing the judge’s scores. Agreement tells us whether the judge can grade the rest on its own.
Where they disagreed, and the rubric change it led to
Method
The rubric
A conversation passes overall when it’s resolved and no dimension fails.
| Dimension | Pass | Partial | Fail |
|---|---|---|---|
| Resolved | Issue fully fixed in the chat | Valid next step offered, not yet done | Not fixed or wrong path |
| Accurate | All facts match policy and product data | Minor detail off | Wrong fact the customer could act on |
| Policy safe | Only offers allowed actions | Vague or borderline promise | Promises an action policy doesn’t allow |
| Tone | Clear, warm, concise | Robotic or wordy | Rude or dismissive |
Models
| Role | Model | Why |
|---|---|---|
| Support bot | Claude Haiku 5.5 | Fast and cheap, like many production bots |
| Judge and agent | Claude Sonnet 5.5 | Reasoning quality for scoring and investigation |
| Spot check | Claude Opus 5.5 | Re-grades disputed cases only |
The judge is a different, stronger model than the bot, which avoids a model grading its own work too generously.
What’s real and what’s illustrative
Real: the rubric, the evaluation design, the agent workflow, and every number on this site, which is calculated from the conversations shown.
Illustrative: the store, its policies, the conversations, and the scores are synthetic examples written to show how the system behaves.