Margot Bellweather is the QA Director at a mid-size property and casualty insurer headquartered in Hartford, Connecticut. Or she would be, if she existed. What follows is a hypothetical conversation with an invented person whose name, we note with some affection, is a misspelling of "bellwether," meaning a leading indicator. Her entire professional crisis is that her leading indicators went quiet. We couldn't resist.
In our imagination, this conversation takes place over coffee in a glass-walled conference room overlooking the Connecticut River. Bellweather has been in claims for twenty-two years, starting as a field adjuster who climbed rooftops to assess hail damage. She moved into quality assurance after discovering she was better at reading patterns across a thousand claims than she was at reading one roof. Eighteen months ago, her firm deployed AI agents across first-notice-of-loss intake and initial claims triage.
She agreed to talk about what happened to her job after the complaints stopped arriving.
You've described the moment you realized something was wrong as "the quarter the dashboard got too clean." Walk me through it.
Margot: I've been reading complaint data for fourteen years. You develop a feel for it. Seasonal rhythms, spikes after storms, the way volume ebbs and flows. About six months after the agents went live on intake, I'm looking at Q2 numbers and everything is just... flat. Complaint volume down 40 percent. Escalation rate down. CSAT holding steady. My first reaction was, great, the agents are working.
My second reaction, about three days later at two in the morning, was: wait, what am I actually looking at?
The CSAT surveys were still being triggered by human-handled interactions. But the humans were now handling maybe a third of the volume they used to. So I wasn't measuring satisfaction across claims. I was measuring satisfaction among claimants who'd already been filtered through the agent and escalated to a person. Completely different population.
A selection bias problem.
Margot: A selection bias problem that looked like stability. That's the nasty part. The number didn't move, so nobody flagged it. But the denominator changed underneath it.
What about the complaint drop itself?
Margot: Right, and this is what kept me up. Research on customer service deflection is pretty clear. Somewhere between 30 and 50 percent of interactions a bot marks as "resolved" aren't actually resolved from the customer's perspective.1 Separately, we know that 56 percent of unhappy customers just... leave. They don't complain. They disappear.2
So when an agent handles first contact and the customer doesn't file a formal complaint, I've got two possible explanations: the agent handled it well, or the customer gave up before reaching anyone who could register the complaint.
My old system couldn't distinguish those. My new system really can't distinguish those.
Because the complaint registration channel itself changed.
Margot: The complaint was never just a complaint. It was a signal. A claimant who called back angry and got transferred to a senior adjuster? That transfer was data. The adjuster's note on the file was data. The fact that it took three calls was data. Now the agent handles the first call, maybe the second, and if the claimant gives up after the second call, I have a completed session log and no complaint. My dashboard says resolution. The claimant says nothing.
And in claims, "says nothing" doesn't mean "satisfied." It can mean "retained a lawyer." It can mean "filed with the state department of insurance." Or it can mean "abandoned a valid claim," which is its own kind of institutional exposure.3
So what do you read now?
Margot: I had to build a completely different sensory apparatus. Which sounds grand, but mostly it meant a lot of spreadsheets and some uncomfortable conversations with our data team.
Session abandonment patterns: where in the interaction does someone just stop? Repeat contact rate by original channel: if someone talks to the agent on Monday and calls the main number on Wednesday, that gap is a story. The characteristics of the human queue, because the cases that reach my adjusters now are harder, and their handle times and rework rates tell me something about what the agent population looks like at the edges.
And trajectory-level review. Not just "did the claim close correctly" but "did the agent follow the right path to get there." Recent research found that over 90 percent of agent runs that passed final outcome checks still had at least one process deviation detected by evaluators.4 So outcome metrics alone are necessary, sure, but they're the tip of something much larger.
That sounds like more work, not less.
Margot: laughs Yeah. Nobody mentioned that in the vendor presentation.
Here's an uncomfortable version of this question. Are you still qualified to judge what you're judging?
Margot: long pause
That's the one I don't love thinking about.
My judgment was calibrated on twenty-two years of reading real claims files, watching how adjusters handled ambiguous coverage questions, seeing which policy language created disputes. I could pick up a file and tell you in ten minutes whether the handling was going to generate a regulatory problem. But now the agent handles, what, 70 percent of initial contacts? I'm not seeing those files the same way. I'm seeing the agent's summary of those files. I'm reviewing what the system tells me about itself.
So I started pulling a random sample every month. Twenty files. I read them cold, from the original submission, without the agent's summary. And I compare my assessment to what the agent did.
It's my calibration exercise. It's how I make sure I can still see.
And what do you find?
Margot: Mostly fine. Occasionally something that makes my stomach drop. A denial that was technically correct but used language that would absolutely trigger a regulatory inquiry if the claimant knew to push back. The agent applied the right rule. It didn't understand the weight of the interaction. There's a difference between a correct decision and a safe one, and that difference lives in twenty-two years of watching what regulators actually care about.
Does your leadership understand this distinction?
Margot: My leadership sees a dashboard that looks great. Complaint volume down, resolution times down, cost per claim down. Those numbers are real. I'm not saying the agents aren't doing useful work. They are.
But when I try to explain that the dashboard might be measuring the channel changing rather than the quality changing, I get polite nods. "The numbers are good, Margot" is a complete sentence to most executives.
I've started thinking of it this way: we used to have a nervous system that ran through every interaction. Claimants complained, adjusters noticed, supervisors flagged. Now we've anesthetized large portions of that nervous system and replaced it with a monitoring screen. The screen shows vitals. But vitals aren't the same as feeling.
What would you tell a QA director at another firm who's about to go through this?
Margot: Start measuring what the system can't see before you lose the ability to see it yourself. Build your independent sample review now, while you still have adjusters who remember what good handling looks like. Track second-contact rates across channels obsessively.
And never let anyone tell you that falling complaint volume is the same thing as rising satisfaction. Sometimes silence is the most dangerous signal there is.
That's a good line. Did you rehearse that?
Margot: I'm a QA director. Everything I say is reviewed for accuracy after the fact.
Margot Bellweather is a fictional character. Her situation is not. The observability challenges described in this conversation reflect documented patterns in enterprise AI deployment, customer service measurement, and claims processing regulation as of late 2026.
Footnotes
-
Research on AI deflection metrics indicates that auditing deflected sessions reveals a calibration gap — a 78% deflection rate often corresponds to a 40–55% actual resolution rate. See https://tianpan.co/blog/2026/05/27/the-deflection-metric-that-lied ↩
-
Industry research estimates that 56% of unhappy customers rarely complain and simply leave. See https://www.abroadworks.com/blog/ai-customer-service-trap ↩
-
Multiple U.S. jurisdictions permit claims reopening well after initial resolution — California for newly discovered damage, New York for material evidence unavailable at original hearing, and the Federal Longshore Act within one year of last payment. ↩
-
Vo, Doshi, and Sheinin (2026) found that 92.6% of enterprise agent runs passing final numerical checks still contained evaluator-detected trajectory deviations. See https://arxiv.org/html/2610.01833 ↩