How to tell if an AI call metric is actually reliable
In short
An AI metric is reliable when you can measure its accuracy on the specific class you send the supervisor, not on the global average. A model answering the same thing repeatedly does not prove it is right: three identical errors also agree with each other.
Every AI call-analysis vendor will hand you an accuracy number. Almost none will tell you which accuracy they mean, and that difference decides whether your supervisor makes good calls on Monday or accuses an agent of something they never did.
What follows are four failure modes I found measuring my own metrics on live collections operations. None of them is fixed by rewriting the prompt, which is exactly what everyone tries first.
1. Consistency is not truth
The most common way to measure an AI metric without a labeled answer key is consistency: run the same call several times and check whether the model answers the same. If it agrees with itself, you call it reliable.
That is reasonable, and it is the only thing available without a golden set. It also has a large hole in it.
In July 2026 my own quality watchdog reported 100% reliability across three consecutive sweeps. The same days the AI provider was down and no run returned a value at all. The system measured how often three runs agreed with each other, and three errors agree perfectly.
3 sweeps
reported as perfect health while the engine was unreachable, generating false alerts that survived a week.
Own measurement, AIM8 Prompt Lab, July 25-27, 2026
2. The average swallows what matters
A collections metric I audited had 97% global reliability. An excellent number in any deck.
That metric sorts each call into several classes. One of them is the one actually sent to the supervisor to act on. Measured on its own, that class sat at 25%.
Own measurement on a collections operation, August 2026. Client not identified.
Three out of four times, the data point reaching the person about to act was wrong. The average hid it perfectly.
3. The four causes demand opposite actions
When a metric goes wrong there are four possible reasons, and mixing them up costs weeks, because the fix for one makes another worse.
| Cause | How it looks | What to do | What NOT to do |
|---|---|---|---|
| Prompt | Fails evenly across all classes | Rewrite the prompt | Swap models before isolating |
| Dilution | Same instruction is right alone, wrong in company | Shrink the batch or move the instruction up | Rewrite the prompt: it was already fine |
| Noise | Small variation with no pattern | Nothing. Live with the error floor | Chase it. Weeks vanish |
| Missing capability | One specific class always fails, the rest do not | Change the engine or narrow the objective | Rewrite the prompt twenty times |
4. Dilution: same prompt, different position
This one surprised me. Call-analysis systems usually send several instructions together in a single request to save cost and latency.
I took one instruction, one call and one model. Without changing a single word, I moved the instruction around:
Own measurement, AIM8 Prompt Lab, July 2026. Small reasoning model.
A “the prompt does not work” case can be 100% structural. Before blaming the text, run the instruction alone and compare it against its performance inside the batch. If it passes alone, it is not the prompt.
5. The question that comes before the prompt
Back to the 25% metric. The cause was neither the prompt nor dilution.
That class required knowing which of the two participants said something. The transcription engine was running without speaker separation. Without it, attribution is not hard, it is impossible. And since the model cannot attribute, it falls back to a default, which in this case was the one that blamed the agent.
Which leads to a design principle I apply everywhere: the model extracts, the code decides. All arithmetic and every combination of conditions moves out of the prompt and into the pipeline, where it is deterministic and auditable.
6. The checklist before you accept a number
- 1
Is this accuracy or consistency?
If it is consistency, you know the model is stable. You do not know it is right.
- 2
Are errors excluded from the count?
A timeout is not a value. If it enters the average, the number is inflated.
- 3
How much of the sample could actually be measured?
Below 80%, the correct output is inconclusive, not a percentage computed over the survivors.
- 4
What is the accuracy of the actionable class?
The one that triggers supervisor action. The global figure is useless here.
- 5
Was the instruction measured alone and in the batch?
If it was only measured inside the batch, you cannot tell text from position.
- 6
Can the engine resolve what the metric demands?
If not, no prompt fixes it. Change the engine or narrow the objective.
7. Narrowing the objective usually ships a better product
When part of an objective is impossible, the answer is not to invent a prompt that will fail. It is to narrow it down to what holds up.
On the same operation, two ways of saying something similar:
| What sounds better | What survives review |
|---|---|
| “The agent accepted the promise without pushing back” | “There are 40 payment promises past 7 days, review them” |
| Wrong 3 out of 4 times | Verifiable against the system |
| Accuses a named person | Points at a batch of cases |
| The supervisor argues with it | The supervisor acts on it Monday |
The small number that survives an audit is worth more than the big one that collapses the first time somebody checks. And for the supervisor the difference is not philosophical: it is whether they open the report again next week.
What an operations director should take away
AI on calls already works well for plenty of things. The problem is not that it gets things wrong. It is that it gets them wrong quietly, backed by an accuracy figure that endorses the error.
Before a metric enters your weekly operations report, ask for three things: accuracy of the actionable class, how much of the sample could be measured, and what happens when the engine does not answer. If all three have an answer, the number is usable.