Before I spent my time on agents and analytics, I spent it building betting systems for horse racing. It is the least fashionable thing on my CV and easily the most useful.
Racing teaches you something that most software careers never force you to learn: being right is not the goal. Knowing how often you’re right — precisely, honestly, in advance — is the goal. Get that second thing correct and you can be wrong most of the time and still come out ahead. Get it wrong and picking winners more than half the time won’t save you.
That distinction has a name, calibration, and it is the single most underused idea in how we deploy AI systems today.
The question nobody asks about a model
When teams evaluate a model, they ask: how accurate is it? What percentage did it get right?
The better question — the one betting forces on you immediately — is:
When this system says it’s 80% sure, is it right 80% of the time?
That’s calibration. Not “is it correct,” but “is its confidence honest.” A perfectly calibrated system that’s only right 60% of the time is enormously more useful than a wildly overconfident one that’s right 85% of the time, provided your decisions have costs. Because with the calibrated system you know when to act, and with the overconfident one you don’t.
Accuracy tells you how the model performs on average. Calibration tells you when to trust it. Those are different questions, and only one of them helps you decide what to automate.
Why accuracy is the wrong target
Consider two systems reviewing transactions for fraud.
System A is right 85% of the time and reports high confidence on nearly everything, including the cases it gets wrong.
System B is right 70% of the time, but its confidence tracks reality: when it says 95%, it’s right about 95 times in 100; when it says 55%, it’s close to a coin flip and says so.
System A has better headline accuracy. System B is worth more, and it isn’t close.
With System B you can route: auto-approve above 95%, send the middle band to a human, auto-reject at the bottom. You’ve turned an imperfect model into a reliable pipeline, because the confidence score carries information. You can compute the expected cost of each threshold and choose deliberately.
With System A you can do none of that. Its confidence is decoration. Every case has to be treated the same, which means treating all of them as untrusted — and a model you can’t selectively trust is a model you can’t build on.
This is why “our model hit 92% on the benchmark” tells me almost nothing about whether you can ship it.
The favourite–longshot bias
Racing markets have a well-documented distortion. Bettors systematically overbet longshots and underbet favourites. The 50-to-1 horse is priced as though it wins more often than it does; the strong favourite is priced as though it wins less often than it does.
The reason is human, not mathematical. A longshot pays a story. People buy the possibility of a dramatic outcome and quietly overpay for it, in aggregate, race after race, for as long as anyone has been keeping records.
Machine learning systems have their own version of exactly this pathology, wearing different clothes.
Models trained to be helpful and agreeable — which is most of the ones we’re deploying — are systematically overconfident at the top of their range. They report near-certainty far more often than near-certainty is warranted. And they’re reluctant to occupy the honest middle, because “I’m about 60% on this” reads as unhelpful and gets trained away.
The distortion has the same root cause as the racing one: the confidence isn’t priced against outcomes. It’s priced against what sounds good. And a system whose confidence reflects what sounds good rather than what happens is, in betting terms, a market you can be exploited by.
Edge is a difference, not a level
Here’s the reframe that took me longest to internalise, and it changes how you look at every model output.
In betting you never ask “will this horse win?” You ask: is my probability different from the market’s, and am I confident enough in that difference to act?
Edge isn’t being right. Edge is the gap between your estimate and the prevailing one. You can lose the majority of your bets and profit handsomely if you’re only betting when that gap is real, and sizing according to how real it is.
Translated to systems: a confidence score is a price. Not a verdict. It’s an assertion about how often this class of prediction turns out correct, and like any price it can be tested against reality and found wrong.
Which raises the operative question for any AI system you’re about to trust: what’s the baseline it needs to beat? A model that’s 80% confident on a task where the naive default is right 78% of the time has almost no edge, no matter how impressive 80% sounds. Most AI business cases never establish the baseline, which means nobody can tell whether there’s an edge at all.
Sizing: how much to act on a belief
Betting isn’t only about which wagers to place. It’s about how much — and getting that wrong ruins people who pick well.
The Kelly criterion formalises it: stake in proportion to your edge, scaled by the odds. Bigger edge, bigger position. Thin edge, small position. Bet too large relative to your edge and variance eventually takes you out, even when your picks are good. That last part is the one people learn expensively.
The analogue for AI systems is direct, and almost nobody builds it:
Grant authority in proportion to calibrated confidence.
Not one global autonomy setting. A gradient. High-confidence, low-consequence actions execute automatically. Lower-confidence or higher-consequence actions get proposed for approval. The irreversible ones stay gated permanently regardless of confidence, because no confidence level justifies an unrecoverable mistake — the Kelly stake on a bet you can’t afford to lose is zero, whatever the edge.
This connects directly to the progressive rollout in The Agent Reliability Handbook. Calibration is what tells you where the thresholds belong. Without it you’re guessing, and the guess is usually far too generous.
The discipline of the pass
The habit that separates people who make money in betting markets from people who don’t is not picking ability. It’s the willingness to sit out.
Most races offer no edge. The correct action is nothing. That’s a genuinely difficult discipline, because doing nothing feels like failure — you did the analysis, you have an opinion, and passing feels like waste.
Every incentive in an AI system pushes the same wrong way. The model is trained to respond. The product wants an answer on screen. “I don’t know” tests badly with users and looks like a bug in a demo.
So abstention has to be built deliberately, as a first-class output with its own success metric, or it will not exist. Measure it. A system that never abstains isn’t confident — it’s uncalibrated, and you simply haven’t noticed yet.
I made this same argument about data questions in Why LLMs Are Bad at Big Data, and it’s the same principle underneath: the cost of a confident wrong answer is much higher than the cost of no answer. In racing that asymmetry is enforced by your bankroll. In software nothing enforces it, so you have to.
How to actually measure it
Calibration sounds abstract. It’s arithmetic, and you can start this week.
Bucket and compare
Take every prediction your system made with a confidence attached. Sort into buckets — 50–60%, 60–70%, and so on. For each bucket, compute what fraction actually turned out correct.
Plot predicted confidence against observed accuracy. A perfectly calibrated system sits on the diagonal. Points below the line mean overconfidence; above means the system is underselling itself.
Almost every system you plot for the first time will sit below the line at the top end. That’s the finding, and it’s actionable immediately — it tells you your 90% threshold is really a 75% threshold, and your routing rules are wrong.
Score it properly
The Brier score gives you a single number: the mean squared difference between predicted probability and actual outcome. Lower is better. It rewards being confident and right, and punishes being confident and wrong far more than being uncertain and wrong.
That asymmetry is the point. It’s a scoring rule that makes honesty the optimal strategy, which is exactly the property you want when you’re deciding how much to trust something.
Then correct it
Miscalibration is usually fixable without touching the model. If your system’s 90% is really 75%, you can learn that mapping from held-out data and apply it — the standard techniques are simple and well understood.
The important part isn’t the technique. It’s that you now have a number that means what it says, which is what makes every downstream threshold defensible.
What this means for what you’re building
Concretely:
Log confidence with every prediction, and log the outcome when you learn it. You cannot measure calibration retroactively if you didn’t store the pairs. This is the one that costs nothing today and is impossible to recover later.
Treat a model’s stated confidence as unvalidated until you’ve plotted it. Verbalised confidence — a model saying “I’m 90% sure” — is particularly unreliable, because it’s generated text rather than a measured quantity. It may correlate with correctness. Check before you route on it.
Set thresholds from the curve, not from intuition. “Auto-approve above 95%” means something once you’ve verified what 95% delivers. Before that it’s a number someone liked the sound of.
Establish the baseline. What’s the accuracy of doing the obvious thing — the existing rule, the majority class, the current process? If your model isn’t clearly beating it, there’s no edge, and the sophistication is decorative.
The checklist
- Every prediction is logged with its confidence and, later, its outcome.
- You have plotted predicted confidence against observed accuracy at least once.
- You know whether your system is overconfident, and by roughly how much.
- Thresholds are set from that curve rather than chosen by intuition.
- A baseline exists, and you know how much the model actually beats it by.
- Abstention is an implemented output with a tracked rate.
- Authority is granted in proportion to calibrated confidence, not as a single global setting.
- Irreversible actions stay gated regardless of confidence.
- Calibration is re-measured after every model change — it does not survive upgrades.
What to do this week
Find one place where your system already produces a confidence score and acts on it. One threshold, one routing rule.
Pull the last few hundred decisions. Bucket them by the confidence they carried. Count how many in each bucket were actually right.
You will almost certainly find that the top bucket is not as good as it claims. That single number — the gap between what your system says and what it delivers — is the most valuable thing you can learn about it, and it costs an afternoon.
Racing gives you that feedback whether you want it or not, because the results are posted and settled within minutes. Software will let you go years without ever checking. That’s the real difference between the two domains, and it isn’t in the mathematics.