I’ve written a lot of head-to-head comparisons — models for business intelligence, for financial modelling, for sales analytics, for code review. They were useful when I wrote them and they have a shelf life measured in months, because every specific claim in them is attached to a version number that has since moved.
The scores expire. The method doesn’t. This guide is the method, separated from the products it was applied to.
If you’re choosing an AI tool for real work, the comparison you read — mine or anyone’s — is a snapshot of a moving target taken by someone whose constraints aren’t yours. What you actually need is a way to run the evaluation yourself, quickly, and reach a decision you can defend in six months.
Start with the job, not the tool
The single most reliable finding across everything I’ve evaluated: the best tool depends on the job, not the brand.
Not “which model is best.” That question has no answer, and pursuing it produces a survey rather than a decision. The useful question is narrow: which tool is best at this specific workflow, for this team, on top of this stack?
So write the job down first, before you shortlist anything. A real one, with edges:
“Our sales ops analyst spends six hours a week reading account notes and producing a pipeline summary. We want that down to one hour, without losing the judgement calls she makes about which deals are actually at risk.”
That’s evaluable. “We want to use AI for sales analytics” is not — it has no failure condition, so nothing can fail it, so every vendor demo will appear to succeed.
The discipline here is the same one I apply to agent work: define the contract before you evaluate the implementation. If you can’t state what success looks like in a sentence with a number in it, you’re not ready to compare tools yet.
Public benchmarks measure someone else’s problem
Benchmark scores are real measurements of real things. They’re just not measurements of your work.
They’re also the most gameable number in the industry — heavily optimised against, sometimes contaminated by training data, and reported by parties with an interest in the result. A two-point difference on a public leaderboard tells you approximately nothing about which tool will handle your account notes better.
Build a private eval set instead. It’s less work than it sounds:
- Take 20–30 real examples of the actual task, pulled from your actual data.
- Include the awkward ones. The ambiguous account note, the malformed export, the edge case everyone knows about and nobody documented.
- Have someone qualified write down the correct output for each. This is the expensive part and the part that pays for itself.
- Run every candidate against the same set. Score them the same way.
Thirty examples of your own work will tell you more than every public benchmark combined, because they measure the thing you’re actually buying. And the set keeps working: it becomes your regression suite when a vendor ships an update, which is when you’ll want it most.
This is the same principle as the golden-question suite in Why LLMs Are Bad at Big Data. Known-correct answers, applied consistently, are how you convert opinion into measurement.
Your existing stack is half the answer
Capability differences between the frontier tools are usually smaller than the integration difference, and integration is where the cost lands.
A team already inside the Google ecosystem gets a materially different result from a Google-native tool than a raw capability comparison would predict — not because the model is better, but because the data is already there, the auth already works, and nobody has to build a connector. The same logic runs the other way for a Microsoft-heavy organisation.
When I’ve compared tools for business intelligence work, the recommendation has repeatedly come down to stack fit rather than model quality, and the honest version of that finding is: the second-best model in the right ecosystem beats the best model in the wrong one, once you count the integration work.
So evaluate the whole path, not the model in isolation. Where does the data live? Who owns the connector? What happens at auth? What breaks when the schema changes? A capability advantage that requires six weeks of plumbing to access is not an advantage this quarter.
Readiness is usually the constraint, not capability
Here’s a pattern I’ve now seen enough times to state plainly: most AI projects don’t fail because the model was bad. They fail because the systems around it were messy.
Adoption is high nearly everywhere. Readiness is not. The data is inconsistent, the definitions are contested, the process has undocumented exceptions, and the “one hour a week” saving assumes a clean input that doesn’t exist.
Which means part of any honest evaluation is evaluating yourself:
- Is the input data consistent enough to automate against?
- Does the team agree on what the output should contain?
- Is the current process documented anywhere other than in one person’s head?
- Is there a clear owner when it produces something wrong?
If several of those are no, the tool choice is not your bottleneck, and switching vendors won’t help. That’s an uncomfortable finding to deliver and it’s usually the most valuable output of the exercise.
Cost per outcome, not cost per seat
Vendor pricing is quoted per seat or per token. Neither is the number you care about.
The number you care about is cost per completed unit of work — per summary produced, per review completed, per ticket resolved to an acceptable standard. That figure includes retries, the cases that fall back to a human, and the review time on the output.
Run that calculation and the ranking often inverts. A tool at twice the token price that needs half the retries and less correction is cheaper per outcome. A cheap tool whose output always needs checking has simply moved the cost onto a salary line where it’s harder to see.
Watch specifically for hidden costs on both sides of the ledger: the review time on generated output, the cost of a wrong answer that reaches a customer, and the engineering time to maintain the integration. Those rarely appear in a pricing page and frequently dominate the comparison.
Distrust the productivity number
Every vendor has a percentage. Fifty percent faster. Thirty percent more output. These are not fabricated, and they’re also not transferable.
Two problems, both structural.
The measurement is usually partial. A tool that makes writing code faster may generate more code to review, more to maintain, and more to eventually delete. Measure the whole cycle — from request to shipped and stable — or you’re measuring the fast part of the process and ignoring where the work moved.
The comparison group is rarely fair. Gains are typically measured on teams that opted in, on tasks selected for suitability, in the enthusiastic phase. Your median team on a Wednesday is a different sample.
Measure your own, on a small number of things that are hard to game: cycle time end to end, rework rate, escaped defects, and how often output is used without modification. That last one is my favourite, because it’s a single number that captures whether the tool is actually doing the job or just producing a first draft someone else finishes.
Specialised beats general more often than you’d expect
In production work, a narrow tool built for one task frequently outperforms a more capable general model, because the specialised tool has the domain constraints baked in — the validations, the formats, the conventions that a general model has to be told about every time.
The tradeoff is lock-in and a ceiling. The general model does the next task too; the specialised one doesn’t.
A reasonable default: specialised for the high-volume, well-defined, repeated workflow where quality is measurable. General for exploratory work where you can’t specify the task in advance. Most organisations need both, and the mistake is buying one and expecting it to cover the other.
How to actually run it
Two to three weeks, time-boxed. The failure mode of AI evaluations is that they expand indefinitely while the market moves underneath them.
- Write the job down. One workflow, one paragraph, with a number in it.
- Build the eval set. 20–30 real examples with known-correct outputs.
- Shortlist three. No more. Include at least one specialised option and one that fits your existing stack.
- Run all three against the same set. Same prompts, same scoring, one person scoring.
- Cost it per outcome, including retries and human review.
- Pilot the winner with real users for two weeks. Measure cycle time and how often output is used unmodified.
- Decide, and write down why.
That last step is the one people skip and the one that pays off later.
Write down what would change your mind
Record the decision with its reasoning, and specifically the assumptions it depends on: “we chose this because our data is in Google’s ecosystem and the volume is under X per month.”
Then you get two things. When someone asks in six months why you’re on this vendor, there’s an answer. And when an assumption changes — the volume triples, the data moves, a competitor closes the gap — you know to re-open the question, because you wrote down what the decision was resting on.
Re-evaluate on a trigger, not a calendar. Annual reviews of AI tooling are simultaneously too frequent to be useful and too slow to catch anything. Better triggers: a pricing change, a capability you were waiting for shipping, your volume crossing a threshold, or the eval set starting to fail on a vendor update.
Keep the eval set. It’s the durable asset from the whole exercise — worth more than the decision it informed, because you can re-run it in an afternoon.
The checklist
- The job is written down as one workflow with a measurable target.
- A private eval set of 20–30 real examples exists, with known-correct outputs.
- Public benchmark scores are not doing any load-bearing work in the decision.
- Integration cost and stack fit are evaluated, not just model capability.
- Your own readiness — data consistency, agreed definitions, clear ownership — has been assessed honestly.
- Cost is calculated per completed outcome, including retries and review time.
- Vendor productivity claims have been replaced with your own measurements.
- At least one specialised option was considered alongside the general models.
- The evaluation was time-boxed.
- The decision is recorded with the assumptions it rests on.
- Re-evaluation triggers are defined, and the eval set is kept.
What to do this week
Pick the workflow you’re most tempted to point a tool at. Collect twenty real examples of it — actual inputs, from actual work, including the messy ones.
Then have whoever does that job today write the correct output for each.
That set is the whole evaluation. Everything else is running candidates against it and doing arithmetic. And if you find that your team can’t agree on the correct output for twenty examples of their own work, you’ve learned the most important thing available this week — and it has nothing to do with which model you buy.
Worked examples of this method applied to specific tools, at specific points in time: business intelligence, sales analytics, financial modelling, and code review. Read those for the shape of the analysis; check the current versions yourself before trusting any specific score.