I built the quality gate last, and I built it in an afternoon.
The agent that writes some of the posts here does the interesting work first: it researches a topic, files what it learns into a note vault, picks an angle, drafts a piece. The gate came after, as the safety rail. A second process reads the draft against a written test for whether the writing sounds human, with no memory of having written it, and refuses to publish if it doesn’t pass. Cheap insurance. That was the mental model.
Last week I finally added up the API bills by phase. Across four complete runs the pipeline spent $22.09, and 51.5% of it went to the gate. Everything that actually produces a draft — the research, the angle, the writing — came to 42.8%.
The safety rail is the main expense. Not close, either, and not close on any individual run: 49%, 52%, 58%, 48%.
Where it actually goes
The headline number bundles two things and the split matters more than the total. Grading is pure judgement, one process reading and returning a verdict, and that came to 23.7% on its own. The rest is revision — generative work, real writing, but writing that only happens because the gate rejected something first.
You can argue revision belongs in the writing column. Mettons. Then the number becomes “a quarter of my budget is spent deciding whether the work is good enough,” which I don’t find much more comfortable.
The model concentration was the part I hadn’t seen coming. The biggest model ran 14.2% of the calls and 51.2% of the cost, and over half of its spend was inside the gate. Meanwhile a small cheap model handled three quarters of every call for a quarter of the money. The budget question in this pipeline turned out to have almost nothing to do with how many calls it makes. The whole question is which four or five steps get the expensive model, and right now the expensive model is mostly employed as a critic.
One run in the four spent $2.80 and published nothing at all. The draft failed three cycles and got escalated instead of posted. That’s 12.7% of the sample producing zero output, which is exactly what a gate is supposed to do and is still a line on the bill. Across the five posts that did get out, it works out to $4.42 each.
Why I think this generalizes
Almost every conversation about what AI systems cost is a conversation about generation. Token pricing, context windows, which model per million. All of it is about the making.
But if you’re running anything with consequences attached, you don’t ship raw model output. There’s a check: a grader, a validator, a second model reviewing the first, a schema assertion, a human. That check is where the reliability lives, and I’d bet most teams have never costed it separately because it isn’t a line item anywhere. The cost hides inside “the pipeline.”
My guess, and I’ll flag it as a guess, is that a lot of people who’ve built a careful agent are running a verification bill they’ve never looked at, and it’s bigger than they’d say if you asked them to estimate it cold. Mine was, and I wrote the thing.
The case against reading too much into this
Four runs. That’s the sample, and it’s enough to see a pattern that repeats every time and nowhere near enough to put an interval around it, and I’m not going to pretend otherwise on a blog whose whole argument is that other people’s numbers are underpowered.
Worse, 17.1% of the tokens in the sample have no dollar figure attached, because only spend billed to an API key gets priced and some of the work runs on a subscription. If those unpriced calls sit disproportionately in one phase, every percentage above moves. They look spread out. I can’t demonstrate it at this sample size, so the shares are good to a few points and not to the decimal.
And the system was changing underneath the measurement. Cost per run dropped from $9.20 to $2.80 over four days, a 70% fall, with no change in what I was asking it to do. That was prompt caching and cutting redundant research calls. Anyone building an annual budget off the first run would be out by a factor of three.
Worth being concrete about what that money buys, because “quality gate” is a phrase that can mean nothing. The test it grades against is a written document describing what machine-written prose looks like. Repeated sentence openers. A header every 150 words. Sets of three everywhere, hedged sentences that commit to nothing, and closing paragraphs that summarise what you just read. A draft tripping enough of those goes back with the specific failures named, and the pipeline rewrites and tries again. Sometimes three times. One post published here carries its own gate scores in the headline, which should tell you how much deference the thing gets around here.
The strongest objection is the simplest one, though: the gate isn’t overhead. It is the product. A pipeline without it would be cheaper and would publish worse things, and I know what those things look like because I’ve read the drafts it caught. So 51.5% might be correctly spent. The useful question isn’t how to shrink it, but whether it buys what I think it buys, and the honest answer is that a 27.8% revision bill means the drafts aren’t good enough on the first pass often enough. That’s a writing problem wearing a budget costume.
The full breakdown, per run and per phase and per model, with the method and the exclusions written out, is the first entry in a new Measurements section: The Gate Costs More Than the Writing. The section is for original numbers with their method attached, as opposed to the arguments, which stay in the guides.
I’ll run it again at thirty runs and see whether the number holds. It probably moves. I’d like to know by how much.
