📂
Category Archive

Category: Evals

Articles

3 stories
Evals

Your Agent A/B Test Was Broken Before You Ran It

In parimutuel betting your own bet moves the odds you get, so you can't test a strategy against half your bankroll. Agents have the same problem, and almost nobody running…

Dominic Plouffe · 6 min Sep 4, 2026
Two clusters of teal and amber dots on a dark ground, separated by a thin vertical line, with faint curved threads crossing between them.
Evals

Half My Agent’s Budget Goes to Checking Its Own Work

A circle split into two wedges on a dark ground: an amber wedge shaped around a pen nib, and a larger teal wedge shaped around a magnifying glass.

I built the quality gate last, in an afternoon, as the cheap safety rail. Then I added up the API bills by…

Dominic Plouffe · 5 min
Evals

I Rewrote the Test Until My Writing Passed It

A wavy line threaded exactly through every dot inside a rectangular chart frame, while a loose scatter of dots outside the frame is left untouched by it.

Five commits in four days, every one adjusting the grader that judges this blog's drafts, every one triggered by a verdict I…

Dominic Plouffe · 5 min