Evals
-
Your Agent A/B Test Was Broken Before You Ran It
In parimutuel betting your own bet moves the odds you get, so you can't test a strategy against half your bankroll. Agents have the same problem, and almost nobody running an A/B test on one has noticed.
-
Half My Agent’s Budget Goes to Checking Its Own Work
I built the quality gate last, in an afternoon, as the cheap safety rail. Then I added up the API bills by phase. Across four runs it took 51.5% of everything the pipeline spent.
-
I Rewrote the Test Until My Writing Passed It
Five commits in four days, every one adjusting the grader that judges this blog's drafts, every one triggered by a verdict I disagreed with. Finance has been making that mistake with real money since the seventies.