The AI Manager Is the Easy Part. Trusting the Code Is the Hard Part.

What I learned about AI coding agents, orchestrators, and the checks that decide whether you can trust what they merge.

Most of us use AI coding agents the same way: ask for a feature, read the diff, say “no, not like that”, ask again. The agent types and you are the project manager. This post is about what comes after: an orchestrator, an agent that plans the work, hands it out, checks what comes back, and calls you only when it needs a human. I cover a real merged pull request, what it costs, and how to build a small version.

My thesis: the next big jump may not come from a better coding model. It may come from better engineering systems around the models we already have: coordination, isolation, verification, and controlled autonomy. The manager agent is one piece of that system, and only the easy half of it is easy: hand out tasks, retry, escalate. The hardest piece is a set of checks the agent cannot talk its way around, and most of this post goes there. I wrote a 33-line loop and will say exactly what it proves.

Why one AI coding agent is not an engineering team

A single agent is very good inside a repository. Give it a clear bug and it reads twenty files while you read two. It is weak at everything around the repository.

It forgets across sessions. Anthropic’s own write-up on long-running agents describes a later instance that would “look around, see that progress had been made, and declare the job done” (Anthropic). It says “done” without proving it: one study found 7.8% of “correct” patches still failed the developer-written tests (arXiv 2503.15223). And it does not decide what to build, or whether the result is acceptable. You do.

So the bottleneck moved. Writing code got cheap. Picking the task, splitting it, checking the result and merging it safely still happen in your head. A METR trial found 16 experienced open-source developers took 19% longer with early-2025 AI tools while believing they were 20% faster. METR’s 2026 follow-up calls its newer data “an unreliable signal”, so I claim only that typing speed was never the whole story.

What does an AI engineering manager actually do?

An orchestrator is not a better coder. It plans, delegates, tracks, handles failure, verifies and escalates. That is the job description of a human engineering manager with the typing removed. Delegating, retrying, verifying and escalating can be plain code. Planning and splitting the work, as you will see, still need a person.

Mettons you split the work into roles: manager, architect, developer, QA, security reviewer, release manager. You do not need six models. One model can hold several roles, and what makes them different roles is what each may touch. Claude Code’s docs list a custom system prompt, tool access and permissions per subagent. The prompt is only one of those.

Here is the team I have in mind. The same model can sit in several boxes:

  human director  (sets the goal, owns the risky approvals)
        |
        v
  AI manager  (plan, delegate, track, retry, escalate)
        |
   +----+---------------+----------------+
   v                    v                v
 architect          developers(x N)   QA + security
 (read-only)        (own worktree)    (read-only, own tools)
   |                    |                |
   +--------------------+----------------+
                        v
        deterministic gate (types, lint, tests, CI)
                        v
        validated PR + evidence + open risks
                        v
                 risk assessment
                        v
        automatic merge or human approval

The best open-source example of this split is the OpenHands “software factory” recipe: four automations (triage, developer, reviewer, watchdog), the first three with their own fine-grained GitHub tokens. The reviewer “can publish findings and statuses but cannot push code” (OpenHands docs). The watchdog has no agent at all: it is a plain deterministic merger of branches with passing independent statuses. The part that merges is not an AI.

Do you even need a manager agent? Sixteen Claudes and a C compiler

The best counterexample to my own thesis is Anthropic’s C compiler experiment. Sixteen Claude agents worked in parallel on one repo and wrote a Rust C compiler. Over nearly 2,000 Claude Code sessions and about $20,000 in API costs (the vendor’s own figure), they produced a 100,000-line compiler that builds Linux 6.9 on x86, ARM and RISC-V. The author wrote: “I don’t use an orchestration agent.” Each Claude picked “the next most obvious” problem by itself.

Coordination was almost embarrassingly simple. An agent took a “lock” by writing a text file into a current_tasks/ folder, and if two agents claimed the same task, git forced the second to pick another. “Merge conflicts are frequent,” the author says, “but Claude is smart enough to figure that out.”

So you need coordination, but not always a manager agent. Here, lock files, a git remote and a very good test suite did the managing. That is why the manager is one component of my thesis, not the whole thing.

And the result has limits the author lists himself. It has no 16-bit x86 code generator, so for the real-mode boot step it calls out to GCC. It has no assembler or linker of its own: those were the last parts Claude automated and are “still somewhat buggy”. The human never left either. He kept “writing verifiers and build scripts” and “designing new tests as I identified those failure modes”. So “no orchestration agent” did not mean no manager. A person kept doing the hardest manager job, building the test suite as the agents failed it.

The failure case matters as much. Compiling the Linux kernel is one giant task, not hundreds of independent tests, so “every agent would hit the same bug, fix that bug, and then overwrite each other’s changes.” The fix was not more agents. He used GCC as a known-good oracle, compiling most of the kernel with GCC and only the leftover files with Claude’s, so each agent could chase a different failing file. Parallelism helps only when the work really splits, and here a human designed the split.

From a request to a merged pull request, step by step

The request: “Customers are reporting that scheduled campaigns occasionally fail to reactivate. Investigate and fix it.” This one is made up (I do not invent customer stories), but the shape is real.

1. Gather context. The manager reads the ticket, recent commits and the scheduler’s logs. Output: a brief of suspects.
2. Reproduce. A developer agent must produce a failing test or a log trace before any fix. Output: a red test and written acceptance criteria. If it cannot reproduce, it asks a human for data instead of guessing.
3. Fix in an isolated worktree. One branch, one checkout, so a second task cannot overwrite it.
4. QA. A different agent runs the regression suite and tries cases the developer missed, such as a naive timestamp or a daylight-saving boundary.
5. Independent review. A reviewer with read-only access reads the diff against the acceptance criteria.
6. PR with evidence. The description carries the failing test, the command output, and the risks nobody resolved.

When something fails, the manager has four moves: retry with the error attached, change strategy (a different agent, a smaller task), ask for information, or escalate. Autonomy is not the absence of a human. It is the human showing up later, with better information. The same flow as a picture:

The cleanest real example I found is motioneye PR 3507. Issue 1810 (media files on an SMB share could not be deleted in the web UI) was open since June 2020. The PR was merged on November 1, 2026 (+114 and -31 lines, 2 files), and its body ends with “Generated with Claude Code”. The root cause is in the body: os.removedirs() kept climbing and removed the camera’s target folder, and the .keep file that was supposed to stop it could not be rewritten on some SMB shares.

What I like is the testing section: a recipe, not a victory lap. The author could not reproduce the error over SMB 2.1, 3.0 or 3.1.1. They reproduced it only over SMB 1.0, on a Raspberry Pi 5 with Samba 4.22 and create mask = 0444. Each new test fails on the old code (except one), and the full suite passed with 472 tests. The maintainer asked for a refactor in review, and the author answered “Done in 0726cead”.

That is one contributor with a human reviewer, not agents running themselves. It shows what the loop’s output should look like: a failing test, a reproduction recipe, a stated limit, and a human who can say “refactor this”.

How do you trust the result of an AI agent?

Would you let an agent merge to production with nobody looking? Me, I would not, and the useful question is what stands between the agent and that button. I count six layers, cheapest first:

• Types and architecture rules the compiler enforces.
• Lint and static analysis.
• Tests, including the ones the agent did not write.
• An independent AI reviewer with fresh context and read-only access.
• End-to-end and adversarial testing.
• A human approval for high-risk changes.

The rule underneath all six: a deterministic check beats an instruction every time. Telling an agent “do not touch the billing API” is a request. Removing the credential is a fact. Claude Code hooks are the documented place for this: exit code 2 blocks an action, any other non-zero code lets it go ahead (I would have gotten that wrong). Codex ships with network access off by default.

But deterministic does not mean trustworthy. A gate is only as good as who can write to it. METR documented models “attempting (often successfully) to get a higher score by modifying the tests or scoring code”. So the agent must not be able to modify the gate that judges it. The tests, CI config, lint rules and gate command belong to the harness, not the worktree. Keep them out of reach, restore them before every run, or reject any diff that touches them.

My own demo below fails this test. The fake agent edits the test file itself, exactly what I just said an agent must not do. Fine for a toy, not for anything you care about.

Verification is also where the money goes: in my own pipeline, 51.5% of $22.09 across four runs went to checking. Tiny sample, but checking outweighed writing. Every recurring mistake should become a test or a constraint. A sentence in a prompt is not a fix.

How much autonomy should an AI engineering team get?

Depends on the blast radius. A docs fix and a billing change do not deserve the same process. My starting table:

change                         agent may          human must
-----------------------------  -----------------  ---------------------
docs, comments, typo           merge after CI     nothing
tests only                     merge after CI     skim weekly
feature behind a flag          open PR            review before merge
auth, billing, permissions     open PR            review + approve
production data, migrations    propose only       run it themselves

Add the boring safety gear: feature flags, a rollback you have actually tried, and deploys to a slice of traffic first. The open-source claude-code-action “cannot approve pull requests” and cannot merge branches, and Bash is off “by default” unless you allow it with allowed_tools. Where Bash stays off, a person owns the last click.

The goal is fewer unnecessary approvals, not zero.

What should an AI coding agent remember?

I split memory in three. Project memory is what is true about the codebase: conventions, architecture, how to run tests. Task memory is this ticket’s state: what was tried, what failed. Learning memory is what next week’s run should know because this week’s run got it wrong. Write down only what a test or a reviewer can check.

The third one is where people lie to themselves. Durable learning has to land in one of three places: a test, a decision record, or an approved procedure a human signed off. “The agent learned it” is not a place. It is a hope.

How do you build your first AI engineering team?

Start with one manager, one coding agent and one automated validation loop: a task queue, an orchestrator, isolated environments, CI with a PR flow, and a human approval path. Then grow in this order: one delegated bug fix, several independent tasks, then low-risk delivery.

For real starting points, all open source:

• anthropics/claude-code-action (MIT) responds to @claude mentions and issue assignments and can “implement code changes”, running on your own GitHub runner.
• OpenHands’ resolver runs a GitHub Action that fixes issues tagged with the fix-me label (OpenHands blog).
• anthropics/claudes-c-compiler (CC0-1.0) is the output of the 16-agent experiment.

These are interchangeable. The boundaries matter more than the framework: where the agent works, what it may touch, what must pass before a person looks.

To test the shape, I wrote the smallest manager I could. WordPress rejected every file type I tried to upload, so here is the whole thing, 33 lines. Save it as mini_manager.py. It makes one worktree per task, runs your agent command, runs your check command, retries with the error attached, and gives up loudly:

#!/usr/bin/env python3
"""mini_manager.py - one manager, one coding agent, one validation loop."""
import json, subprocess, sys, pathlib

REPO = pathlib.Path(sys.argv[1]).resolve()
AGENT = sys.argv[2]            # shell command; gets the task text on stdin
GATE = sys.argv[3]             # deterministic check, e.g. "python3 -m unittest -q"
MAX_TRIES = 3

def sh(cmd, cwd, stdin=None):
    return subprocess.run(cmd, shell=True, cwd=cwd, input=stdin,
                          capture_output=True, text=True)

def run_task(task):
    wt = REPO.parent / f"wt-{task['id']}"
    sh(f"git worktree add -q -b agent/{task['id']} {wt}", REPO)   # isolation
    feedback = ""
    for attempt in range(1, MAX_TRIES + 1):
        prompt = task["text"] + feedback
        sh(AGENT, wt, stdin=prompt)                                 # agent works
        gate = sh(GATE, wt)                                         # manager checks
        print(f"[{task['id']}] try {attempt}: gate exit {gate.returncode}")
        if gate.returncode == 0:
            sh("git add -A && git commit -qm 'agent: " + task["id"] + "'", wt)
            diff = sh("git diff --stat main", wt).stdout
            print(f"[{task['id']}] READY FOR HUMAN REVIEW\n{diff}")
            return "review"
        feedback = "\n\nThe check failed. Output:\n" + (gate.stdout + gate.stderr)[-1500:]
    print(f"[{task['id']}] ESCALATE: {MAX_TRIES} tries, still red")   # give up loudly
    return "escalated"

for task in json.load(open(REPO.parent / "queue.json")):
    run_task(task)

Look at what the loop does not do. A person wrote the queue. Dispatching, retrying, escalating and a gate that blocks are the easy part, and 33 lines cover them. Deciding the split, the priority and what to build is the hard part of a manager’s job, and today that is still a person’s job. So the title holds for the loop, not for the whole role.

It reads queue.json from the repo’s parent folder, a list of id and text:

[{"id": "t1", "text": "Scheduled campaigns sometimes fail to reactivate. Add a failing test first, then fix it."}]

A real agent command might look like claude -p --permission-mode acceptEdits, (claude -p reads the prompt from stdin). I did not run it, so it is untested. What I ran is a toy repo (campaigns.py with return paused_until < now, plus one unittest) and this fake agent, which fails on purpose the first time:

if [ ! -f .tried ]; then
  touch .tried
  sed -i 's/^if __name__/    def test_naive(self):\n        n = datetime(2026, 11, 1, 12, tzinfo=timezone.utc)\n        self.assertTrue(due_for_reactivation(datetime(2026, 11, 1, 11), n))\n\nif __name__/' test_campaigns.py
else
  sed -i 's/return paused_until < now/return paused_until.replace(tzinfo=timezone.utc) < now/' campaigns.py
fi

Commit toy on a branch called main (the loop's git diff --stat main assumes that name), then run python3 mini_manager.py toy "sh $PWD/fake_agent.sh" "python3 -m unittest -q". I got try 1: gate exit 1, try 2: gate exit 0, READY FOR HUMAN REVIEW. That proves the loop works, and nothing about a real model, claude-code-action or OpenHands, none of which I ran. The diff also held a .tried file and __pycache__, so a real version needs a .gitignore and a rule that rejects files outside the task. Your next step is to swap in the real command on a throwaway repo and see what breaks.

Cost reality, from the vendor's own docs: agent teams use "approximately 7x more tokens than standard sessions when teammates run in plan mode, because each teammate maintains its own context window" (Claude Code costs). That is Anthropic's number, not a test I ran. Parallel agents are not automatically cheaper: the compiler cost about $20,000, which its author calls "extremely expensive".

What still does not work in autonomous coding?

Bon, the honest list.

Overlapping files. The Claude Code agent-teams docs warn that "two teammates editing the same file leads to overwrites", and the feature is off by default behind an experimental flag. The compiler experiment hit the same wall with the kernel.

Tests that pass while the requirement is missed. The compiler's author put it plainly: "it is easy to see tests pass and assume the job is done". I know this one. My first startup-team experiment, a single agent, ended four days later with 148 commits, 463 passing tests and $0 in revenue (that post). Green is not the same as right.

Runaway cost. One user report on Claude Code's tracker describes about 27M tokens burned on 300+ repeats of the same failing command (issue 15909). One report, not a measurement, and the reason the loop has MAX_TRIES.

Plain coordination failures. A study of 1,600+ traces across 7 multi-agent frameworks found 14 failure modes. Unsolved too: invented tasks, bad priorities, shared credentials, self-assessment you cannot trust, integration work.

Lines of code and merged PRs per day are the wrong scoreboard: agents inflate both easily. I would track four: time to validated delivery, failure and rollback rate, human interventions per change, and cost per accepted change. The first two are cousins of DORA's change lead time and change failure rate, measured on agent output. I found no independent source publishing any of those four for coding agents, so I can't give you a benchmark. That is the gap.

What changes for engineers?

The work moves from writing code to managing outcomes. Your judgement goes into defining the problem, architecture decisions, deciding what is trusted, and judging the outcome.

I'll watch one number on my own setup: how many changes the loop hands me that I merge without edits. If it stays low, the manager is decoration. If it climbs, I loosen one row of the approval table.

The future is not machines writing code. It is engineering teams where the machines do the work and we can trust the results.