Most posts about AI team structure draw an org chart and stop there. A box for the head of AI, three boxes under it, a dotted line to product. None of them says who owns the number that tells you whether the thing got better, and that is the only box that matters.
Here is the version I actually use. I am the CTO of Trellis, a small company that builds AI tools for Amazon sellers, and I also run an AI agent team that writes and checks the drafts for this blog. In this post I go through the four seats I would fill, in the order I would fill them, and what the “AI engineer vs ML engineer” question looks like when you have five people instead of fifty. Then I show my own pipeline with numbers I counted on Nov 8, including the part that is embarrassing.
There are two kinds of AI teams. A research team produces models: it trains, it tunes, it publishes. A product team puts models, mostly someone else’s, into a product that a customer pays for. This post is about the second kind, because that is what a startup builds. If you train foundation models on a hundred GPUs, this is not for you.
My short answer: your AI team structure is a list of who decides what, not a picture of who reports to whom. Four decisions, four seats. The seams between them are where the work breaks.
The first two seats: the builder and the person who owns evaluation
The first seat is the builder. This is the person who ships the feature: calls the model, wires the retrieval, writes the prompts, adds the tools, makes it run behind a button a customer clicks. You hire this person first because without them you have a demo and no product. They decide what the feature does and when it is done enough to ship.
The second seat is the one most teams skip, and it is the one I would fill before anything else: the evaluation owner. This person decides what “better” means and has the power to say no to a release. Not a dashboard. A person with a veto.
Why a person? Because “does the new model or the new prompt help” is never answered by a single number you glance at. Somebody has to build the test set from real customer cases, decide which failures are acceptable, and keep the test set honest when the builder is tempted to tune against it. If the builder owns that too, you have a student grading his own exam, and franchement the scores only go up.
You do not need a custom framework to start. The open-source EleutherAI lm-evaluation-harness, a framework for few-shot evaluation of language models under an MIT license, is a good place to see how a serious eval is organised: tasks, metrics, and a fixed way to run them. Your own product tests will look different. The idea is the same. Fixed cases, fixed scoring, one owner.
At a small company these two seats are often two hats on two people who sit next to each other. That is fine. What is not fine is one hat on one head.
The last two seats: the data path and the bill
The third seat is the owner of the data and tooling path. Where does the data come from, how clean is it, how does it get into the prompt or the index, and what happens when the source changes its format on a Tuesday. For Trellis this is not abstract: Amazon listings, reviews and ad reports all arrive in their own shape, and a stale or malformed input can break the feature even when the model is fine.
The fourth seat is the owner of the bill and the latency. Every call has a cost and a wait time, and both grow quietly. Someone must own the budget per task, the choice between a big model and a small one, the caching, the timeouts. If nobody owns it, the invoice owns it. I would hire this person fourth, but I would name the owner on day one, even if it is me on a Sunday night reading the API bill.
Now the question people actually type into search: AI engineer vs ML engineer. On a big team the titles mean different things. An ML engineer trains models, works on data, and runs offline evaluation. An AI engineer does retrieval, tools, prompts, serving and product integration. Those are real skill sets.
At five people the distinction collapses. What you need is someone who can put a model into a product and measure whether it got better. Hire for the title and you end up with two people who cannot ship: one who has a beautiful offline score and no endpoint, and one who has an endpoint and no idea if it works. Hire for the seam instead. Ask: who owns this decision, and can they do both halves of it?
I am not giving a salary figure, on purpose. I have no source I trust, and an invented salary is the first thing a reader catches.
The worked example: this blog’s pipeline, counted on Nov 8
I do not want to leave you with theory, so here is my own team, with numbers I counted in the last hours. This blog is written and checked by a small team of AI agents that I run on Yaaf, the open-source harness I build, whose own site describes it as a way to build a company of AI agents with work the platform verifies. I describe design choices here, not results. Nothing below is a benchmark.
The pipeline, as it runs today, has five stages. A research seat produces a digest of candidate subjects. A writer seat picks one and writes the post. An independent review seat fact-checks every number against its own source and returns one verdict per claim. The post is pushed to WordPress as a draft. Then I publish it, by hand.
That last stage is a row in the pipeline, and I count it as one. Here is what I read from the WordPress REST API on Nov 8: 93 posts on dplouffe.ca, 65 published and 28 drafts. Of the 28 drafts, 5 come from this project: the posts for Nov 4, 6, 7, 8 and 9. The other 23 are my own older drafts, dated March to September, and I will not pretend the agents made them.
The last post published on the site is dated Nov 4. Nov 6, 7, 8 and 9 each have a draft and no live post, and the Nov 4 draft is still unpublished too. The reason is the human step, me. A machine that drafts daily and a CTO who publishes when he has time are two different speeds, and the slow one sets the output.
What the 75 rows in topics.json really say
The record that the pipeline keeps, topics.json, held 75 rows on Nov 8. It is not one row per daily post: the 75 rows cover only 58 distinct dates, and some dates have several rows.
I joined it to the live site two ways. Thirty rows carry a WordPress post id that is published. Another 33 rows carry no id, but each has a dplouffe.ca address that matches a published post, so I matched them by URL. That makes 63 of the 75 rows live. Five rows point at a draft. The last seven point at ids that are in the WordPress trash, all moved there on Nov 6 within two seconds of each other, which looks like one bulk action. Counting this took one REST call per status, one call per missing id and one file read. A team that cannot count its own output in a minute does not know its structure.
Two failures that taught me where the seams are
The first failure is a checker. We have a check that asks, each day: is there a draft on the site for that date? It compared the moment we last read the site against the start of the day. A post dated Nov 9, pushed correctly on Nov 8, was read before midnight and so looked invisible. The check could not pass until the day had already started. The fix, written into the checker’s own comments, was to key on the post’s own date and not on when we looked.
I now have a sentence for this: a check nobody can pass is not a standard, it is a tax. It cost runs, and it made a fine post look broken. Bon, but the real cost is the next one: when people learn that a gate lies, they start to route around all gates.
The second failure is a number in the wrong slot, and this time the miss was the review seat’s too. The Saturday draft said a small open model shipped with a 128,000-token context window. The review on Nov 4 traced that number to the model card and passed it. The card’s own table says something else: 128,000 is the vocabulary size, and the context length is 256K tokens. The fix landed on Nov 6, two days later, and I can show you the before-and-after copy of the post, but I cannot show you who spotted it, because no file records that. A fact printed in the wrong field is exactly what a second pair of eyes is for, and here the second pair read the right page and still looked at the wrong row.
Both failures live in a seam, not in a model. One was a handoff between a check and a clock. The other was a handoff between a writer and a source. That is my argument about team structure: the boxes are easy, the seams are the job.
For a longer look at the human side of this, I wrote earlier about trusting code that an AI manager ships.
What the counts say about this AI team structure
Let me map my pipeline onto the four seats, because it is the same shape. The research seat is the data and tooling path: it decides what material reaches the writer. The writer is the builder. The review seat is the evaluation owner. It is supposed to hold a veto, because a post with an unverified number should not go out, and I come back to how well it does that below. The fourth seat, the bill, is me, reading usage numbers when I should be sleeping.
The run numbers, pinned to a window
Here are the numbers I read on Nov 8, from the WordPress API and from the platform’s own activity report. I pinned the window so you can re-read it: 2026-11-01 18:18:22 UTC to 2026-11-08 18:18:22 UTC. In that window the writer seat had 27 runs. Twenty-three finished and 4 were recovered by the platform after the sandbox stopped answering and left no reason (at my first read, 18:18Z, one run was still going; a later re-read shows it done). That seat claimed 22 tasks and completed 20, and wrote about 225,000 output tokens. The platform’s own checks, the ones that grade the work and not the writer, ran 57 times in the same window, 55 finished and 2 recovered.
What the chart says
Look at the top chart before you look at the bottom one. It is mostly good news: 63 of 75 rows have a live post. I first read it wrong, as 30 live out of 75, and I only caught that when the review seat counted the same rows again. The honest signal is smaller and sharper. Five drafts have been waiting since Nov 4, and twenty-three finished runs is plenty of output. So the bottleneck sits downstream, in a human step that no seat on the team owns. The seven trashed rows are a separate question, someone cleaned up on Nov 6, and I do not know why.
That is the useful lesson about an AI team structure: draw the chart of where the work stops, not the chart of who sits where. The first one shows you the seat you forgot to staff.
The weekly review I run on an AI team structure
Once a week I ask four questions, one per seat, and I answer them with a count, not a feeling. If I cannot answer one with a count in one sitting, that is the finding.
For the builder: how many things shipped, and how many came back? For the evaluation owner: how many releases did you block, and was each block right? For the data path: how many runs died on stale or malformed input? For the bill: what did one finished task cost, and did that number move?
The second question is the one people skip. A veto that is never used is either a perfect team or a person afraid to use it, and from the outside you cannot tell which. So I count. In this pipeline there are 9 review notes on disk, covering 6 posts, and none of the nine let a post through unchanged: the Nov 3 post took four notes and the last one was still a fail on two one-line fixes. That is a veto that gets used. It is also a veto that missed the 128,000 above, so I trust it as a second reader and not as a guarantee.
Hamel Husain wrote the clearest version of this idea in his post Your AI Product Needs Evals: look at lots of data yourself, remove the friction from doing it, and do not lean on a generic framework to grade your product.
Notice what I am not asking: no dashboard, no OKR, no story-point velocity. A good AI team structure makes the questions cheap to answer. If the team needs a meeting to know whether the eval owner blocked anything, the information path is broken.
One habit I added after the check failure from earlier: every question has a named source file or API call. When I wrote the figure above, the caption names the files. A number without a place to re-read it is an opinion with decimals. I trust my team more when I can re-count it.
What to check before you copy this setup
Five things decide whether this shape works in your company, and each one is a question about your setup rather than mine.
First, it is one pipeline, one blog, one week. I counted 27 runs on one seat. That is a story about how my team behaves, and it is not a benchmark of anything. A product team shipping features to Amazon sellers has different failure modes than a team writing posts, and I would not carry my percentages over to yours.
Second, the evaluation owner in my pipeline is an AI agent with a gate, not a human with a career at stake. That works for fact-checking a number against a model card. It does not work for the harder call, “is this feature good for the customer,” which needs someone who has talked to the customer. At Trellis that seat is a person, and I would never move it to a script.
Third, the structure does not fix the human bottleneck. Five drafts are waiting on me while I write this. No seat chart saves you when the last step is one busy founder with a browser tab. The fix is to staff the step or automate the safe part of it, and I have done neither yet.
Fourth, four runs of 27 were lost with no recorded cause. The platform says plainly that it cannot answer why. I count them, I retry the work, and I do not pretend the structure prevents them. A team structure can name an owner for a failure. It cannot always remove the failure.
Last, four seats is what held here; three or six may hold better in yours.
What I would do on Monday
If you have five people or fewer and you are about to hire your first AI person, this is the order I would follow.
Start by naming the four decisions on one page: who decides the feature, who decides “better,” who decides the data path, who decides the budget. Write one name next to each. Two names can be the same person. A blank is the real answer to why things go wrong.
Next, build the first fifty test cases from real customer data this week, before you hire anyone. In 2023 the Latent Space essay The Rise of the AI Engineer argued that the AI engineer should own the product-specific data and evals, while the ML engineer deals with foundation-model concerns. That is still the cleanest split I know, and at your size it is one person with two hats, as I said earlier.
Then count something. Pick one output of your team, find the stage where it stops, and write the count and the date on a page. You will see in ten minutes which seat you forgot.
An AI team structure is not an org chart you adopt once. It is a short list of decisions with names next to them, and a weekly count that tells you which one is slipping. If you try it, tell me where the list broke for you. I want to know, and I will take the honest answer over the polite one.
