Ask an agent to explain how it arrived at a number and it will produce something fluent, well-structured, and completely unverifiable. It will describe a process. Whether that description matches what actually executed is a separate question, and the explanation itself contains no evidence either way.
This is the trap at the centre of AI explainability, and it is worth stating plainly: a generated explanation is not evidence. It is a plausible narrative produced by the same system whose behaviour is in question, after the fact, with every incentive to sound coherent. If the model got the number wrong, the explanation will explain the wrong number just as smoothly.
The alternative is not a better prompt. It is instrumentation. Capture what happened mechanically, and let the explanation be assembled from the record rather than narrated from memory.
Two questions people are actually asking
“Explain what you did” collapses two different needs that require different artifacts.
Audit: what did this system do to my data, when, under whose authority? This is a compliance and accountability question. It is answered by a run record, and it needs to be complete rather than readable.
Provenance: why should I believe this specific number? This is an epistemics question, asked by whoever has to act on the answer. It is answered by a traceable path from figure to source, and it needs to be checkable rather than complete.
Most teams build neither, then discover they needed both on the same afternoon.
What a checkable answer contains
Not prose. A structured record attached to each figure, captured at execution time:
- The query, verbatim — the exact statement that ran, not a paraphrase of it.
- The source objects — tables, views, or documents touched, with their identifiers.
- The as-of timestamp — when the underlying data was last true, not when the query ran.
- Filters and their origin — what was excluded, and whether the user asked for that or the agent chose it.
- Row counts in and out — the single cheapest signal for fan-out, over-filtering, and silent truncation.
- Definitions used — which metric definition, at which version.
- What was refused or unavailable — the tables it could not reach, the queries that were rejected.
Every one of those is a fact the system knows at execution time. None requires the model to introspect. That is the entire design principle: if the model has to remember it, it isn’t evidence.
Row counts deserve their own paragraph
If you record one thing, record rows in and rows out at each step.
Nearly every quiet data failure shows up there first. A join that fans out shows a row count that grew when it should have stayed flat. An over-eager filter shows a collapse from 40,000 to 12. A truncated result shows a suspiciously round number. None of these produce errors, and all of them are visible in two integers.
It is the cheapest instrumentation in this entire guide and the one most often left out, because nobody thinks of a row count as telemetry.
Layered disclosure
Provenance is only useful if someone reads it, and different readers need different depths. Three layers works well:
The number, with its caveats inline. “£2.9m, as of 06:00 today, excludes refunds and intra-company transfers.” One line. This is what most people need, and attaching it to the figure means the caveat travels with the number into whatever slide deck it ends up in.
The assumptions, on request. Which definition, which date range, what was excluded and why, what was missing. This is the layer that catches the “that’s not what I meant by active users” problem, and it should be one click away.
The query and the record, always available. For the person who needs to reproduce or dispute it. Rarely opened; essential when it is.
The failure mode to avoid is burying everything in the third layer. A caveat that requires three clicks is a caveat nobody sees, and an unread caveat provides documentation rather than protection.
Reproducibility is the real test
The honest test of provenance is not whether the explanation reads well. It is whether someone can re-run this in six months and get the same number.
That requires more than storing the query. It requires pinning what the query ran against: a snapshot identifier, a table version, an as-of timestamp the warehouse can honour. Time-travel features in modern warehouses make this genuinely straightforward, and they are the difference between “here is the SQL” and “here is the SQL and the world it ran in.”
Without that pin, re-running produces a different number and you cannot tell whether the original was wrong or the data simply moved. That ambiguity is what turns a fifteen-minute check into a two-day investigation, and it is the same capability that makes deterministic replay possible for agent runs generally.
When the answer turns out to be wrong
Which it will. The value of all this is what becomes possible on that day.
With a captured path, the postmortem is mechanical: re-run the query against the pinned snapshot, confirm it still produces the reported figure, then walk back through the record — was the definition wrong, the filter wrong, the source stale, the grain misjudged? You will usually land on a specific missing clause, and that clause becomes a test. The incident produces a permanent improvement.
Without it, you have a wrong number, a fluent explanation of the wrong number, and a conversation about whether to trust the system. That conversation has no evidence in it, so it gets resolved by seniority, and nothing is fixed.
The categories of thing you will find are catalogued in Hallucination Is a Data Contract Problem.
Where the model is genuinely useful here
Having argued that generated explanations are not evidence, the fair qualification: the model is good at translating a captured record into something a non-technical reader can act on.
The rule is directional. The model may summarise the record for a human. The model may not be the record. Generate the prose from the structured artifact, keep the artifact attached, and make the path from summary to evidence one click. Then a summary that drifts from the record is a bug you can detect, rather than an interpretation you cannot check.
Earning trust without giving it away
A pattern that works: begin with everything visible. Every answer arrives with its query, its as-of, its row counts, whether anyone asked. It is noisy, and for a few weeks people will actually read it — which is exactly the period when they are calibrating how much to trust the thing.
As confidence builds, collapse the detail behind disclosure. The caveat line stays inline; the rest moves one click away.
What must never happen is removing the capability. Retention on run records is a compliance question rather than a storage one, and the moment you cannot reconstruct last quarter’s number is the moment someone asks about it. Storage is cheap; the inability to answer is not.
The uncomfortable part
Most organisations cannot currently do this for their humans. Ask where a number in last quarter’s board deck came from and you will often get a spreadsheet, a person who has left, and a shrug.
Agents make that unacceptable, because volume changes the calculus. One analyst producing forty numbers a quarter, each defensible in conversation, is a workable system held together by the fact that you can ask them. An agent producing four thousand is not, and there is nobody to ask.
So the honest framing is that this is not overhead the agent imposes. It is a standard your data function probably should have met already, and the agent is simply the first consumer that cannot compensate for its absence by being reasonable. Every organisation I have watched build this for an agent has ended up using it for the humans too.
The access controls that make these queries bounded are in Letting an Agent Touch the Warehouse Safely. The failure modes that go undetected without a record are the accountability family in Agent Failure Modes.