Skip to content
The article title beside a black and white photo of a loupe on a printed page, with the word AUDIT cut across both.

I had AI agents audit my API. The example on its homepage was made up.

How I split work across agents for task fit, token spend and independent audits, and why the author never checks its own work.

Before a partner built on the AllCars car-data API, I had it audited: about 700 live requests across the 50 most searched models, the answers cross-checked against the database, and four independent reviews. Security came back clean. The defects were in what the API said: 5 critical, 11 high, 9 medium.

One finding I didn't expect. The example response on the API's own homepage was made up: field names the API never returns, and a road tax of €59 where the code calculates €58. The playground beside it said every response was a real capture from the live API. Two of the eight weren't. Every sample car on that page, a Golf, a C-Class, a Qashqai and a 3-Series, had a single comparable: a Toyota Yaris. By that page's logic, every car is secretly a Yaris.

AI agents wrote every line of AllCars. Different agents found these problems because they were given a different job: check what production actually returns.

I run several agents on the same work for three reasons that are easy to blur together. Token spend is one: not every task needs the most capable model or the whole history of the project in its context. Task fit is another: some models handle some kinds of work better, and the allocation should follow that. The third is audits: the agent that writes the code is not the agent that checks it.

Each reason needs its own justification, and they pull against each other. An independent audit costs tokens. A cheap allocation turns expensive when the worker needs repeated correction or gets buried in irrelevant context. A model that is strong on an ambiguous design question can be a poor use of budget for a mechanical change with a clear acceptance command. I don't have a universal ranking that settles this for every task.

So I allocate the task before I allocate the agent. What decision is needed? What evidence does the worker need, and which files may it change? How will someone establish that the result works? With those answered, choosing a model becomes a concrete decision instead of picking a favorite and sending it everything.

What token spend actually measures

Token counts are worth watching, with one caveat: tokens are not money. A token count is an odometer, not a fuel receipt. A count of traffic doesn't tell you the bill, especially when much of that traffic is cached. The more useful question is what the tokens bought: a checked change, a resolved question, or another pass over the same material.

Delegation has its own overhead. Separate contexts keep a worker focused, and they also duplicate instructions, source material and earlier decisions. If three workers each read the same history, that reading has happened three times. Book club, but every member buys their own copy. Audits consume resources as well, so each one needs a reason, a scope and an end.

The example response on my API's homepage was made up. The audit used 700 live requests across the 50 most searched models and four independent reviews, none by the author, and found 5 critical, 11 high and 9 medium issues, with security clean.

The worker brief

This is the brief I use to make an allocation reviewable:

It helps even with one agent. With two or more, the boundaries matter more, because every assumption now has to survive a handoff.

A hypothetical example shows how the pieces fit. A ticket asks for an empty search result to say that nothing matched and to offer a way to clear the filters. Implementation goes to a model suited to that code, with the relevant source, the allowed files and a concrete acceptance check. It returns the change and the output of running that check. Deterministic checks run before any model review, so the reviewer doesn't spend tokens discovering a build failure that an executable check would have caught.

Then a different agent reviews the change and runs the verification itself. Its job is narrow: does the empty state appear, and does clearing the filters actually change what the user sees? If it finds a problem, it returns a reproduction. It wasn't asked for another general approval of the diff.

The brief every agent gets before it starts: objective, evidence, ownership, acceptance and return. Author and reviewer are never the same agent.

Independence needs a real boundary

Calling the second agent a reviewer doesn't make it independent. Two personas can run on the same model. That's one reviewer in a fake moustache. Two different models can share the same mistaken assumption if both receive the same incomplete brief. So the record says who wrote the change, who checked it, which models they used and what each one actually verified.

That record keeps the audit honest over time. A verdict refers to the exact content checked, with the commands and results. If the code changes afterwards, the old verdict is no longer evidence for the new version.

For the empty-state task, a third agent would need a purpose of its own. Another worker rereading the same diff and offering another general opinion adds token spend without resolving anything new. A focused check of one disputed behavior has a clear reason to exist.

My rule for adding a worker is to name the uncertainty it will resolve and the evidence it has to return. If I can't name either, the extra agent has no job to do.

To try it, pick one example in your own docs and run it against production today. If it doesn't match, that's your first audit finding. If every comparable is a Yaris, that's your second.

Takeaway

Allocate the task before the agent: give every worker a bounded uncertainty to resolve, keep author and auditor separate, and judge token spend by what it bought.

Next

Book a call All posts