The Factory Makes Its Agents Argue

With me, seemingly every reasonable feature starts with the same question.

“How hard could it be to add a couple more reviewers?” — me, moments before building a tiny judicial system

After getting the Factory to build its own operator console, I had a fairly reasonable thing to worry about: who was deciding whether any of this code was actually good enough to keep?

The Factory already had a reviewer: one agent wrote the code, and another looked at it before approving it or sending it back for another attempt. Better than asking the coder to admire its own work, obviously, but the whole quality gate still depended on one reviewer's interpretation of the task. If it missed the same assumption the coder missed, well, that was that; if it misunderstood perfectly reasonable code, the coder got to spend another pass explaining itself.

So I wanted more eyes on the same change, maybe two more, though I hadn't really thought about what they should do once they'd looked. Would they vote, debate, or pick a reviewer to review the reviewers? This became the Task Arena in August, and I got a much better understanding of why “just add another agent” is usually the beginning of a project.

Background

For those who haven't been following the Factory posts, the idea is an autonomous software production line: I give it requirements, it breaks the work into tasks, and agents implement and review the changes. The original review process was simple enough to fit on one line:

coder -> reviewer -> approve or rework -> merge

The Arena was meant to make that review step more useful without turning one coding task into a whole new project to manage. Three reviewers would inspect the same candidate independently, and then a judge would read what they found and decide what happened next, while the rest of the Factory still saw one task.

That sounds pretty straightforward, though it was not quite the same thing as three agents having a meeting.

Three opinions, three rooms

The first thing about making the agents argue is that I didn't actually want them talking to each other. Slightly misleading name, I know.

If reviewer two reads reviewer one's explanation before looking at the code, there's a good chance it starts responding to that explanation instead of doing its own inspection. Then reviewer three gets an even more convincing version of the same premise. At that point, do I have three opinions, or one opinion with a very persuasive introduction?

So each reviewer gets the same candidate and task context without seeing the other reviewers' reasoning. Their tools are read-only too, so reviewing the implementation can't quietly turn into rewriting it and leave us comparing reviews of different changes.

They can still all be wrong: give three agents a bad task description and they can diligently agree on the wrong thing. I also didn't assign guaranteed specialties like security or performance in this first version. I had three independent inspections, not a security team, a performance team, and a tiny robot QA department. That was a useful start, but it isn't magic.

Counting isn't the hard part

Once all three reviews are in, the Factory checks whether enough reviewers approved. The default is two-thirds, which means I need two approvals from a panel of three.

I was already adding a lot of model calls to this process; counting to two did not need to be another one. Ordinary Rust can handle that just fine, thank you very much.

The part that took more thought was what a majority actually meant. Passing quorum gives the judge permission to approve, but doesn't make approval automatic: the judge still reads all three critiques, including the one from the reviewer who voted against the change. We wait for all three even when the first two approve, because the last objection might be the useful one.

Imagine two reviewers are happy with a cleanup function, while the third spots that it deletes a record exactly at an expiry boundary where the task says to preserve it. The majority doesn't make that boundary case disappear, and the judge can still send the candidate back for rework.

And the reverse is true too: if the panel misses quorum, the judge cannot approve the candidate, no matter how persuasive its explanation is. That rule belongs to the system, not to whichever agent happens to sound most confident. I want the agents to help me interpret the work; I don't want them negotiating the rules as they go.

“Needs rework” is not a work order

The first version only sent the panel to the judge if quorum passed. If the panel rejected the candidate, the next step seemed obvious enough: send it back to the coder. Why pay a judge to tell me what the vote count already established?

Yeah, that didn't work as well as I thought.

Knowing that something needs rework doesn't tell the next coder what to do, and my fallback brief, assembled from critique categories, was a pretty blunt way to turn three actual reviews into another coding attempt. I'd saved a model call and made the handoff worse.

So now every completed panel goes through the judge. A failed quorum still means rework—the judge can't change that—but it can read the critiques and turn them into something useful for the next pass: a summary of the problem, an ordered list of actions, and guidance on how to verify the fix.

The judge can still summarize an objection badly, turning a useful detail into a vague action item that sends the next coder in the wrong direction. But at least the handoff gives me something specific to inspect: I can check whether the judge preserved the objection and whether the coder followed the brief, rather than trying to work out where a long conversation went sideways.

The process has to remember

Because the Factory can stop one agent process and start another at any time, the review can't depend on keeping a particular conversation alive to remember what happened.

The Arena records the review cycle, each panel seat's status, the votes, the quorum result, and the judge's ruling in durable state. After a restart, the Factory can pick up pending work instead of quietly assembling a fresh panel. The important bit isn't the name of the state object; it's that restarting the process shouldn't rewrite the history of the decision.

Retries needed a little thought too, because a reviewer crashing before it submits a vote is not the same thing as a judge asking for code changes. One is an operational failure and the other a quality problem, so they use separate budgets rather than letting a broken agent process eat up the coder's chances to improve the implementation.

The default allows four reworks after the initial candidate; if it still needs another revision after those four, the task fails. It can also fail earlier if operational recovery uses up its budget. At some point the Factory has to say it couldn't finish instead of keeping everyone busy indefinitely.

Once approved, a candidate goes to the existing merge queue; the Arena doesn't replace tests or perform the merge itself. It just makes the review part of the production line something I can take apart and inspect.

Then I looked at the bill

The first sustained Arena task produced 52 agent sessions, with a reported cost of $55, and the panel accounted for roughly 81 percent of that cost.

I wanted a couple more eyes on the code. The eyes were now most of the bill.

Getting those numbers was its own exercise. The Factory could tell me who had voted, remember the votes through a restart, and stop the judge from approving without quorum. Ask it what the whole thing cost, though, and I had to go spelunking in OpenCode's private database.

This was one observed task, so it doesn't tell me what an average Arena task costs or whether the reviews were worth their price in bugs caught. But it did make one thing pretty obvious: I can't make sensible decisions about spending less if I can't see where the money went.

The Factory could explain who made a decision and why, but not the bill for making it. So usage and cost became another requirement: record them durably for every operation, attempt, retry, panel member, and judge.

What I learned

I went in thinking the hard part would be getting three agents to review the same code, but it turns out the hard part was deciding how to use their opinions without letting the loudest or most confident one take over.

The reviewers need room to think independently, and even a majority can't erase a good objection. If a candidate is rejected, the next coder needs to know what to do; the Factory also has to remember the decision after the agents—and the processes running them—are gone.

I also need to know what all that careful review costs. Apparently, even my agents needed someone to keep track of the expenses.

See you next time reader, Shardul