Don't let the agent grade its own fixes: the design of a loop that brought quality to convergence in one night

Summary

When I put AI agents in charge of polishing quality, the design that worked best was splitting the work into two agents: one that makes the fixes, and one that judges the result. If one agent both fixes and judges, it grades its own fixes too kindly, and you lose confidence in its verdict that quality has converged. Converged here means the state where new findings stop coming up for a run of rounds. So I built it so that the judging agent never sees a word of the fixing agent’s explanations, and decides pass or fail only from numbers measured by machine from the built product and from facts about the structure of the screen.

Here is what I measured over 10 rounds of this loop in one night. The count of new findings the judging agent detected went 4, 2, 3, 2, 2, 1, 0, 2, 2, 0, and it did not fall monotonically. After reaching zero once, it went back up. And at the end I stopped the loop myself, before it met the mechanical condition for convergence.

This article explains how to split the fixing role from the judging role so that self-judgment can be trusted. It also covers how the loop failed to stop on the mechanical condition for convergence alone. This article is a discussion based on the operating records of a quality loop I actually ran for one night.

Who this is for and what you can take away

This article is for people designing a mechanism that has AI agents raise quality through repetition. You will see why the fixing role and the judging role are split, and what the judging agent is shown and what it is kept away from. You can also take away how to combine a mechanical condition for convergence with human judgment.