I compared three AI agents on the same AI model, and the only differences that came out were elapsed time and the number of retries

Summary

When I ran the aggregation script, three zeros lined up: zero failures, zero rewrites of the tests, zero human interventions. I had compared three AI coding agents, running them 144 times with the same model loaded in each. So on the measures that look at the quality of the result, no difference appeared among the three.

The differences appeared only in elapsed time and the number of retries. For example, on the scale tasks, which an implementation written in one pass cannot pass, the mean elapsed time over the 12 runs per agent split three ways, from 100.8 seconds to 151.2 seconds. I also compared handing each agent its set of configuration files against not handing it over. Whether it was handed over showed up in none of the measures I could take, that is, the success rate, the number of retries, and the elapsed time. I had defined a field in the records, but nothing was implemented to write it, so token consumption is unmeasured. Also, I ran the three agents in separate time slots, so I cannot say the differences in elapsed time come from the agents alone.

In the body I first give the conclusion from the 144 runs and explain the conditions of the comparison and how I measured. Next I write about where no difference appeared, the quality of the result and the configuration set, and I show elapsed time and the number of retries, where differences did appear, in a table by agent and task set. After that I explain the token consumption I could not measure, and the first design I threw away, which tried to measure by success rate. Finally I show the steps by which I rebuilt the unit tests so I could trust the aggregation, and the range of what this comparison cannot say. The bibliography of the studies I referred to is collected at the end.

What you can take away

For anyone in a position to choose an AI coding agent, this article explains the following three things.

• You will be able to set up the same comparison on your own tasks. I explain how I pinned the model, how I divided task sets of differing character, how I counted pass or fail and the number of retries, and why I ran everything serially
• You will be able to judge what to measure instead at a difficulty where the success rate reaches its ceiling. I explain how I came to move what I measured from the quality of the result to elapsed time and the number of retries
• You will learn how to write unit tests that catch a malfunction in the aggregation even when every measure in the data is zero. I explain the steps by which I fixed the correct counts using synthetic input with failure rows mixed in

All of it is written from the 144-run comparison I set up myself, and from how I changed the way I measured partway through.

This article is a comparison and discussion based on primary data from work I am pursuing as personal research. In the body I name the three AI coding agents I compared. The content does not represent the views of any of their providers.