Summary
In a repository I run on my own for research preprints and research operations, I rewrite research prose until the tests pass, the same way I would with code.
I have turned the quality checks into a set of tests written in YAML. The ones a machine can decide are judged by code functions, and the ones that cannot be decided without reading the meaning, such as how an argument is built, are judged by a large language model acting as a reviewer. When I get a point in review, I turn it into a rule on the spot, so that a person does not have to make the same point again.
But the formal checks that mechanically constrain things like banned words and sentence length, far from raising quality, erased the individuality of the writing and made the prose uniform, so partway through I removed the checks I had been adding. When a test fails, it returns a suggested fix, along with a distinction of whether that failure can be fixed by rewriting.
Rewriting until the tests pass
In software development, there are tests as a way to automatically check whether the code you wrote works as expected. For a while now I have been writing research prose with the approach of rewriting it until those tests pass. Tests for prose might sound strange, but what I am doing is simple. I take the points I get in review and turn them into check rules on the spot, so that a human never has to make the same point twice. That is all it is. For example, there was a time when I turned three points I received at once into rules on the spot and fixed two pieces of writing together.
I split the checks into two layers. The ones a machine can decide, like character counts and syntactic consistency, are judged by code functions, and the ones that cannot be decided without reading the context, like the flow of introduction and development or of cause and effect, are left to a large language model acting as a reviewer. In number, too, the side that reads the meaning is far larger. At the time of writing this article, there were 131 checks handed to the reviewer and 15 checks seen by machine functions. I keep this whole set of checks as a suite of tests in YAML, and when I run it, for the writing as a whole, each check comes back as a pass or a fail.
With this, the basis for rewriting changed too. I came to look at the list of failing checks and decide from it where to fix.
Dropping the checks that were only for form
This setup was not in its current form from the start. When I began building it, I was heading toward turning every point I received into a rule without omission, and I was adding checks a machine can decide, like a list of banned words and an upper limit on sentence length.
But checks that are only for form, like being too strict about character counts, forcing a number of paragraphs, or mechanically banning particular turns of phrase, look at first like they raise quality, while in fact they strip the style out of the writing and make it uniform. A word may be unclear about what it refers to when it stands alone, but if there is a sentence just before that explains it, the meaning comes through. Even usages that hold only because of context, the machine checks reject across the board.
So partway through I changed direction, and I left on the machine side only the checks whose answer is uniquely determined, while I moved the judgments involving meaning and logic toward having the reviewer read them together with their context. To each check I kept, I have tied a source for why that rule exists, one by one, so that whether the basis is a document that sets writing conventions or a point received in review, it can be traced later.
A failing test returns how to fix it
Another part of the design is that when a test fails, the test side returns how it should be fixed. Every failing check comes with a suggested fix, and it also returns the distinction of whether that failure can be fixed by rewriting or whether the underlying data is simply not enough. A failure that cannot be fixed is not a problem with the writing but a task of data collection. If the data is not enough, I do not touch the writing and just go back to measuring.
I have also decided how to handle the case where the reviewer’s judgment could not be run. A check that did not run is not counted as a failure but counted separately. If a failed judgment and a flaw in the writing get mixed together, the number of failures itself stops being usable as a guide to where I should fix things.
What I could confirm goes only as far as stopping recurrence
The checks that turn points into rules accumulate on the side of the system, not in a person’s memory. That the recurrence of the same kind of point is stopped by the checks has become something I can confirm from the test results. But that is as far as I can say for certain about the effect right now, and whether the prose written this way has actually become easier for readers to read is not something the passing of a test can tell me.
How much the reviewer returns the same judgment for the same input, that reproducibility, I have not yet been able to put into numbers either. I have put in a mechanism that judges two more times only when the first judgment is not a pass and takes a majority vote over the three, but whether that makes the judgment stable is not something I have been able to confirm. If the judgment wavers, trust in the list of failing checks itself is shaken, so I see this stability as the single most important thing to confirm, since it bears on the premise of the setup. Extending this setup to long-form deliverables is a matter still further ahead, and what I hold for certain right now is only this one point: that the checks stop the recurrence of the same point, so a person does not have to say it twice.
(Added 2026-07-11: In this article as published, I had written the mechanism that takes a majority vote over multiple judgments as a plan not yet in place. When I checked the past records during an operational review, I found that a limited mechanism, which judges two more times and takes a majority vote only when the first judgment is not a pass, was already in place on 2026-06-24, before publication. So I have corrected the passage above to read that the mechanism is in place but its stability cannot yet be measured.)