Summary
I ran into an incident where every test passed, yet the meaning check that was supposed to judge quality had never run even once. This happened while I was writing a research abstract, a long piece of natural-language prose, in a development loop that rewrites the text until the tests pass. When the reviewer’s authentication expired, the meaning check was skipped entirely, and the tests that looked like they had passed stopped being evidence of safety.
The tests have two layers: a deterministic check that can judge without looking at context, and a meaning check that calls an LLM through the CLI as the reviewer. A judgment takes one of three values, pass and fail plus an error that stands for not having been able to run the review at all.
I put this mechanism in about a week after I started running the repository for the research abstract and its operations, and over more than two weeks after that I kept adding checks, extending the same loop to a second abstract, and restoring the reviewer’s authentication and updating the model.
Audience and takeaways
This article is for developers and writers who want to bring a development loop that rewrites prose while watching whether the tests pass into their own writing, not just their code. There are three takeaways. The first is a criterion for sorting: drop the checks that misjudge when they are enforced without looking at context. The second is a three-value design that does not mix the state where the review could not be run into the pass-or-fail of quality. The third is a failure where dropping every assertion uniformly left the text full of hedging so that no claim stood, and how I fixed it. The material is the operational record of running this mechanism for more than two weeks in a repository for a research abstract and its operations.
Bringing the code development loop into long-form writing
When you write code, the tests return pass or fail the moment you write it. Whether this development loop can be brought into long natural-language prose as well, such as papers, specifications, and proposals, is the theme this time. What I targeted was a research abstract. The test definitions live in YAML, and a test runner built on pytest expands one item into one test through parameterization, and a way of working where you rewrite until the tests pass has started to run.
Let me note up front that these tests are not a tool that writes in my place. I am only turning the comments I have actually made on abstracts, one by one, into rules and then into checks, so all the machine can reproduce is those past comments, and deciding what to write is still the writer’s job.
Dropping checks that misjudge when enforced without context
There is a single axis that separates the checks I dropped from the checks I kept. I dropped the checks that misjudge even legitimate usage when they are enforced in isolation without looking at context, and I kept the checks that do not misjudge because the value being checked, such as a character count or a number of occurrences, can carry a range. At first I had put in only rigid, enforcement-style checks: enforcing the length of a single sentence, strictly enforcing the number of paragraphs, and rejecting forbidden words in isolation without context. But they misjudged even writing that only makes sense once there is context, and they took away freedom in how to write, so after building them once I moved them out of the runner and deleted them as an old implementation.
What I placed under this sorting was a policy of not assuming that the evaluation of a paper can be carried by deterministic checks alone, and of putting weight on judgments of meaning and logic. I moved only the items that can be decided without looking at context, such as a range of character counts excluding whitespace and typesetting commands, whether a specified word is present or absent, and regular-expression matches and their counts, to the deterministic check implemented as local functions inside the runner, and I left the consistency of meaning, logic, and chapter structure to the layer that calls an LLM through the CLI as the reviewer. Fast and cheap judgments go to the deterministic check, judgments that need context go to the reviewer, and both live together in a single runner.
Keeping errors separate from the quality judgment
I gave the reviewer a fixed instruction to judge pass or fail using only the criteria I gave it, and to list every violation without omission, and I forced the response into structured data of the judgment, the location, and the reason. This is to keep fixes from dragging on as violations get pointed out a few at a time. When typesetting backslashes get mixed into the response, parsing the structured output fails, so I also added a step that corrects them and re-runs the parse.
The third of the three values, error, refers to the state where the reviewer could not be run at all because of expired authentication, a timeout, or a failure to parse the response. I keep this separate from the quality judgment of the text and do not mix it into fail. I took the view that if, when failures increase, I cannot separate whether the problem is in the text or in the infrastructure, this loop does not serve its purpose.
In fact, I ran into an incident where the reviewer’s authentication expired, every reviewer was skipped as an error, and although the tests all looked like they had passed, the meaning check had not run even once. The lesson that became clear at this point is that even when all the tests pass, it does not follow that the meaning check actually ran.
Letting assertions pass within the range of observation, suppressing variance with a majority vote
Among the axes of judgment, the handling of assertions took the most tuning. During the period when I uniformly failed anything that asserted without regard to the scale of what was observed, the text turned into nothing but hedged phrasing and no claim would stand. So I inserted the scale of what was observed, such as how many people and how many cases, into the reviewer’s instruction, failed only generalizations that went beyond that range, and reworked it so that a reasoned assertion kept within the range passes as is. Along with that, on a failure I always have the reviewer return even the distinction between whether a fix in phrasing is enough or whether I should go back to collecting data.
To suppress the reviewer’s variance, I implemented it so that if the first run passes it stops there, and only when it is not a pass does it judge twice more and take a majority vote over the three runs. That said, this is a compromise that favored speed, and I have not measured whether the reproducibility of getting the same judgment on the same abstract within at most three attempts is good enough for practical use. I do see it pointed out that a single reviewer run has variance in its judgment and that a stable consensus needs repeated attempts, but I have not yet been able to confirm primary literature that backs up the number of attempts needed. I cannot say anything more right now than that I struck a balance between speed and reproducibility.
The deterministic check to CI, the reviewer to the local loop
From introduction to being usable in practice took essentially two days, and over the roughly two weeks after that I added checks little by little, against primary literature on how to write papers and against my own feedback. Because per-chapter checks leave the gaps between one chapter and the next unseen, I also later added a whole-piece test that reads across all chapters to see whether a single thread runs from the question to the claim.
I also decided the division of operation. Only the deterministic check runs automatically in CI every time, and the reviewer that handles the meaning judgment is reserved for the manual loop I run by hand. I built the mechanism up on a single abstract, and I extended the same loop to an abstract on a different topic.
What the machine could offer went only as far as material for judgment
Having run it this far, the range the machine could carry has come into view. What the machine could do went only as far as offering, as material for judgment, the comments I have made over and over. Whether test-driven work raised the quality of the abstract itself, I cannot assert, because I have not taken the primary data of how the number of failures changed over time. That is as far as running it made things visible, and I refrain from declaring anything beyond that as an effect of the tool.
The line between the judgments that can be left to the machine and the judgments that only a person can keep making will surely keep moving from here, and how far it moves is something I want to make out as I keep running this operation.