I built evaluators that don't pass what they can't judge, in two separate places

Summary

I ran a mechanism that checks AI-generated answers by machine for several months, in two places of a different nature, and in both I arrived at the same single principle. The principle is that whatever the mechanism could not judge should not, on its own, be counted as either a pass or a fail. The two places are a service I develop at my personal company, and a repository for research drafts and research operations. I cannot verify every answer with human eyes, so there is no choice but to have a machine check them.

Why does the machine check need an AI judgment at all? Because just seeing whether the intended words appear in an answer cannot confirm the correctness of meaning: whether the answer is telling a fabrication with no factual basis, or whether it has stepped into an area the task forbids it to touch. So I split the check items in two. I use the AI judgment only for items that require reading the meaning of the answer, and for items where it is enough to see whether the shape of the response data is broken, I check with a simple character match.

How to count an answer whose judgment did not run came out opposite in the two places. In the personal-company service, I treat an answer that could not be judged as dangerous and stop it. In the research repository, I count text that could not be judged separately, as a third value that is neither pass nor fail. The way of counting is opposite, but the principle is the same: do not quietly mix what could not be judged into the passes.

I also failed once on the check criterion itself, on what in the writing to mark down. The first criterion was whether phrasing that avoids assertion was present. Under this criterion, even the genuinely good way of writing, stating the conclusion up front and then attaching the reasons, got marked down, and the writing grew weaker and weaker. After that failure, I changed the criterion for marking down from whether such phrasing is present to whether the assertion has reasons attached. Along with this, I had the writer also state how much observation the claim rests on, and added that to the material for judgment.

Midway through the implementation, I also hit a failure where the judgment model, in the processing where it thinks before giving an answer, used up the ceiling on the number of tokens available for output, and the judgment JSON that came back was cut off partway and came back empty. Here I widened the output ceiling from 1000 to 2000 tokens, and on top of that, so that the judgment result can be salvaged even from a broken response, I dealt with it by building the parsing in three stages. In verifying vector search, the order of the search results changed slightly every time, so I could not verify with expected values that fixed the results in advance. The mechanism that, when answer generation got congested, quietly switched from a strong model to a light one, I abolished, because no one can notice that the answer quality has dropped.

The body that subscription lets you read is built to trace the failures one by one. It starts from what a character match cannot check about an answer, compares the way of counting what could not be judged across the two places, goes through the failure of mistaking the criterion for marking down and how I fixed it, the problem of the judgment JSON being cut off partway and the problem of verification where the search results shake every time, and the decision to stop the automatic model switching, and finally writes the three principles that remained beyond the failures. How much counting on the dangerous side wrongly stops answers that are fine, and whether the judgment model’s habit of preferring the text it wrote itself also bears on this use, I have not yet been able to measure, and in the body I note each time that these are open questions from before measurement.

Who this is for and what you can take away

This article is for developers who are trying to build AI judgment into their own service, and who are unsure whether they can trust the judgment model’s output as is and what to return when the judgment does not run. You can take back into your own development a pattern of judgment: whether to count an answer that could not be judged as a pass or a fail, how to decide the criterion for marking down writing so that the writing does not break, and how to verify a search whose results shake every time nonetheless. The reading value of this article is not that it lines up designs that worked, but that it puts out, as they were, the failures that became visible only once things were running and how they were fixed. You can read, all the way to the countermeasures, events that ran opposite to what I expected before running: a criterion that dislikes assertion uniformly weakening even the good way of writing that states the conclusion up front, and a mechanism that switches to a light model when things get congested hiding the drop in quality. Because in two places of a different nature I arrived at the same principle from separate approaches, I think this pattern can be used regardless of place. This article is a reflection based on a first-hand record of several months of operating a service I develop at my personal company and a repository for research drafts and research operations.