I built evaluators that do not pass what they cannot judge, in two separate places

Summary

I ran a mechanism that checks AI-generated answers by machine for several months, in two places of a different nature, and in both I arrived at the same single principle. The principle is that whatever the mechanism could not judge should not, on its own, be counted as either a pass or a fail. The two places are a service I develop at my personal company, and a repository for research drafts and research operations. I cannot verify every answer with human eyes, so there is no choice but to have a machine check them.

The machine check needs an AI judgment because just seeing whether the intended words appear in an answer cannot confirm the correctness of meaning: whether the answer is telling a fabrication with no factual basis, or whether it has stepped into an area the task forbids it to touch. So I split the check items in two. I use the AI judgment only for items that require reading the meaning of the answer, and for items where it is enough to see whether the shape of the response data is broken, I check with a simple string match.

How to count an answer whose judgment did not run came out opposite in the two places. In the personal-company service, I treat an answer that could not be judged as dangerous and stop it. In the research repository, I count text that could not be judged separately, as a third value that is neither pass nor fail. The way of counting is opposite, but the principle is the same: do not quietly mix what could not be judged into the passes.

I also failed once on the check criterion itself, on what in the writing to mark down. The first criterion was whether phrasing that avoids assertion was present. Under this criterion, even the genuinely good way of writing, stating the conclusion up front and then attaching the reasons, got marked down, and the writing grew weaker and weaker. After that failure, I changed the criterion for marking down from whether such phrasing is present to whether the assertion has reasons attached. Along with this, I had the writer also state how much observation the claim rests on, and added that to the material for judgment.

Midway through the implementation, I also hit a failure where the judgment model, in the processing where it thinks before giving an answer, used up the ceiling on the number of tokens available for output, and the judgment JSON that came back was cut off partway and came back empty. Here I widened the output ceiling from 1000 to 2000 tokens, and on top of that, so that the judgment result can be salvaged even from a broken response, I dealt with it by building the parsing in three stages. In verifying vector search, the order of the search results changed slightly every time, so I could not verify with expected values that fixed the results in advance. I abolished the mechanism that quietly switched from a strong model to a light one when answer generation got congested, because no one can notice when the answer quality drops.

The body that subscription lets you read is built to trace the failures one by one. It starts from what a character match cannot check about an answer, compares the way of counting what could not be judged across the two places, goes through the failure of mistaking the criterion for marking down and how I fixed it, the problem of the judgment JSON being cut off partway and the problem of verification where the search results shake every time, and the decision to stop the automatic model switching, and ends with the three principles that remained after the failures. How much counting on the dangerous side wrongly stops answers that are fine, and whether the judgment model’s habit of preferring the text it wrote itself also bears on this use, I have not yet been able to measure, and in the body I note each time that these are open questions from before measurement.

Who this is for and what you can take away

This article is for developers who are trying to build AI judgment into their own service, and who are unsure whether they can trust the judgment model’s output as is and what to return when the judgment does not run. You can take back into your own development a pattern of judgment: whether to count an answer that could not be judged as a pass or a fail, how to decide the criterion for marking down writing so that the writing does not break, and how to verify a search whose results shake every time nonetheless. This article does not line up designs that worked. Its reading value is that it puts out, as they were, the failures that became visible only once things were running and how they were fixed. Two events ran opposite to what I expected before running, and you can read both all the way to the countermeasures. A criterion that dislikes assertion across the board weakened even the good habit of stating the conclusion up front. A mechanism that switched to a light model under congestion hid the drop in quality. Because in two places of a different nature I arrived at the same principle from separate approaches, I think this pattern can be used regardless of place. This article is a reflection based on a first-hand record of several months of operating a service I develop at my personal company and a repository for research drafts and research operations.