Pass the conversation history as it is, and the AI starts imitating its own past answers

Summary

A report from live operation said that a conversational AI’s answers were breaking down. I chased the cause, found two of them, and fixed both. One was that a string processed for on-screen display was being reused as it stood in the next turn’s conversation history. The other was that an answer format born from a special instruction, one that was supposed to apply to that turn only, stayed in the history, and the turns that followed came out in the same format.

In a comparison of six generations from the same input, before the fix, section headings crept in at 4 and 5 per generation; after the fix, both generations came out at 0. What worked was fixing the two at once. Fixing them one at a time left 4 in each case, with no improvement.

In the body I start from the breakage as it was reported, and write how I told apart two paths: the one where processing for display mixes into the history, and the one where the format from a special instruction stays behind. On that basis I explain the fix, which is restoring past answers to their canonical form before passing them to the history, and then write, in order, the comparison numbers that confirmed the two causes by measurement, and the stabilizing of the restoration that I put in the next day.

Who this is for and what you can take away

This article is for people building a conversational AI feature that carries a conversation history. You can take away a design principle: formatting for display and the input you pass to the model belong on separate paths. I show it through a defect I actually hit and a measured comparison. You can also take away the steps for telling causes apart when the breakage will not go away, and how to undo the assumption that there is only one cause. This article is a practical record based on the primary records of implementation and fixes in a service I develop at my own company.