Summary
When you move a generative AI model to a new generation, the evaluation setup you already have is usually tuned for the old generation. Measured with that setup as it stands, the new generation can come out lower than it really is. I measured this switch on a service I develop at my personal company, and the fulfillment rate for the reason tags attached to citations came out at 26% for the new generation as it was, which looked worse than the old one. Once I made the conditions fair and measured again it recovered to 42%, and in absolute counts the new generation was ahead. On a dedicated test that machine-judges what the old generation cannot do, the old generation scored 11/16 against the new generation’s 16/16. And after passing the evaluation and putting it into production, a response timeout appeared, something that had never shown up offline, and I went as far as rolling the depth of thinking back one step.
In the body I start with the design of the switch, a design for confining the cost increase to answer generation alone. Next I explain the mechanism by which an old-generation metric measures a new generation low, and how the look of a rate changes when the denominator moves, and then move on to how I came to judge by reading the output itself instead of the aggregate. On that basis I write in order about the construction and the results of the dedicated test of 8 tasks that machine-judges the difference in ability, and then the measurements of the timeout that appears only in production and the trade-off of lowering the thinking level.
Who this is for and what you can take away
This article is for people who have generative AI built into a product and who face a decision about switching each time a model generation turns over. You can take away the concrete mechanism by which evaluation conditions built for the old generation measure a new one low, and the steps for checking what a metric assumes and making the conditions fair. I also write how to build an ability-difference test you can reuse as the check on whether to adopt the next generation, and how to handle the production axes that offline evaluation does not contain, namely response time and platform constraints. This article is a practical record based on the primary records of this switch on a service I develop at my personal company.
The rest of this article is paid
You can read the rest by buying this article on its own, or with a subscription that covers every paid article.
The paid part is about 6,000 characters, roughly a 12-minute read.
Read just this article
From 300 JPY
Buy this article on its own. The exact price is shown at checkout. Purchased articles stay readable whenever you sign in with the email used at purchase.
Read every paid article
Standard is 490 JPY / month
Four new paid articles ship every month. For yearly billing and the full comparison, see the plans.
Add dialogue and columns
Premium is 980 JPY / month
Premium adds the subscriber-only columns and dialogue with matsumotory-kun on top of every paid article. See the plans for details.
Prices include tax. Purchases and subscriptions start after you sign in. Subscribers and readers who already bought this article can sign in and read the full article; for the plans, see the plans page.
Sales are available in supported regions only; the Terms list where we sell. All charges are in Japanese yen.