In two completely different apps, what decided the search strategy in the end was measurement

Summary

When vector search responses swelled from 5 seconds to 17 and timeouts came thick and fast, I nearly pinned the cause on the concurrency of the searches. I run two completely different apps at my own company, each for several months now. What that showed me was this: both the search strategy and the adoption or rejection of each small tweak came down in the end to measuring on my own app’s data and the searches actually made against it, not to general wisdom or vendor recommendations.

The measured numbers betrayed my prior estimates at every turn. I had estimated the cost of vectorizing from the ratio of a document’s token count to its character count, and had roughly put that ratio at 2.0. Before vectorizing all the documents, I first vectorized just one tenth as a trial and measured, and the ratio turned out to be about 0.674, which brought the projected cost for the full set down by about 70 percent. For the width of the search that vector search uses to look for nearby candidates, too, I expected that widening it would pick up better candidates, so I widened it from the default of 40 to 100 and measured; but the similarity of the candidates lined up at the top of the results barely changed, while a single search took about 300 times longer to run, so I found there was no benefit to raising the default width. Vectorizing a search sentence after inflating it with a large language model was supposed to catch the intent even of a short search sentence and raise accuracy. Yet the gap in similarity, which is the clue for telling two kinds of candidates that sit far apart from each other, was about 0.13 with the plain search sentence but shrank to about 0.04 once inflated, and since the numbers showed it was counterproductive, I withdrew it.

I mistook the cause of a slow search more than once. When the very first search right after building the index came back abnormally slow, I started out blaming the disk. Both times the real cause lay somewhere else entirely: in the database statistics, in the way the search was assembled, and in the cache.

The body available to subscribers follows these failures in the order they happened. It starts with how I nearly blamed the hardware for the slowness and mistook the cause again and again, then traces how I closed off the software side one item at a time and fixed it. Next come the measured numbers that betrayed my prior estimates, the decision to retract a tweak that did not work, with the conditions of the verification at the time attached, and the design that moved the main axis of search quality from keywords and dictionaries to semantic closeness itself. It closes with re-deriving borrowed numbers from current primary sources, and with a tweak I still cannot say worked, left without a conclusion, ending on what was left in hand after running two apps.

Who this is for and what you can take away

This article is for developers who have vector search running in a real service and are unsure whether they can adopt vendor comparison articles or public benchmarks as they stand. You can take away a pattern of judgment for deciding, by measuring on your own app’s data and its actual searches, whether to make full-text search or vector search the default, how to tell a tweak that helps accuracy from one that does not, and whether to look for the real cause of a slow search on the hardware side or the software side. That two apps this different arrived at the same conclusion along separate lines of development is itself, I think, evidence that this pattern of judgment is not a coincidence of one app alone. This article is a reflection based on the operational first-hand records of the search infrastructure of two apps I develop at my own company.