In two apps of completely different natures, what decided the search strategy in the end was measurement

Summary

When vector search responses swelled from 5 seconds to 17 and timeouts came thick and fast, I nearly pinned the cause on the concurrency of the searches. What running two apps of completely different natures, each for several months, at the company I develop them at on my own, showed me was this: both the search strategy and the adoption or rejection of each small tweak came down, in the end, not to general wisdom or vendor recommendations but to measuring on my own app’s data and the searches actually made against it.

The measured numbers betrayed my prior estimates at every turn. I had estimated the cost of vectorizing from the ratio of a document’s token count to its character count, and had roughly put that ratio at 2.0. Before vectorizing all the documents, I first vectorized just one tenth as a trial and measured, and the ratio turned out to be about 0.674, which brought the projected cost for the full set down by about 70 percent. For the width of the search that vector search uses to look for nearby candidates, too, I expected that widening it would pick up better candidates, so I widened it from the default of 40 to 100 and measured; but the similarity of the candidates lined up at the top of the results barely changed, while the run time of a single search alone grew worse by about 300 times, so I found there was no benefit to raising the default width. Vectorizing a search sentence after inflating it with a large language model was supposed to catch the intent even of a short search sentence and raise accuracy. Yet the gap in similarity, the clue for telling apart two kinds of candidates of widely differing nature, was about 0.13 with the plain search sentence but shrank to about 0.04 once inflated, and since the numbers showed it was counterproductive, I withdrew it.

Who this is for and what you can take away

This article is for developers who have vector search running in a real service and are unsure whether they can adopt vendor comparison articles or public benchmarks as they stand. You can take away a pattern of judgment for deciding, by measuring on your own app’s data and its actual searches, which to make the default between full-text search and vector search, how to tell a tweak that helps accuracy from one that does not, and whether to look for the real cause of a slow search on the hardware side or the software side. That two apps of differing natures arrived at the same conclusion along separate lines of development is itself, I think, evidence that this pattern of judgment is not a coincidence of one app alone. This article is a reflection based on the operational first-hand records of the search infrastructure of two apps I develop at my own company.