Building search for a membership site as two tracks, free and paid

Summary

I designed and built the search for a membership site as two tracks: a keyword search anyone can use, and a meaning-based search for subscribers only. On the free side, I made it cost nothing no matter how many times search is used. A membership site’s search carries a condition of its own: what the search is allowed to touch differs between readers who read for free and readers who subscribe. I did not fold the search into one but split it into two tracks precisely to fit this condition. This article is a record of the process of designing and building those two tracks.

Let me lay out the overall shape first. On the free side, I built a static index from the published parts alone and had the reader’s own device query it, so it costs nothing no matter how many times search is used. On the paid side, it is a search by nearness of meaning, opened only to readers who have subscribed, and here I put three separate safeguards: a brake so the cost does not swell with the number of readers, a data-layer permission so the paid body text can never leak, and a privacy structure that keeps none of the reader’s questions.

What I most want to write about in this article is two decisions where we took back something we had once built. One is how I came to throw away, in two stages, the whole store meant to cache search questions for speed, and arrived at a general line of not reusing readers’ input across readers. The other is the story of stopping a search that always returns some number of results for whatever you type, and instead honestly returning zero results for a question that does not reach a certain line of nearness of meaning, a line I decided by measurement rather than by principle. In the body readable by subscribers, I write out concretely and in order, from the two-track boundary to the data permission that stops leakage, the cost brakes, the foundation of the embeddings, the floor for zero results, and the structure that keeps no questions.

Who this is for and what you can take away

This article is for people building a membership site, or a mechanism to sell paid articles, on their own, and wondering how to provide search. You can take away the boundary that splits the build of search between free readers and paid readers, and the idea of a static index that makes the free side cost nothing. To run a paid meaning-search cheaply and safely, I also write the method of cutting cost loose from the number of readers, and the form of stopping leakage with data permissions rather than application code. I go as far as one answer to the balance between privacy and efficiency: how much of the content of a reader’s search to hold at hand, and where to stop holding it. This article is a practical record based on the process of building this very site.