The embedding input limit did not justify the chunk unit

Summary

In the search feature of a service developed at my own company, the input limit of the embedding model turned out not to justify the choice of chunk unit. The search targets long specialized documents divided into levels. I call one entry at the level that search returns as a result an upper unit, and I call each block of body text that an upper unit is divided into a smallest component. At first, the AI agent explained the unit by the headroom under the input limit and by the chunk limit set for a different corpus. After I pointed this out, the AI agent looked again at the distribution of body text lengths and at published research, and chose to embed the smallest components.

The 90th percentile of the body text length of the smallest components was 360 characters, about 243 tokens, roughly 3% of the input limit of 8,192 tokens. Of the upper units, 64.5% have only one smallest component, so for these upper units, the text passed to the embedding model is the same whether the upper unit or each smallest component is embedded.

We (the AI agent and I) did not score the two options against our own ground truth set, so whether the newly chosen unit is better has not been confirmed. This conclusion covers only documents with a clear structure. It does not deal with text that has no structure, or with data in which queries and documents are about the same length.

What you can take away

Developers who decide the chunk unit can get the following four things from this article.

• You can count the smallest components in each upper unit, and estimate the range in which the two options pass different text and the amount of input.
• You can check which metric each reported value in published research on splitting methods and preambles was measured with.
• You can re-estimate the memory of an HNSW index, and consider changing settings before moving to a larger instance type.
• You can use a procedure for building a ground truth set without a person labeling answers, and four steps for deciding the unit.

This article rests on the design documents and code of a service developed at my own company, the AI agent’s work records, and published research. It does not include measurements that compare the two options on our own data.