Tagged “chunk-size”
-
Why chunk size is a tradeoff, not a setting
Small chunks retrieve precisely and answer incompletely. Large chunks do the reverse. What each end actually breaks, and how to find your corpus's middle.
-
Documents you should not chunk
Some documents are one idea, and splitting them destroys the only thing they said. How to recognise them, and how a mixed corpus handles both kinds.
-
The ceiling your embedding model puts on chunk size
Every embedding model truncates past some input length, silently. That limit is a hard cap on chunk size, and it is not the cap you should aim for.
-
Merging chunks that came out too small
Structural splitting produces one-line chunks. Merging fixes that, and merging across the wrong boundary makes a chunk about two unrelated things.
-
What your chunk-length distribution tells you
Histogram your chunk lengths. A spike at the maximum, a pile of near-empty chunks and a long right tail each name a specific bug in your splitter.
-
Small to big: match on the fragment, return the section
Parent/child chunking splits the matching unit from the reading unit. It dissolves most of the size trade-off and adds a store to keep in sync.
-
Characters, words, tokens: what your splitter is counting
A 1,000-character limit is not a 1,000-token limit, and the two disagree most on the content you care about most. Pick a unit and know its error bars.