Why chunk size is a tradeoff, not a setting
Someone asks what chunk size they should use and gets a number. They use it. Retrieval is mediocre in a way nobody can quite characterise, and the number is never revisited because it came from a tutorial and tutorials are where numbers come from.
There is no correct chunk size. There is a trade-off with two failure modes at the ends, and a region in the middle that depends on your documents, your queries, and your embedding model. This post is about what the two ends break, because you can’t find the middle without being able to recognise them.
What a chunk has to do at once
A chunk is doing two jobs that pull in opposite directions.
It has to match. At query time the chunk’s embedding is compared against the query’s. For that comparison to work, the chunk needs to be about the thing the query asks about — and one vector can only represent so much. A chunk covering four topics has a vector that sits somewhere between all four, close to none of them.
It has to answer. Once retrieved, the chunk is what the model reads. If the answer needs three sentences and the chunk contains one of them, retrieval succeeded and the system still failed.
Small chunks are good at the first job and bad at the second. Large chunks are the reverse. That’s the whole trade-off, and everything below is a consequence of it.
The small end: precise and useless
Take a support policy paragraph and cut it small:
--- chunk 1 ---
Customers may request a refund within 30 days of purchase.
--- chunk 2 ---
Refunds for annual plans are prorated from the cancellation
date.
--- chunk 3 ---
This policy does not apply to enterprise agreements, which are
governed by the terms of the individual contract.Now the query: “can I get a refund on my annual plan?”
Chunk 2 matches beautifully. Its vector is about almost nothing except prorated annual refunds. This is the strength of small chunks — a tight, unambiguous vector, and a retriever that can distinguish it from the fifty other paragraphs in the document.
And the answer is wrong, or at least dangerously incomplete. Chunk 1 has the 30-day window. Chunk 3 has the exception that might make the whole thing inapplicable. The model reads chunk 2, sees a confident statement about prorating, and tells the customer their refund will be prorated — neglecting to mention it must be within 30 days and doesn’t apply to them at all.
The failure modes at the small end:
Truncated answers. The retrieved chunk contains part of the answer. The model doesn’t know it’s partial, so it answers confidently and incompletely. This is the worst failure in the whole space because it looks like success.
Lost antecedents. “This policy,” “the above table,” “it,” “they” — small chunks are full of pronouns whose referents were cut away. A chunk beginning “This does not apply to enterprise agreements” is nearly meaningless in isolation.
Fragmentation. More chunks means more candidates, and the top-k now has to be larger to cover the same amount of source material. Retrieve five small chunks and you may get five fragments of one paragraph rather than five distinct facts.
The large end: complete and unfindable
Same source, cut large — the whole policy section as one chunk, say two thousand words covering refunds, cancellations, plan changes and enterprise terms.
Now the query matches against a vector that represents all of it. And here is the mechanism that matters: an embedding is a fixed-size vector regardless of input length. Compressing two thousand words into the same number of dimensions as fifty words means the specifics get averaged out. The resulting vector is a reasonable representation of “billing policy in general” and a poor representation of any particular sentence in it.
So a query about prorating on annual plans competes against every other billing question, and the chunk that contains the answer looks about as relevant as three others that don’t.
The failure modes at the large end:
Diluted vectors. The chunk is about too much to be strongly about anything. Recall drops for specific queries.
Everything looks equally relevant. When your chunks are large, most of them cover most topics, and score differences between them get small. Ranking becomes noise.
Wasted context. Retrieve five large chunks and most of what reaches the model is irrelevant. It costs money, and — separately — filling the window with mediocre material makes the relevant part compete for attention. How to spend a window is a different subject; the point here is that chunk size sets the unit of that spending.
Coarse citations. “Here’s the relevant section” is less useful than “here’s the relevant paragraph,” and a two-thousand-word chunk can only give you the former.
Same text, two sizes
To make the trade-off concrete, one source cut both ways:
=== SMALL (≈1 paragraph) ===
--- chunk 4 ---
Rate limits are applied per API key. The default limit is
documented in your dashboard.
--- chunk 5 ---
Exceeding the limit returns a 429 response with a Retry-After
header indicating when to retry.
--- chunk 6 ---
Limits reset on a rolling window, not at a fixed time.
=== LARGE (whole section) ===
--- chunk 4 ---
Rate limits are applied per API key. The default limit is
documented in your dashboard. Exceeding the limit returns a
429 response with a Retry-After header indicating when to
retry. Limits reset on a rolling window, not at a fixed time.
[...continues with authentication, pagination, versioning,
error codes, webhooks, and SDK installation...]Query: “what do I do when I get a 429?”
Small wins on retrieval — chunk 5 is exactly about 429s. But the complete answer needs chunk 6 too (rolling window, so a fixed wait is wrong), and nothing guarantees both come back.
Large wins on completeness — everything’s there. But its vector represents an entire API reference, so a 429 query has to beat every other section of the docs to surface it, and it may not.
Neither is right. The useful size for this corpus is somewhere around “one coherent subsection,” and you’d find that by looking at the documents, not by picking a number.
Where the middle is
Some structure for the search, none of it a number.
Start from the document, not from the parameter. Ask: what is the smallest span of this document that answers a realistic question on its own? In a FAQ that’s a Q&A pair. In an API reference it’s an endpoint. In a legal contract it’s a clause with its subclauses. In a narrative article it might be several paragraphs. That span is your target chunk, and it’s a property of the corpus. Which is also the argument for splitting on structure rather than characters.
Look at what your queries need. Fact-lookup queries (“what’s the refund window”) favour smaller chunks. Explanatory queries (“how does the refund process work”) favour larger. Mixed workloads are the argument for a parent/child arrangement — embed small for matching, return large for reading.
Check your embedding model’s input limit. Every model truncates past some number of tokens. Chunks exceeding it are silently cut, and the tail is embedded as if it doesn’t exist. This is a hard ceiling and it’s worth verifying you’re under it, because nothing errors when you’re not.
Measure, don’t reason. Build twenty to fifty real queries with known-correct source passages. Index your corpus at three or four candidate sizes. For each, check: did the correct passage come back, and was it complete enough to answer from? Pick the winner. This takes an afternoon and beats any recommendation, including this one.
Expect to revisit it when the corpus changes character or you change embedding model.
The uncomfortable part
Different documents in the same corpus often want different sizes. A knowledge base containing both one-line FAQ entries and forty-page policy documents has no single good answer, and forcing one is a compromise that serves neither.
The honest options are to chunk by document type with different rules per type, or to accept that one class of document will retrieve worse and know which. Both are better than a global setting nobody chose deliberately.
What you should not do is pick 512 because everyone else did. Everyone else has a different corpus.