Overlap, and what it actually buys
Overlap is the second number in every chunking tutorial and the least examined. Chunk size gets debated; overlap gets set to ten or twenty percent of it and never mentioned again.
It is doing something specific, it has a real cost, and in a lot of pipelines it is compensating for a splitter that shouldn’t have been used in the first place.
What it’s for
A boundary can fall in the middle of an idea. Overlap means the material near a boundary appears in both adjacent chunks, so whichever one is retrieved, the idea is intact in at least one.
Without overlap:
--- chunk 1 ---
...the migration runs in two phases. The first phase copies
all existing rows to the new table while the application
continues writing to the old one.
--- chunk 2 ---
The second phase acquires a brief lock, copies rows written
since the snapshot, and swaps the table names. Total lock
time is typically under a second.Query: “how long is the table locked during migration?”
Chunk 2 answers it. Fine — this boundary happened to fall between two complete thoughts. Now shift the cut forty words earlier:
--- chunk 1 ---
...the migration runs in two phases. The first phase copies
all existing rows to the new table while the application
continues writing to the old one. The second phase acquires
a brief lock, copies rows written since the
--- chunk 2 ---
snapshot, and swaps the table names. Total lock time is
typically under a second.Now chunk 2 opens mid-sentence and chunk 1 stops mid-sentence. Chunk 2 still contains the answer, but its vector is polluted by a sentence fragment with no subject, and “brief lock” — the phrase most likely to match the query — is in chunk 1, which doesn’t have the duration.
With overlap, the tail of chunk 1 is repeated at the head of chunk 2:
--- chunk 1 ---
...the migration runs in two phases. The first phase copies
all existing rows to the new table while the application
continues writing to the old one. The second phase acquires
a brief lock, copies rows written since the
--- chunk 2 (overlapping) ---
The second phase acquires a brief lock, copies rows written
since the snapshot, and swaps the table names. Total lock
time is typically under a second.Chunk 2 is now a complete statement. That is the entire mechanism: overlap is insurance against a badly placed boundary.
What it costs
Four costs, in roughly the order they bite.
Index size scales with it. Overlap of 20% means roughly 20% more chunks, 20% more vectors, 20% more storage, and 20% more embedding calls at index time. On a small corpus this is irrelevant. On a large one it’s a line item, and on a corpus you re-index frequently it’s a recurring one.
Near-duplicates crowd the top-k. This is the one that actually degrades quality. Two chunks sharing 200 characters of text have similar vectors, so if one scores highly the other usually does too. Ask for five passages and you may get three that are substantially the same text:
Retrieved for "how long is the table locked?":
1. [chunk 2] "The second phase acquires a brief lock..."
2. [chunk 3] "...copies rows written since the snapshot, and
swaps the table names. Total lock time is..."
3. [chunk 1] "...The second phase acquires a brief lock,
copies rows written since the"Three of five slots spent on one passage. The other two documents that might have had relevant material never made it. Higher overlap makes this worse, and it’s why “just increase the overlap” is rarely the fix people hope for.
Duplicated text in the window. Whatever reaches the model contains the same sentences more than once. That’s paid-for space returning nothing.
Attribution gets muddy. When a sentence exists in three chunks, “which chunk did this come from” has three answers. Deduplicating for citation display becomes a step you have to write.
When it’s worth it
Fixed-size splitting on continuous prose. This is overlap’s home ground. If you’re cutting flowing text at arbitrary character counts, boundaries will land badly and overlap is the cheapest mitigation available.
Documents with heavy anaphora. Text that constantly refers backwards — legal writing, academic prose, anything full of “the aforementioned,” “this approach,” “such cases.” A chunk that opens with “This approach fails when…” needs the previous sentence to mean anything.
Small chunks. The smaller the chunk, the higher the proportion of it that sits near a boundary, and the more likely a cut damages it.
When you can’t parse the document. No structure available, no better option.
When it isn’t
You split structurally. If chunks end at heading, clause, or function boundaries, the boundary is where the idea ends. Repeating the last paragraph of section 3 at the top of section 4 adds noise, not context. Structural splitting makes overlap largely redundant, and this is one of the underrated arguments for it.
Discrete records. FAQ entries, product rows, log lines, catalogue items. Each is self-contained. Bleeding one into the next actively harms it — a chunk containing the end of one product’s description and the start of another matches queries for both and answers neither.
Tables and code. Overlap doesn’t repair a split table; it gives you two broken tables sharing rows. These need format-specific handling, not more of the same wrong tool.
You already carry context another way. If every chunk gets a heading path and a document title, the severed-context problem is mostly solved without duplication. That’s a strictly better trade — you’re adding a small amount of unique framing rather than a large amount of repeated body text.
Better tools for the same problem
Overlap is a blunt instrument aimed at a real problem. Sharper ones exist.
Sentence-boundary alignment. Don’t cut mid-sentence at all — move the boundary to the nearest sentence end. This eliminates the fragment problem entirely, at zero index cost, and it removes most of the reason people reach for overlap. If you take one thing from this post, take this: align boundaries to sentences before you add overlap.
Contextual prefixes. Prepend the document title and heading path rather than the previous paragraph. A handful of tokens per chunk, unique rather than duplicated, and it addresses the actual failure (chunk doesn’t know what it’s about) rather than a proxy for it.
Parent/child retrieval. Embed and match on small chunks; return the larger parent for reading. The matching unit is precise, the reading unit is complete, and no text is duplicated in the index. More machinery to build, and it’s the right answer often enough to be worth the machinery.
Sentence-window retrieval. Index individual sentences; on retrieval, return the sentence plus its neighbours from the original document. Overlap applied at query time instead of index time — you get the surrounding context without storing anything twice. Requires keeping position information on each chunk, which you should be doing anyway.
Notice what these have in common: they fix the boundary problem without paying the duplication cost. Overlap is what you use when you can’t do any of them.
If you do use it
- Overlap on whole sentences, never on character counts. Overlapping 200 characters means starting a chunk mid-word. Overlap N complete sentences instead.
- Keep it modest relative to chunk size. The near-duplicate problem grows fast, and past a certain point you are indexing the same corpus several times over.
- Deduplicate the retrieved set before it reaches the model. Detect substantially overlapping passages and keep one. This recovers most of what overlap costs you at the top-k, and it’s maybe thirty lines of code.
- Never overlap across document boundaries. The last chunk of one document must not contain the first lines of the next. Sounds obvious; happens constantly when documents are concatenated before splitting.
The honest summary
Overlap is a patch for boundaries you couldn’t place well. If you can place boundaries well — because the document has structure, or because you align to sentences — you need much less of it than the default suggests, and possibly none.
Test it the way you’d test anything else here: same corpus, same query set, one variable changed. Look specifically at how many of your top-k results are near-copies of each other. That number tells you more than the retrieval score does.