Merging chunks that came out too small
You switch from counting characters to splitting on document structure, re-index, and the chunk count triples. Most of the new chunks are a heading and a sentence. Retrieval gets worse in a way that looks like the small end of the size trade-off, because that is exactly what it is.
Splitting on structure means accepting whatever sizes the structure produces, and real documents have a lot of very short sections. Merging is the second half of structural splitting, not an optimisation you add later.
Why structure produces confetti
Three patterns, all common.
Headings with almost nothing under them. A subsection that exists for navigation rather than content: two sentences, or a single cross-reference, or a sentence and a link.
Deep nesting. A document with four heading levels, where the leaves are one paragraph each. Split at the leaves and every chunk is a paragraph with a long heading path.
Structural elements emitted as chunks. A heading followed immediately by a subheading produces a chunk containing only the first heading. Horizontal rules, empty sections and stub sections do the same. These show up as a pile at the bottom of the length distribution.
The result is an index where a large fraction of chunks are too small to be about anything, and each of them is a candidate competing with your real content.
The split, shown
=== SPLIT AT EVERY HEADING ===
--- chunk 1 ---
## Notifications
--- chunk 2 ---
### Email
Email notifications are on by default.
--- chunk 3 ---
### SMS
SMS notifications require a verified phone number.
--- chunk 4 ---
### Webhook
Webhook notifications are configured per workspace.
--- chunk 5 ---
## Billing
--- chunk 6 ---
### Invoices
Invoices are issued monthly in arrears.Six chunks, none of which is a useful retrieval target. Chunk 3 is one sentence whose vector is about phone verification; a query about “how do I turn off notifications” matches none of them well, because no chunk contains the comparison the user is implicitly asking for.
=== MERGED WITHIN PARENT ===
--- chunk 1 {section: "Notifications"} ---
[Notifications]
## Notifications
### Email
Email notifications are on by default.
### SMS
SMS notifications require a verified phone number.
### Webhook
Webhook notifications are configured per workspace.
--- chunk 2 {section: "Billing"} ---
[Billing]
## Billing
### Invoices
Invoices are issued monthly in arrears.Two chunks, each about one subject, each containing the subheadings that make it navigable. And
crucially, chunk 1 stops at the end of Notifications rather than running on into Billing.
The merge rules
1. Only merge siblings under a shared parent. This is the rule everything else depends on. Two
adjacent subsections of Notifications belong together. The last subsection of Notifications and the
first of Billing do not, however small they both are, and a splitter that merges purely by adjacency
will produce that chunk sooner or later.
2. Merge upward into the parent, not sideways across it. When the children of a section are all small, the natural chunk is the section. Emit the parent with all its children rather than gluing children together into arbitrary groups.
3. Set a target, not a minimum you must reach. A merge loop that keeps consuming until it hits a minimum size will cross a parent boundary to get there if you let it. Make the parent boundary a hard stop and the size a preference. A small chunk that is genuinely the only thing in its section should be allowed to stay small.
4. Keep every heading in the merged text. After merging, the chunk covers several subsections, and the subheadings are the only thing indicating that. Dropping them to save tokens removes the structure you merged in order to preserve.
5. Attach the heading path of the merge’s root. A merged chunk’s heading_path is its parent’s path,
and the child headings live in the body. Keeping that field
accurate matters more after merging than before, because the chunk
now spans more of the document.
6. Absorb structural fragments rather than emitting them. A chunk that is only a heading should be prepended to the next chunk. A chunk that is only a horizontal rule or whitespace should be dropped. Handle these before the size-based merge so they do not consume the merge budget.
7. Merge before you consider overlap. Merged chunks end at real section boundaries, and overlap is insurance against boundaries that are not real. Doing overlap first means paying to duplicate text across boundaries you were about to remove.
What merging costs
Dilution, which is the whole point of not overdoing it. A merged chunk covers more topics, so its vector is less specific. Merge three subsections about email, SMS and webhooks and the chunk is about notification channels in general — better for “how do notifications work,” worse for “why didn’t my SMS arrive.” You have moved along the size trade-off deliberately, which is fine, and you should know which direction you moved.
Uneven sizes remain uneven. Merging fixes the bottom of the distribution and does nothing about the top. A corpus with two-line sections and nine-page sections still has both after merging.
Boundary decisions become harder to explain. “This chunk is section 4.2” is a clear provenance statement. “This chunk is sections 4.2 through 4.5, merged because each was under the target” is correct and less legible, and when a citation points at it, a reader has to work out which part was relevant.
A wrong merge is worse than a small chunk. A chunk containing the end of the refunds policy and the start of the shipping policy will be retrieved for questions about both, and it will answer with material from the wrong one. Small chunks fail by being incomplete; badly merged chunks fail by being misleading.
When to reach for something else instead
If merging keeps forcing you toward chunks that are too big to be specific, the actual answer may be that you want both sizes: small chunks for matching and larger ones for reading. Merging is a single-level compromise, and the compromise only exists because one span of text is doing two jobs.
And if the sections are small because the documents are small, they may not want splitting or merging at all — a corpus of short, self-contained articles is already chunked, and merging distinct articles together is the worst version of rule 1.
How to tell if it worked
Look at the bottom of the length distribution before and after. The pile of near-empty chunks should be gone, and the count of chunks whose body is empty after removing headings should be zero.
Then check the rule that actually breaks. For every merged chunk, verify that all of its content came from one parent section — you have the source offsets, so this is an assertion you can run over the whole index rather than a sample. Any chunk spanning two top-level sections is a merge that crossed a boundary it should have stopped at, and one such chunk is enough to justify fixing the rule.
Finally, read a sample of the merged chunks and ask the sufficiency question: does this chunk now answer something completely? If merging produced chunks that are longer and still incomplete, the merge did not help and the sections were not the problem.