Small to big: match on the fragment, return the section

A user asks a specific question, the right sentence comes back, and the answer is still incomplete because the sentence needed its paragraph. You make the chunks bigger. Now the sentence no longer comes back at all, because it is buried in a chunk about six other things.

That loop is the chunk-size trade-off and it is unwinnable as long as one span of text has to do both jobs. Parent/child chunking stops asking it to: index a small chunk for matching, store a pointer to a larger one, and hand the larger one to the model.

Two units, one document

The structure is two levels of the same document.

Child chunks are small — a sentence, a few sentences, a subsection. These are what get embedded and indexed. They are small on purpose, so each vector is about one thing.

Parent chunks are larger — the enclosing section, the full page, sometimes the whole document. These are stored but not embedded. Each child records which parent it belongs to.

At index time you produce both. At retrieval time a child matches, and what you actually read is its parent. Every child of the same parent resolves to the same text, so if three children of one section match, you deduplicate to one parent rather than shipping three overlapping fragments.

The split, shown

Here is one subsection of a runbook, split as children with their parent recorded.

--- PARENT p12  (section: "Failed deploys → Rolling back") ---
## Rolling back

A rollback restores the previous release artifact and the
previous configuration together. Rolling back the artifact
alone leaves the new configuration in place, which is the
most common cause of a rollback that does not fix anything.
Database migrations are not reverted by a rollback. If the
release included a migration, check whether it was
backwards compatible before rolling back.

--- CHILD c12.1  (parent: p12) ---
A rollback restores the previous release artifact and the
previous configuration together.

--- CHILD c12.2  (parent: p12) ---
Rolling back the artifact alone leaves the new
configuration in place, which is the most common cause of a
rollback that does not fix anything.

--- CHILD c12.3  (parent: p12) ---
Database migrations are not reverted by a rollback. If the
release included a migration, check whether it was
backwards compatible before rolling back.

Query: “why didn’t the rollback fix it?”

c12.2 is an almost exact match — its vector is about nothing except that failure mode. A 1,000-character chunk containing the whole section would have matched less sharply, because its vector also encodes artifacts, configuration and migrations.

And what gets read is p12, which contains the migration caveat that c12.2 does not mention and the user needs. Precision from the child, completeness from the parent. Neither chunk had to compromise.

Variants worth knowing apart

Sentence-window. Children are single sentences; the “parent” is assembled on demand from the sentences either side, using position. Nothing larger is stored — you keep an ordered index and expand outward at read time. Cheapest to build, and the window is a fixed number of neighbours rather than a meaningful unit, so it can still cut across a section boundary.

Fixed two-level. Children are small fixed chunks; parents are larger fixed chunks; each parent contains a known set of children. Simple, uniform, and the parent boundaries are as arbitrary as any fixed-size boundary.

Structural hierarchy. Children are the leaves of the document’s own structure, parents are their enclosing sections. This is the version that actually earns the machinery, because both units are meaningful: the child is a paragraph or a clause, the parent is the section it belongs to. It requires that you can parse the structure in the first place.

Summary-indexed. The child is not an excerpt at all — it is a generated summary or a list of questions the parent answers, embedded in place of the text. Powerful when the parent is long or badly written, and it makes your index depend on a generation step whose output can be wrong.

Whole-document parents. Children index into entire documents. Only sensible when documents are short; otherwise you have replaced a chunking problem with a very large reading unit.

What it costs

Two stores, and a join. You need the vector index for children and a keyed store for parents, plus a lookup on every read. That lookup is a second network hop in the request path if the parent store is remote. Not difficult, but it is infrastructure that a single-level pipeline does not have.

Deduplication becomes mandatory, not optional. Several children of one parent will frequently match the same query. Without a dedupe step you return the same parent text repeatedly, which is exactly the near-duplicate crowding that overlap causes — arrived at by a different route.

The parent is what you pay to read. Small children are cheap to embed and large parents are expensive to include in a prompt. Retrieving five children can mean reading five sections, and if your parents are large you have quietly moved to a much bigger reading unit than you would have chosen deliberately.

Two levels of consistency to maintain. When a document changes, children and parents must be regenerated together. A child pointing at a stale parent, or at a parent ID that no longer exists, is a failure that surfaces as a missing citation rather than as an error. Make the parent ID derivable from the document and its structure rather than from a counter, so a re-split does not scramble the mapping.

Chunk-level metadata has to be decided at both levels. The child needs enough to be filtered; the parent needs enough to be cited. Usually this means the same field set on both, with the child additionally carrying its parent ID and its offset within the parent.

What it does not fix

Answers that span parents. If the answer needs section 3 and section 9, a two-level hierarchy retrieves one of them. Parent/child sharpens matching and widens reading; it does not connect distant parts of a document.

Children that cannot stand alone. A one-sentence child full of pronouns still has a poor vector. c12.3 above works because it restates its subject; a child reading “This is not reverted either” is unmatchable no matter what its parent contains. The child still needs enough context to be about something — a heading prefix at minimum.

Bad parents. If the parent boundary is arbitrary, you have precise matching into an arbitrary span, and the completeness half of the trade-off is not actually solved. The quality of this strategy is the quality of the parent unit.

Whether to build it

Ask what your queries look like. If they are mostly specific lookups against long documents — a policy, a manual, a reference — the gap between the matching unit and the reading unit is wide, and parent/child closes it. If your documents are already short and self-contained, there is no gap and this is machinery for nothing; those documents are better left alone.

How to tell if it helped

Compare against a single-level index at the parent size, and separately against one at the child size, on the same query set. Parent/child should beat the large index on whether the correct passage was found at all, and beat the small index on whether what came back was sufficient to answer. If it only beats one of them, you have added a store and a join for something one level was already doing.

Then look at what you are actually returning. Log the parent lengths for a day of real queries. If the average parent is much larger than you expected, the machinery is working and you are paying for it in reading volume, which is a trade you should make on purpose rather than discover.