Documents you should not chunk

Chunking is a default rather than a decision, so it gets applied to everything. For a class of documents that is actively harmful: the split removes the property that made the document useful, and no parameter choice recovers it.

Recognising these is worth more than tuning, because the fix is free. You stop doing something.

Documents whose meaning is the whole

Documents that are one statement. A change record. A decision log entry. An approval. A short release note. The document is one assertion, already the size of a chunk, and splitting it produces halves that assert nothing.

Documents whose value is the comparison. A two-option trade-off analysis, a before/after summary, a competitor matrix. The content is the relationship between the parts. Retrieve one column of a comparison and you have a claim with its counterweight removed — which is not an incomplete answer but a misleading one.

Documents that are a single structured object. A configuration file, an API schema, a data dictionary entry, one contract clause held as its own file. Splitting a JSON document mid-object gives you invalid syntax and a fragment whose keys have lost their parent path.

Short documents generally. Anything that comfortably fits your size limit. If a document is already under the ceiling, the correct splitter output is the document, and this overlaps heavily with corpora that are already chunked by their authors.

Documents with a conclusion that qualifies everything above it. An incident review whose last paragraph says the root cause was later found to be something else. Split it, and the corpus now contains the superseded analysis as freely retrievable content.

The split, shown

A decision record, split at a size limit and then kept whole.

=== SPLIT ===
--- chunk 1 ---
## ADR-014: Use a single shared index

Status: superseded by ADR-021

We considered one index per tenant and one shared index with
tenant filtering. Per-tenant indexes give hard isolation and
simpler deletion.
--- chunk 2 ---
We chose the shared index because per-tenant indexes did not
fit the operational budget at the tenant count we projected,
and filtering was judged sufficient for our isolation
requirements.

Chunk 2 is the strongest retrieval target for “should we use one index or many,” and read alone it says a shared index was chosen, for reasons, with no mention that this decision was superseded. Chunk 1 has the status line and the alternative that was rejected. Either chunk alone misrepresents the document; the first by omitting the decision, the second by omitting that the decision no longer stands.

=== KEPT WHOLE ===
--- chunk 1  {doc_type: adr, id: ADR-014,
              status: superseded_by ADR-021} ---
[Architecture Decisions → ADR-014: Use a single shared index]
Status: SUPERSEDED by ADR-021

We considered one index per tenant and one shared index with
tenant filtering. Per-tenant indexes give hard isolation and
simpler deletion. We chose the shared index because
per-tenant indexes did not fit the operational budget at the
tenant count we projected, and filtering was judged
sufficient for our isolation requirements.

One chunk: the options, the choice, the reasoning and the status. The status is also a field, so the whole class of superseded decisions can be filtered out at query time rather than argued with.

How to recognise them

Four tests, applied per document class rather than per document.

Does any part of it make a claim the rest contradicts or qualifies? If yes, splitting manufactures quotable falsehoods. This is the strongest signal on the list.

Is the document already under your size limit? Then there is nothing to decide. Check the median length per document class; classes whose median is well under the ceiling should be passed through explicitly rather than by luck.

Would a person hand you half of it as an answer? If the honest response to “what does this document say” requires all of it, the document is one unit.

Is the structure a single object rather than a sequence? Prose is sequential and tolerates cutting. A schema, a matrix, a form and a config file are not sequences, and cutting them yields invalid fragments rather than partial ones.

Making pass-through explicit

The important part: do not rely on the size limit to protect these documents. A splitter that happens not to split a short document will start splitting it the day the document grows, or the day someone lowers the size limit for unrelated reasons, and nothing will announce the change.

Instead, route these classes to a no-op handler by document type. Then add three things:

An explicit ceiling with a structural fallback. Above some size you do split, and when you do, split on the document’s own headings rather than on a count. A too-long ADR becomes context/decision/ consequences, not two halves.

A field recording that pass-through was chosen. So a bad chunk is traceable to a decision rather than to an accident of length.

The document’s own status and dates. A whole-document chunk is often a snapshot of a position that changed. Status and effective date are what let a query avoid the superseded ones, and on this class of document they matter more than any splitting decision.

What it costs to keep them whole

Larger chunks with less specific vectors. A whole document averaged into one vector is the large end of the size trade-off, and it will lose to a sharper chunk on narrow queries. That is a real cost and it is the right cost to pay when the alternative is a fragment that lies.

Uneven sizes in your index, since a whole-document class sits at a different scale from your split classes. Read the length distribution per document type rather than corpus-wide once you do this.

A ceiling you must still respect. Whole is not unbounded. Verify these against your embedding model’s input limit, because keeping a document whole is exactly how a chunk ends up over it and silently truncated.

Coarse citations. “This decision record” rather than “this paragraph.” Acceptable for a one-page document and not for a fifty-page one.

If the specificity loss is genuinely painful, the answer is not to split them — it is to match on small units and return the whole document, which keeps the document intact as the reading unit while giving the retriever something sharper to match against.

How to tell you got it right

Count chunks per source document and look at the documents that produced exactly two. Two chunks means a document just over the limit, cut roughly in half, and in this class the second half usually contains the conclusion. That list is short and reading it tells you immediately whether your ceiling is falling in the middle of a document class that should have been passed through.

Then take the classes you decided to keep whole and check the opposite failure: are any of them so large that a single vector cannot represent them, or so large they are being truncated? Pass-through is a decision with a size range, and the range has a top as well as a bottom.