The Chunking Problem

The ceiling your embedding model puts on chunk size

There is one number in a chunking pipeline that is not a trade-off. Your embedding model accepts inputs up to some length, and text past that length does not get embedded. It is not compressed, weighted less, or handled gracefully. It is cut off, and nothing tells you.

Every other decision on this site is a judgement call about your corpus. This one is a constraint, and the only question is whether you are inside it.

What truncation does

An embedding model produces a fixed-size vector from an input sequence. That sequence has a maximum length. Text beyond it is discarded before the model sees it, and the vector you get back describes the part that fit.

The consequences compound.

The tail is unsearchable. Content past the limit contributes nothing to the vector. It is present in your stored chunk, so a reader who receives the chunk can see it, and no query will ever retrieve the chunk because of it. You have material in your corpus that cannot be found.

Nothing errors. The call succeeds, returns a vector of the right shape, and the pipeline records success. Some client libraries warn, some silently truncate by default, and the behaviour is a configuration detail rather than a guarantee. This is why the failure survives for months.

It is concentrated in your longest chunks, which are usually the ones you deliberately kept whole: the big table, the long procedure, the section you refused to split. The atomic-unit handling that protects meaning is the same handling that produces over-limit chunks.

Partial vectors are plausible. A chunk whose first half was embedded still retrieves for queries about its first half. It does not look broken. It looks like a chunk that is inexplicably bad at some queries and fine at others.

The split, shown

A procedure kept whole to protect it, against the same procedure split under the ceiling.

=== ATOMIC UNIT, OVER THE LIMIT ===
--- chunk 1  (kept whole: 9-step procedure + preamble) ---
## Restoring from a snapshot

[preamble and steps 1–6 ...]
                              ← model's input limit falls here
7. Re-point the application at the restored instance.
8. Verify replication has resumed before removing the
   snapshot.
9. Delete the temporary security group created in step 2.

[embedded: preamble through step 6]
[discarded silently: steps 7, 8, 9]

The stored chunk contains nine steps. The vector describes six. A query about “removing the snapshot after a restore” matches on nothing, because the only text answering it was not embedded. If the chunk is retrieved for some other reason, all nine steps are shown — so the content is intact and the index is lying about it.

=== SPLIT UNDER THE CEILING, LABELLED ===
--- chunk 1  {procedure: "Restoring from a snapshot", part 1 of 2} ---
## Restoring from a snapshot  (steps 1–6 of 9)

[preamble and steps 1–6 ...]

--- chunk 2  {procedure: "Restoring from a snapshot", part 2 of 2} ---
## Restoring from a snapshot  (steps 7–9 of 9)
Preceding steps: create a temporary security group, restore
the snapshot to a new instance, wait for it to become
available.

7. Re-point the application at the restored instance.
8. Verify replication has resumed before removing the
   snapshot.
9. Delete the temporary security group created in step 2.

Splitting the procedure is a real cost — it is the thing to avoid where you can. Splitting it deliberately, with position labels, is strictly better than keeping it whole and having a third of it be invisible.

Aim below the ceiling, not at it

The limit is a cap, not a target, and two things sit between them.

Anything you prepend counts. A heading path, a document title, a generated context sentence, a serialised metadata block — all the enrichment that makes a chunk self-describing consumes input length. If you size chunks at the ceiling and then add a prefix, every chunk is over.

Length and specificity trade off long before the ceiling. A chunk at the maximum input length has a vector averaging a great deal of content, which is the large end of the size trade-off. Models that accept long inputs do not thereby produce good long-input embeddings; accepting a long sequence and representing it precisely are different properties. The ceiling is where correctness stops. Useful size is usually well below it.

So: choose a chunk size for retrieval reasons, verify it is comfortably under the ceiling including prefixes, and treat the remaining headroom as room for the atomic units you refuse to split.

The procedure

  1. Find your model’s input limit and write it down where the pipeline can read it, not in someone’s memory. It is a property of the specific model, and it changes when you change models.
  2. Count in the model’s own tokeniser. A character-based estimate is least reliable on exactly the content most likely to be over.
  3. Measure the final string, after all prefixes and enrichment. The thing to check is what gets sent, not what came out of the splitter.
  4. Assert, do not hope. Refuse to embed anything over the limit. Fail the ingest, or split further, or log loudly — but do not let a silent truncation be a valid outcome of your pipeline.
  5. Report the over-limit cases as a list, not a count. Each one is a document and a decision: split it, restructure it, or accept the loss knowingly.
  6. Re-run the check when you change embedding models. A new model with a different limit and a different tokeniser invalidates every previous verification.

What this does not solve

It does not tell you what size to use. Being under the ceiling is necessary and says nothing about whether your chunks are any good.

It does not help with content that genuinely will not fit any chunk — a wide table with two hundred columns, a single enormous generated document. Those need restructuring at the source, or a parent/child arrangement where the matching unit is small and the large object is only ever read.

And it says nothing about the generation side. How much retrieved material a language model can accept is a separate limit with separate arithmetic, and the two get conflated constantly. The ceiling that truncates a chunk is the embedding model’s.

How to tell whether it is happening now

Do not reason about it. Take the exact strings your pipeline sent to the embedding model — including prefixes — tokenise them with that model’s tokeniser, and find the maximum.

If the maximum equals the model’s limit exactly, you are truncating, because a distribution of natural text lengths does not pile up precisely at a round number by chance. A spike at the ceiling in the length distribution is the same signature seen from further away.

Then check the population above the limit rather than at it, group it by document type, and read the list. It is usually short, usually one or two document classes, and usually the ones you were most careful with.

chunk-size