What your chunk-length distribution tells you

There is one plot worth making about a chunking pipeline: the distribution of chunk lengths. It takes a few minutes, it needs no labels and no query set, and the shape of it identifies most splitter defects by inspection.

The reason it works is that a splitter’s bugs are bugs about size. Every one of them leaves a signature in the histogram, and the signatures are distinguishable.

Make the plot correctly

Three details decide whether the plot is informative.

Measure in the unit of your constraint. If your ceiling is an embedding model’s input length, plot token counts, because a character histogram cannot show you which chunks are near that ceiling. Characters and tokens diverge most on the content most likely to be over.

Plot per document type, not just overall. The whole point is to find the one class that is being mangled, and a combined histogram hides it. Small multiples, one per type.

Include the chunks you would rather not count. Empty chunks, whitespace-only chunks, chunks that are a single heading. Pipelines often filter these before storage, which is fine — but count them before the filter, because their number is a diagnostic.

The shapes and what they mean

A hard spike at exactly the maximum. A large mass of chunks all sitting at the size limit means the limit, not the document, is choosing your boundaries. Some of that is unavoidable on long flowing prose. A tall spike means most of your corpus is being cut by the counter, and if you thought you were splitting structurally, you are not — the structural stage is falling through to the fallback.

A pile at the very bottom. Chunks of a few characters. Almost always one of: a heading immediately followed by a subheading, so the splitter emitted the heading alone; list items split individually; a document whose extraction produced fragments; or the trailing remainder of every document. The remainder case is the common one and it is worth checking specifically — if every document contributes one tiny final chunk, your loop is emitting a leftover instead of merging it back.

A long right tail past the maximum. Chunks larger than your configured limit. This should be impossible and frequently is not, because atomic-unit handling overrides the size cap: a table or a procedure that must stay whole. That is intentional. What is not intentional is a chunk several times the limit, which means an atomic unit was never checked against the ceiling at all, and if the ceiling was your embedding model’s input length, the tail of that chunk was silently discarded.

A bimodal distribution within one document type. Two humps means two behaviours. Usually a structural splitter succeeding on some documents and falling through on others — the hump at the size limit is the failures. Sample from each hump and compare the source documents; the difference between them is your bug.

Very wide spread with no structure. Characteristic of structural splitting on documents with uneven sections, and it is a genuine property rather than a defect. It still costs you: a two-line chunk and a nine-page chunk are competing on the same similarity scale, and the short one has a much sharper vector. Wide spread is the signal to consider merging the small ones.

A gap. No chunks in some length band, with mass either side. Usually an artefact of a merge rule — anything under the threshold got merged up past it — and usually harmless. Worth understanding rather than ignoring, because a gap you cannot explain means a rule you did not know you had.

What a bottom-of-the-histogram chunk actually is

The pile at the bottom is the shape people find hardest to interpret, so it is worth seeing what produces it. Here is one document’s output, with the tiny chunks marked.

--- chunk 1  (14 chars)  ← bottom bucket ---
## Rate limits

--- chunk 2  (9 chars)  ← bottom bucket ---
### Scope

--- chunk 3  (198 chars) ---
Limits are applied per API key rather than per account. An
account with four keys has four independent budgets, and
exhausting one does not affect the others.

--- chunk 4  (11 chars)  ← bottom bucket ---
### Reset

--- chunk 5  (156 chars) ---
Limits reset on a rolling window measured from the first
request in the window, not at a fixed time of day.

--- chunk 6  (3 chars)  ← bottom bucket ---
---

Four of six chunks are in the bottom bucket and none of them contains any content. The splitter is emitting every heading as its own chunk because a heading is followed by a separator it treats as a boundary, and chunk 6 is a horizontal rule. The fix is not a size parameter — it is that a heading belongs to the text beneath it, which is what merging is for and what prefixing the heading path onto its body achieves directly.

The point about the histogram is that the bottom bucket had a count of four here, and the count is what made you go and look.

The other counts worth having

The histogram is the plot. These four numbers belong next to it, and each is one query.

Chunks per document. A document that produced one chunk may be correctly passed through or may have failed to parse. A document that produced four hundred is either enormous or was split per line. The extremes of this count find broken extraction faster than anything else.

Proportion of chunk text that is duplicated. For each chunk, how much of its text appears in another chunk. High values mean overlap is set aggressively, and the cost lands as near-duplicates crowding every result set — which overlap always risks and which the histogram alone will not show you.

Chunks with no heading path. Should be a small number and often is not. Each one is a chunk that does not know what document section it came from.

Chunks beginning with a lowercase letter. A direct measure of how often boundaries fall mid-sentence. It will never be zero, and the trend across splitter changes is more useful than the absolute value.

What the distribution cannot tell you

It says nothing about whether the boundaries are in the right places. A histogram of a corpus split perfectly at section boundaries and one split at random offsets can look similar if the sections happen to be evenly sized. Length is a proxy for a splitter’s behaviour, not for chunk quality.

It says nothing about content. A corpus of chunks that are all navigation menus produces a tidy distribution.

And a healthy-looking distribution is not evidence of anything much — it means no size-shaped bug is present. The failures that remain are semantic, and you find those by reading the chunks.

Use it as a regression check

The real value is not the first plot, it is the second one. Store the distribution as a small set of numbers — count, median, the tails, the four counts above — every time you index. Then a splitter change that was supposed to affect markdown and moved the transcript distribution shows up immediately, and so does the day a new document class arrives and lands entirely in the bottom bucket.

That is the cheapest early warning available for an ingestion pipeline: not a quality metric, just a shape you have seen before and would notice changing.