Read fifty chunks — The Chunking Problem

Almost nobody looks at their chunks. Pipelines get built, indexes get populated, retrieval gets described as “not great,” and at no point does a human read the actual text that was stored.

Sampling and reading fifty chunks takes about twenty minutes and finds the majority of chunking defects in a new pipeline. It is not a substitute for measurement — it is what you do first, because it finds the bugs that would otherwise make your measurements meaningless.

Why reading beats scoring here

A retrieval score tells you that something is wrong. It does not tell you that your table headers are being severed, and it cannot, because the score has no concept of a table header.

Chunking defects are categorical and visible. A chunk either has its heading or it does not. It either begins mid-sentence or it does not. It either contains three orphaned table rows or it does not. These are yes/no properties of a piece of text, which makes a human eye the correct instrument and an aggregate number the wrong one.

There is a second reason, less obvious. An aggregate score averages over document types, so one badly handled class — the slide decks, the scanned PDFs, the transcripts — disappears into a corpus-wide mean. Reading a stratified sample cannot average anything away.

Building a labelled query set and measuring retrieval quality properly is a separate discipline and you will want it eventually. This is the cheap thing that comes first.

How to sample

Stratify by document type. Not fifty random chunks from the whole index — fifty spread across your document classes, weighted toward the ones you suspect. Random sampling from a corpus that is 90% markdown tells you about markdown.

Include the extremes deliberately. Take the ten shortest chunks and the ten longest in addition to your random sample. Both ends are where splitters misbehave, and neither is likely to appear in fifty random draws.

Sample chunk positions, not just chunks. Include some first chunks of documents and some last ones. Last chunks are where trailing fragments, signature blocks and orphaned appendices collect.

Look at the stored record, not a rendered view. You want the exact text that was embedded, including any prefix you added and any metadata you attached. If your pipeline embeds a different string than it stores, print the embedded one — that is the thing retrieval sees.

What to look at

For each chunk, ask five questions in order. They take a few seconds each.

1. Could I tell what this is about, if I had never seen the document? The single most useful question. If the answer is no, note why: no heading, pronoun with no antecedent, table rows with no header, a bare list of labels.

2. Does it start and end at a plausible place? Lowercase first character, a sentence trailing off, a dangling identifier. These indicate the boundary was chosen by a counter.

3. Is it one thing or several? A chunk covering four subjects will have a diluted vector, which is the large end of the size trade-off appearing in a specific place you can point at.

4. Would this alone answer a question completely? Not “is it relevant” — is it sufficient. A chunk containing half a policy will be retrieved and will produce a confident partial answer, which is the worst failure mode available.

5. Is the metadata right? Heading path present and correct, source identifiable, date sensible. A wrong heading path is worse than a missing one, because it is embedded.

What twenty minutes of reading looks like

Four consecutive chunks from one document, as stored. Read them the way the retriever will see them — one at a time, with nothing either side.

--- chunk 31  {heading_path: ["Access control"]} ---
## Access control

--- chunk 32  {heading_path: ["Access control"]} ---
Permissions are evaluated at request time, not at session
start. A permission revoked mid-session takes effect on the
next request. There is no cache to invalidate and no
--- chunk 33  {heading_path: ["Access control"]} ---
sign-out required.

| Role | Read | Write | Admin |
| --- | --- | --- | --- |
| Viewer | yes | no | no |
--- chunk 34  {heading_path: []} ---
| Editor | yes | yes | no |
| Owner | yes | yes | yes |

Roles are assigned per workspace, not per account.

Five defects, all visible without a metric. Chunk 31 is a heading with no body — the splitter cut immediately after it. Chunk 32 ends mid-sentence. Chunk 33 opens with the two words that finish that sentence, then starts a table. Chunk 34 is three table rows with no header row and no heading, and its heading_path is empty, which means the heading tracker lost its state at the table.

Question 1 fails for 31 and 34. Question 2 fails for 32 and 33. Question 5 fails for 34. That is one document, four chunks, and a to-do list.

What the sample usually shows

The defects cluster. In practice a first read finds some subset of:

Chunks that are entirely navigation, footers, tables of contents or signature blocks.
Table fragments with no header row.
Chunks whose first line is a heading and whose body is empty, because the splitter cut immediately after a heading.
Chunks that are one sentence, adjacent to chunks that are three pages.
The same text appearing in three consecutive chunks, which means overlap is set higher than it needs to be.
Chunks with a heading path from the wrong section, because the heading tracker did not reset.
Chunks that are empty, or whitespace, or a single punctuation mark.
One document type that is uniformly unusable — usually decks, scans or transcripts.

Every one of those is a fix in the splitter, and every one of them is invisible in a mean score.

Write down what you find

Keep a list, not an impression. For each defect: what it was, which document type, and how many of the fifty had it. That list has three uses.

It orders the work, because a defect appearing in twenty of fifty chunks matters more than one appearing once, and it is now obvious which.

It becomes a set of automated checks. Nearly every defect above can be expressed as a query over the index: chunks under N characters, chunks starting with a lowercase letter, chunks containing a pipe character but no header separator, chunks with an empty heading path, chunks whose text is a substring of another chunk. Write each one once and run it on every re-index. This is where reading turns into regression protection.

It becomes your candidate list for real evaluation. The defects you found are hypotheses about what is hurting retrieval, and a labelled query set targeted at those hypotheses is far more informative than a generic one.

Read the neighbours too

One extension worth the extra ten minutes. For ten of your chunks, pull the chunk before and the chunk after from the same document and read the three in sequence.

This finds a different class of defect: text that fell between chunks and was lost entirely, text duplicated across a boundary, and boundaries that split a sentence in a way that is invisible when you read either side alone. It also shows you whether a parent unit would have helped, because you can see directly how much of the answer sits on the other side of the cut.

When to do it again

Every time you change the splitter, and every time a new document type enters the corpus. Both are moments when a class of defect can appear across the whole index without any metric moving noticeably.

Keep one fixed sample of source documents for this — the same twenty files every time — so that reading the output twice tells you what changed rather than what varies. Fifty chunks from a stable sample, before and after, is the cheapest before/after comparison available and it answers questions no score can: not “is it better” but “is it doing what I told it to.”