Choosing separators for a recursive splitter

A recursive splitter takes a list of separators, tries the first one, and falls back to the next when the resulting pieces are still too large. Everybody configures the chunk size. Almost nobody configures the separator list, which is where the actual behaviour lives.

The default list is written for generic English prose. Changing it for your corpus is the cheapest improvement available to anyone who is not ready to write a parser.

What the list actually controls

The mechanism, stated precisely, because the details are where the surprises are.

Given a piece of text that is too long, the splitter tries to divide it on the first separator in the list. If the resulting pieces are all within the size limit, it stops. If any piece is still too long, it recurses into that piece with the next separator, and so on. When the list runs out it cuts at the size limit regardless.

Three consequences follow, and each one bites somebody.

Order is priority, not preference. The first separator that produces small-enough pieces wins. A separator late in the list only ever applies to text that the earlier ones could not divide finely enough.

The last entry is your true fallback. If the list ends with the empty string, the splitter will cut mid-word rather than exceed the size. If it ends with a space, it will cut mid-sentence but not mid-word. That single choice is the difference between the dangling identifier fragments you see in a bad index and their absence.

Reaching a separator means the previous one failed. So a chunk cut at a comma is a chunk that could not be divided at a paragraph or a sentence — which usually means it is one very long sentence, and that is worth knowing rather than papering over.

The split, shown

The same markdown, split with a generic list and then with one extended for the format.

=== GENERIC LIST: ["\n\n", "\n", ". ", " ", ""] ===
--- chunk 1 ---
## Supported regions

The following regions are available on all plans.

| Region | Code | Latency tier |
| --- | --- | --- |
| London | lon-1 | A |
--- chunk 2 ---
| Madrid | mad-2 | A |
| Berlin | ber-1 | B |

Additional regions can be enabled on request.

The paragraph separator \n\n divided the section into heading, prose, table and closing line — but the table was still over the limit, so the splitter recursed into it with \n, which is a row boundary. Every row is a valid split point as far as \n is concerned. Chunk 2 is two rows with no header.

=== EXTENDED LIST, MARKDOWN-AWARE ===
separators: ["\n## ", "\n### ", "\n\n", "\n", ". ", " "]
atomic:     table blocks, fenced code blocks
--- chunk 1 ---
## Supported regions

The following regions are available on all plans.
Additional regions can be enabled on request.
--- chunk 2 ---
## Supported regions — regions table

| Region | Code | Latency tier |
| --- | --- | --- |
| London | lon-1 | A |
| Madrid | mad-2 | A |
| Berlin | ber-1 | B |

Two changes did that. Heading patterns were put at the top of the list, so the splitter prefers to divide at a section rather than at a blank line. And the table was marked atomic, so \n was never allowed to reach inside it. The trailing empty string was also removed, so nothing will be cut mid-word.

Building a list for your corpus

1. Put the largest structural separators first. For markdown, the heading patterns, including the leading newline so a # inside a code fence or a sentence does not match. For contracts, the clause number pattern. For transcripts, the speaker-label pattern. The top of the list is where your format knowledge goes.

2. Keep the generic ones in the middle. Paragraph break, line break, sentence end. These handle everything your format-specific entries did not divide finely enough.

3. End with a space, not an empty string. Cutting mid-word produces fragments that match nothing. Exceeding the size by a few characters to finish a word is almost always the better trade — unless the corpus is a script with no spaces, where the space entry matches nothing and you need script-specific punctuation instead.

4. Add your script’s punctuation. 。, ！, ？, ، and the full-width Latin forms. A sentence separator of ". " finds nothing in Chinese or Japanese, so the splitter silently falls through to the end of the list on that part of your corpus.

5. Mark atomic regions before splitting, not as separators. Tables, fenced code blocks, numbered procedures. A separator list cannot express “never cut inside this,” so this has to be a pre-pass that extracts those regions as units, exactly as it is for tables and code. This is the single most valuable addition and it is not a separator at all.

6. Decide whether the separator is kept. A heading separator must be kept and attached to the following piece, or every chunk loses the heading that introduces it. A sentence separator should keep its full stop. A paragraph break can be discarded. Splitters differ in what they do by default and it is worth checking rather than assuming, because losing headings this way is invisible in the configuration and obvious in the chunks.

7. Do not include separators that are not boundaries. A comma or a semicolon in the list means the splitter will cut inside a clause when it has to. Sometimes that is the least-bad option; more often it produces fragments and you would rather exceed the size slightly. Leave them out and see whether anything actually breaks.

What a separator list cannot do

It does not know what the separator means. A blank line before a heading and a blank line between two paragraphs of the same subsection are the same string. Putting the heading pattern first mitigates this and does not remove it.

It does not propagate context. No heading path, no section title, no parent reference. That has to be added around the splitter, and it remains the highest-value thing you can attach to a chunk.

It is still driven by the size target. Separators choose where near the target to cut, not whether to cut. A document with one section slightly over the limit will be divided, at the best available boundary, whether or not dividing it makes sense.

It cannot express hierarchy. “Split at level 2, and if a level-2 section is too large, split it into its level-3 children, and carry the parent heading down” is a tree walk. A flat ordered list approximates it and cannot represent it, which is the boundary between tuning a recursive splitter and writing a structural one.

How to tell if it helped

Change one entry at a time and diff the chunk output over a fixed set of fixture documents. Separator lists are order-dependent, so two changes at once produce a result you cannot attribute.

Two specific things to count before and after. Chunks beginning with a lowercase letter, which measures how often you are cutting mid-sentence. And chunks containing a table-row pattern without a header separator, which measures whether \n is still reaching inside your tables. Both are one query each and both should fall when the list is right.

Then look at where the splitter is actually landing. Log which separator produced each chunk — most of the work is a one-line addition — and count the distribution. If most of your chunks were cut by the last entry in the list, your format-specific separators are matching nothing, and the list you carefully ordered is not being used at all.