Routing documents to the right splitter

Every post on this site argues for handling some document type differently, and a corpus contains several of them at once. The thing that makes those arguments usable is not any individual splitter. It is a dispatcher: a stage that decides, per document, which splitter runs.

Without one you have a single global configuration, which means the strategy that suits markdown is being applied to transcripts, spreadsheets and slide decks, and at least one of those is being destroyed.

Why one splitter cannot be right

The units of meaning are different per format, and there is no configuration that expresses all of them. A table’s unit is the table or the row with its header. Source code’s is the function with its signature. A transcript’s is a topic segment with its participants. A contract’s is a numbered clause. A help article’s is the whole article.

Those are not points on a size scale — they are different definitions of a boundary. A parameter cannot interpolate between them, so a global setting is not a compromise between the formats. It is the correct answer for one of them and wrong for the rest.

The corollary is that a mixed corpus with one splitter has at least one document class that retrieves badly, and you can predict which: whichever is least like prose.

The shape of a dispatcher

Five parts, and the order matters.

1. Classify the document. Not by file extension alone. Extension is a hint that is often wrong — a .txt that is actually a transcript, a .md that is a data dump, a .pdf that is a scanned form. Combine the extension, the source system, and a content sniff: does it have markdown headings, does it have pipe-delimited rows throughout, does it have speaker labels at line starts, does it parse as source code.

2. Route to a handler. A small registry mapping document class to splitter. Each handler owns its own boundary rules and its own size behaviour.

3. Extract atomic units first, within the handler. Tables, code fences, figures, procedures. Pull them out as units before any size logic runs, and chunk the prose around the holes they leave. This is the step people skip, and it is why mixed documents fail even when the routing is right.

4. Apply a common post-pass. Whatever the handler did, every chunk then gets the same treatment: heading path attached, metadata fields populated, size verified against the embedding model’s ceiling, empty chunks dropped. Handlers should not each reimplement this.

5. Record what happened. The class, the handler, and the reason on every chunk. Without this you cannot debug a bad chunk, because you do not know which code path produced it.

The split, shown

One document containing prose, a table and a code block, handled globally and then dispatched internally.

=== ONE SPLITTER, SIZE-DRIVEN ===
--- chunk 1 ---
## Retry configuration

Retries are controlled by three settings, shown below.

| Setting | Default | Effect |
| --- | --- | --- |
| max_attempts | 3 | total tries including the first |
--- chunk 2 ---
| backoff_factor | 2 | multiplier on the delay |
| retry_on | 429,5xx | statuses that trigger a retry |

Set them at client construction:

    client = Client(retry=RetryPolicy(
--- chunk 3 ---
        max_attempts=5, backoff_factor=2,
        retry_on=[429, 503]))

The table is split from its header, the code is split mid-expression, and chunk 2 is half a table plus half a code block — one chunk containing two different kinds of damage.

=== UNITS EXTRACTED, THEN PROSE CHUNKED ===
--- chunk 1  {content_type: prose} ---
[Retry configuration]
Retries are controlled by three settings, shown below.
Set them at client construction.

--- chunk 2  {content_type: table, part_of: "Retry configuration"} ---
[Retry configuration — settings table]
| Setting | Default | Effect |
| --- | --- | --- |
| max_attempts | 3 | total tries including the first |
| backoff_factor | 2 | multiplier on the delay |
| retry_on | 429,5xx | statuses that trigger a retry |

--- chunk 3  {content_type: code, lang: python,
              part_of: "Retry configuration"} ---
[Retry configuration — example]
client = Client(retry=RetryPolicy(
    max_attempts=5, backoff_factor=2,
    retry_on=[429, 503]))

Three chunks, three content types, each whole, each labelled with the section it belongs to so they can be reassembled or retrieved independently. The handler for this document class is a markdown handler that knows about tables and code; it did not need to know anything about transcripts.

Choosing a default that is honest

Most corpora have a long tail of formats that will never get a handler. The default matters, and there are two defensible choices.

A careful fixed-size splitter — sentence-aligned, heading-prefixed, never cutting across documents. This is what fixed-size splitting is for: it works on anything and fails predictably.

Exclusion. Some document classes should not be in the index. Minified assets, generated code, binary extracts, archived duplicates, slide decks that duplicate a document you already have. A dispatcher makes this expressible as a routing decision rather than a hack, and a corpus is allowed to say no.

What is not defensible is a default that silently applies a format-specific splitter to a format it does not understand — a markdown handler on a plain-text export, for instance, will find headings that are not there.

What a dispatcher costs

Per-format code to maintain. Every handler is a thing that can break independently, and formats change.

Classification errors, which are silent. A transcript routed to the prose handler produces chunks that look fine and cut mid-turn. The failure is in the routing, and you will diagnose the splitter.

Inconsistency across the corpus. Chunks from different handlers have different sizes and different character, so the length distribution becomes a mixture and has to be read per document type.

Combinatorial testing. Every handler needs its own fixture documents, and the post-pass needs testing against all of them.

Over-engineering. A corpus that is 98% one format needs one splitter and an exclusion rule, not a registry. Build the dispatcher when you have the second document class that genuinely needs different treatment, not before.

How to tell if it is working

Start with the routing table, not the chunks. Count documents by assigned class and by handler, and read the counts. The number you are looking for is how many documents fell to the default: if it is large, the dispatcher is decorative. If a class you know exists has a count of zero, classification is not detecting it.

Then sample chunks per handler rather than per corpus. Ten chunks from each handler, read properly. This is the version of chunk review that finds routing bugs, because a chunk that looks wrong for its declared content type tells you the document was misclassified rather than mis-split.

Finally, keep a fixed set of fixture documents — one per class, including two or three deliberately awkward mixed ones — and diff the chunk output whenever a handler changes. Splitters are the kind of code where a change intended for one format quietly alters another, and the diff is the only thing that notices.