Semantic splitting: cutting where the topic turns

A document has no headings and no useful punctuation rhythm, but it plainly changes subject four times. A character splitter cannot see that. A structural splitter has nothing to parse. The topic shifts are real and every mechanical boundary misses them.

Semantic splitting attacks this directly: embed the document sentence by sentence, measure how similar each sentence is to the one before it, and cut where that similarity drops. The boundaries come from the meaning of the text rather than from its formatting or its length.

The mechanism

The procedure, in the form nearly every implementation takes:

1. Split the document into sentences. These are candidate boundaries, not chunks. Every eventual cut will be at one of them, so the sentence splitter’s quality sets a floor on everything after.
2. Embed each sentence. Optionally embed a small sliding group of sentences instead, so that a single short sentence does not produce a wild vector.
3. Compute the similarity between each adjacent pair. You now have one number per sentence gap: a curve along the length of the document.
4. Cut at the troughs. Where similarity between neighbours is low, the subject changed. Choose the cut points by a threshold, or by taking the lowest N percent of gaps, or by looking for local minima.
5. Enforce size limits afterwards. Merge segments that came out tiny, split segments that came out larger than your model will accept.

Step 4 is where all the design decisions hide, and step 5 is where people are surprised, because semantic splitting does not respect any size target on its own.

The split, shown

Here is a stretch of a support article with no headings, first cut at a fixed size, then cut at the similarity trough. The similarity values are illustrative — the point is the shape of the curve, not the numbers.

=== FIXED SIZE ===
--- chunk 1 ---
Password resets are handled entirely by email. The link in
the reset email expires after one hour. If the link has
expired, request a new one from the sign-in page. Sessions
are unaffected by a password change; existing sessions
--- chunk 2 ---
remain valid until they expire normally. Two-factor
enrolment is separate from password management. Enrolling a
new authenticator app revokes the previous one immediately.

=== SEMANTIC ===
[sentence-gap similarity, illustrative: high · high · high · LOW · high]

--- chunk 1 (topic: password reset) ---
Password resets are handled entirely by email. The link in
the reset email expires after one hour. If the link has
expired, request a new one from the sign-in page. Sessions
are unaffected by a password change; existing sessions
remain valid until they expire normally.
--- chunk 2 (topic: two-factor enrolment) ---
Two-factor enrolment is separate from password management.
Enrolling a new authenticator app revokes the previous one
immediately.

The fixed-size version puts the last sentence of the password-reset topic and the first two sentences of the two-factor topic in the same chunk, and splits the session-validity statement across the boundary. The semantic version cuts at the one place the document actually changes subject.

Notice the cost, too: chunk 2 is much shorter than chunk 1. Semantic boundaries do not produce even sizes, and they were never trying to.

What it is good at

Undifferentiated prose that changes subject. Its home ground. Long articles, essays, meeting minutes without headings, transcripts of a discussion that wandered — anything where the topic structure is real and unmarked.

Finding the boundary a heading would have marked, when there is no heading. In effect it recovers the structure the author did not write down.

Producing chunks that are genuinely about one thing, which is the property chunk size is a crude proxy for. A chunk defined by “the span where the subject stayed the same” is closer to what you wanted than any span defined by length.

What it invents

This is the part the technique’s fans skip.

A similarity trough is not a topic boundary. It is a place where two adjacent sentences are worded differently. Those coincide often enough for the method to work and not often enough to trust it. A single vivid example sentence in the middle of an otherwise uniform paragraph produces a trough, and you get a cut in the middle of one idea.

Smooth transitions produce no trough at all. Well-written prose deliberately bridges between topics — “which brings us to the second problem” — and a good bridge is a sentence that resembles both sides. The better the writing, the weaker the signal.

Lists and tables destroy the curve. Consecutive table rows are extremely similar to each other, so the whole table reads as one topic even if it is enormous. Consecutive bullet points in a heterogeneous list are extremely dissimilar, so a five-item list produces five cuts. Neither is what you want, and both are common. If your corpus has lists or tables, route them elsewhere before this stage.

The threshold is a parameter you have to tune, and it does not transfer. A cut-off that works on long-form articles produces confetti on terse reference material. You have swapped one corpus-dependent number for another; the difference is that this one is harder to reason about.

What it costs

An embedding call per sentence at index time. Roughly an order of magnitude more embedding work than embedding chunks, since sentences are much smaller than chunks. On a large corpus this is the dominant cost of ingestion, and it recurs every time you re-index.

A second pass to fix sizes. Segments come out uneven, so you need merging for the small ones and fallback splitting for the large ones. That fallback splitter is doing real work on your longest segments, which means part of your corpus is not semantically split at all.

Non-obvious behaviour. When a chunk boundary is wrong, “the similarity between sentence 41 and 42 was below threshold” is not an explanation anyone can act on. Compare with “the section ended here.” Debuggability is a genuine property of a splitter and this one has less of it.

Sensitivity to the sentence splitter. Abbreviations, decimal points, bullet lines without terminal punctuation. If sentence detection is wrong, every step after it inherits the error.

Where it belongs

Not as your default. As a targeted tool for the document class that defeats everything else.

The honest sequence is: structural splitting where the document has structure; careful fixed-size splitting where it does not and the prose is undifferentiated; semantic splitting where it does not and the topic shifts matter enough to pay for finding them. That last set is smaller than the technique’s popularity suggests, and it is not empty.

There is also a cheaper approximation worth trying first: split at paragraph boundaries and merge adjacent paragraphs while they stay similar. Same idea, one embedding per paragraph instead of one per sentence, and paragraph breaks are already a weak topic signal an author chose.

How to tell if it helped

Do not start with a retrieval metric. Start by printing the segment boundaries with one sentence of context on each side, for twenty documents, and reading them. A human can judge “is this where the subject changed” in a second per boundary, and this is the fastest review available for this particular strategy because the claim being made is so specific.

Then check the length distribution. If semantic splitting has produced a large population of one-sentence chunks, the threshold is too aggressive and the merge step is doing more of the work than the similarity curve is. At that point you are paying for an embedding per sentence to arrive somewhere a paragraph splitter would have reached for free.