Where a sentence actually ends

You read a sample of chunks and several of them are two words long. One is Dr. One is approx. One begins 5 mm, measured at the flange, and the digits that number belonged to are in the fragment before it. Somewhere between the paragraph and the chunk, something decided a period was the end of a sentence.

This matters more than it looks, because sentence boundaries are the bottom rung of every boundary hierarchy. When a section is too big and a paragraph is too big, sentences are the last real unit before you are reduced to counting characters. If that rung is rotten, the whole fallback path is.

What a period is actually doing

The separator . — period, space — is the usual implementation of “split on sentences”, including as the third or fourth entry in a recursive splitter’s separator list. English writes far more periods than it writes sentence endings. The classes that break it:

Abbreviations. Dr., Inc., Fig., No., approx., et al., vs., Ave., Ltd. Every domain has its own set, and technical corpora have hundreds — units, standards, part-number prefixes, statute abbreviations.

Initials. J. R. R. Tolkien is four fragments. Author lists in a references section are the worst case in any academic corpus.

Numbers. Decimals (3.5), versions (v2.11.0), section numbers (4.2.1), money, dates written 31.07.2026, IP addresses, file extensions. A period between two digits is almost never a sentence end, which is a rule cheap enough to encode.

Numbered list markers. 1. Rotate the key. The period after 1 looks identical to the one after key, so the marker gets split off from the step it labels.

Terminators inside a quotation or a bracket. She said "it depends." Then she left. The period sits before the closing quote, so a naive rule cuts between "it depends. and " Then — leaving an unbalanced quote in both halves.

The genuine collision. Manufactured in the U.S. The tolerance is tighter than the ISO minimum. Here the period after U.S is also a sentence end. An abbreviation exception list gets this one wrong in the other direction, and no rule resolves it without looking at what follows.

No terminator at all. Headings, table cells, list items, slide bullets and captions frequently end with no punctuation. To a period-based splitter, a heading and the paragraph beneath it are one sentence.

Other scripts. 。, ！, ？, ।, ؟ are sentence terminators your regex has never heard of. An Arabic or Hindi document splits into exactly one sentence, which is the whole document.

The split, shown

One paragraph, cut on . and then on real sentence boundaries.

=== SPLIT ON PERIOD-SPACE ===
--- s1 --- Fig.
--- s2 --- 4 shows the flange assembly as supplied by Kaltec Ltd.
--- s3 --- The bore is 12.
--- s4 --- 5 mm and the wall thickness is 2.
--- s5 --- 0 mm, per ISO 4032.
--- s6 --- Torque to 18 Nm approx.
--- s7 --- ; do not exceed 22 Nm.

Seven fragments, five of which are unreadable, and any chunk assembled from a run of them starts mid-number. s3 asserts that the bore is twelve.

=== SEGMENTED WITH NUMERIC AND ABBREVIATION GUARDS ===
--- s1 --- Fig. 4 shows the flange assembly as supplied by
           Kaltec Ltd.
--- s2 --- The bore is 12.5 mm and the wall thickness is
           2.0 mm, per ISO 4032.
--- s3 --- Torque to 18 Nm approx.; do not exceed 22 Nm.

Three sentences, each of which survives being retrieved alone. Note that s1 and s3 both end at an abbreviation’s period, which is only correct because the guard consulted what came next.

The procedure

1. Use a segmenter, not a separator. Sentence segmentation is a solved-enough problem with existing implementations for most languages. Writing your own from a regex is the mistake; the useful work is configuring one.

2. Encode the cheap numeric rules first. A period with a digit on both sides is never a boundary. A period followed by a lowercase letter is almost never one. These two catch most of the damage before any list of exceptions is involved.

3. Build the abbreviation list from your own corpus. Take every token that ends in a period, count them, and read the top few hundred by frequency. Your corpus’s abbreviations are not the generic list’s abbreviations, and this is twenty minutes of work that pays for itself immediately.

4. Never cut inside an unbalanced quote or bracket. Track depth as you scan. If a candidate boundary would leave either side unbalanced, move it past the closing mark.

5. Treat a line with no terminal punctuation as its own unit. Headings, cells and bullets are sentence-equivalent, not sentence-continuing. Segment within a block, never across a line break that follows an unpunctuated line.

6. Detect the script before choosing the rules. One splitter configuration cannot serve a mixed corpus. Route by language, and note that for text with no spaces between words the whole assumption set changes again.

7. Keep sentences as the fallback rung, not the chunk. A one-sentence chunk is rarely what you want. Segment to sentences so you can assemble chunks that end cleanly, and so that overlap can be measured in whole sentences.

What it still breaks

Text with no punctuation at all. Raw speech-to-text output arrives as an unpunctuated stream. No segmenter helps; either restore punctuation as a preprocessing step or split on speaker turns and pauses instead.

Legal drafting. A clause can run four hundred words with semicolons and lettered sub-paragraphs and one period at the very end. Segmentation returns one sentence, correctly, and it is still too big.

Line-wrapped and hyphenated text. Where the source has hard wraps mid-sentence, or words broken across lines, sentence detection fails upstream of itself — the input is not sentences yet. That is a different problem.

Bulleted fragments that continue a stem. “You may not: · resell the licence · sublicense it.” Each bullet is a sentence fragment whose subject is in the stem, and segmenting them apart loses the prohibition.

How to tell if it worked

Histogram your sentence lengths. A spike below about five characters is abbreviation and initial damage; there is no legitimate three-character sentence in a technical document. Read the shortest fifty and you will identify your missing exceptions in a single pass.

Then check the chunk boundaries the sentences fed. Count chunks whose first character is a digit, a lowercase letter, or a closing bracket or quote. Each one is a chunk that begins mid-thought, and the count should be near zero on a corpus of edited prose. If it isn’t, you have found the rung that needs fixing before anything above it will hold.