Splitting text with no structure left to find

Every argument for splitting on structure assumes there is structure to find. Then a corpus arrives where there isn’t: scanned inspection reports, plain-text exports from a system retired in 2009, minutes typed into a fixed-width terminal. No markup, no heading levels, nothing to parse.

The chunks you get are slabs. They begin mid-word, they contain a page number and a running footer in the middle of a sentence, and every one of them looks the same length because length is the only thing deciding where they end.

What “no structure” actually looks like

Hard-wrapped lines. The text was wrapped at some column width, so there is a newline every seventy or eighty characters regardless of meaning. The \n separator now fires mid-sentence, and \n\n either never fires at all or — if the source was double-spaced — fires between every pair of wrapped lines.

Words broken across lines. inspec- at the end of one line and tion at the start of the next. Both halves are now tokens that mean nothing, and any chunk boundary near them produces a chunk starting with tion.

Running headers and footers inline. The extractor read the page furniture as body text, so every forty-something lines the document says CONFIDENTIAL — Rev C and Page 14 of 62 in the middle of a paragraph. Deciding what those are is separate from why the page break itself is not a chunk boundary.

Headings that are only typography. The section title was bold and centred. In plain text it is an ALL-CAPS line, or a short line surrounded by blank lines, or a line beginning 4.2. It is still a heading; it just isn’t marked as one.

Interleaved columns. A two-column page read left to right, line by line, so every line is half of one column and half of the other. Nothing at the splitting layer can repair this.

Wrong words. In OCR output some words are simply not the words in the document. 1 for l, rn for m, a table of figures read as prose. Whether to OCR at all, and with what, is a decision made before this stage; here you inherit the result.

The split, shown

An extracted page cut at a character target, then after repair.

=== CUT AT A CHARACTER TARGET, TEXT AS EXTRACTED ===
--- chunk 41 ---
ties. The flange assembly showed no measurable defor-
CONFIDENTIAL — REV C
Page 14 of 62
mation after the third cycle. Surface temperature at
the probe remained within the stated range for the dura
--- chunk 42 ---
tion of the test.
4.3 DEFERRED ITEMS
Two items were deferred to the next inspection window

Chunk 41 starts on ties., contains a footer and a header in the middle of its only sentence, and ends on dura. Chunk 42 opens with the rest of that word and then swallows a heading that belongs to whatever follows it.

=== AFTER UNWRAPPING, FURNITURE REMOVED, OUTLINE INFERRED ===
--- chunk 41  [Rev C · §4.2 Findings · p.14] ---
The flange assembly showed no measurable deformation
after the third cycle. Surface temperature at the probe
remained within the stated range for the duration of the
test.
--- chunk 42  [Rev C · §4.3 Deferred items · p.14] ---
Two items were deferred to the next inspection window.

Same source, same extractor. The difference is entirely in what happened before the splitter ran.

The procedure

1. De-hyphenate first. A line ending in - where the next line starts with a lowercase letter is a broken word: join it. Keep a small exception list for genuine trailing hyphens, and accept that you will occasionally join a real compound.

2. Unwrap the lines. The rule that works on fixed-width text: a line break is a paragraph break only if the line before it is noticeably shorter than the fill width, or ends in terminal punctuation, or the next line is indented. Otherwise it is a wrap and the lines join with a space. Measure the fill width from the corpus rather than assuming eighty.

3. Find the furniture by repetition, not by position. Collect every line in the document, count them, and look at the lines appearing many times at roughly regular intervals. That is your running header and footer. Strip them from the body and keep the page number as metadata.

4. Infer the outline from typographic signals. In descending reliability: a line starting with a section number pattern (4.2, A.1, IV.); a short line in ALL CAPS with no terminal punctuation; a short line surrounded by blank lines; a line indented differently from the body. Score candidates rather than trusting one signal.

5. Print the inferred outline and read it. Before splitting anything. Thirty seconds of reading tells you whether your heading detector found a document or found noise, and no downstream measurement will tell you as clearly.

6. Split on the inferred outline where it exists, sentence runs where it doesn’t. Where you found an outline, use it. Where you found nothing, assemble chunks from whole sentences to a size target — which requires that sentence boundaries actually be right, because they are now the only boundary you have.

7. Prepend a document-level context line to every chunk. With no section titles, the document’s own identity — title, revision, date, page — is the entire available context, and without it a slab of inspection prose is unattributable.

8. Mark the corpus low-confidence. Record on each chunk that its boundaries were inferred. When something retrieves badly later you will want to know whether you are debugging a splitter or an extractor.

What it costs and what it still breaks

The repair pass is real code with real judgement in it, and unlike a splitter configuration it is lossy — every step above is a heuristic that will be wrong somewhere in a large corpus.

Interleaved columns. Unrepairable here. Two-column output has to be re-extracted with layout awareness; if you cannot, this corpus is not chunkable and pretending otherwise produces confident nonsense.

Tables. A space-aligned table survives unwrapping badly — the unwrapper sees short lines and treats each row as a paragraph, or joins the rows into a single line of interleaved numbers. Detect runs of lines with aligned whitespace and exempt them from unwrapping, then treat the run as one atomic block.

ALL-CAPS false positives. Acronym-dense text, warning blocks and correspondence headers all look like headings. This heuristic is the one that most often invents an outline that isn’t there.

Poetry, addresses, code, and forms. All legitimately short-lined. Unwrapping destroys them, and they need to be routed away from this pipeline entirely — which is a routing decision made per document, not per corpus.

How to tell if it worked

Count chunks whose first character is lowercase or whose last word is not a word. Both are direct evidence of unrepaired wrapping, and on a repaired corpus the count should be small and explainable.

Then grep your chunk bodies for the running header string. Any hit is furniture that survived step 3, and one surviving footer is usually thousands of chunks’ worth.

Last, read the inferred outlines for ten documents you know something about. If the outline is plausible for all ten, the heuristics are holding. If it is plausible for six, you do not have a chunking result yet — you have a document class that needs a different extractor, which is a more useful conclusion than a tuned chunk size would have been.