The Chunking Problem

What to do with a table of contents

Read a sample of your chunks and eventually you hit one that is forty section titles and forty page numbers. It is the contents page of a manual. Every important term in the document appears in it, and not one thing is asserted about any of them.

Chunks like this are not broken. Nothing was mis-split. The splitter did exactly the right thing to text that should never have reached it, and the result is a decoy that stands where the section it names should be.

The class of text this is

Signpost text points at content instead of being content. In a real corpus it arrives as:

  • Contents pages, including the two-level and three-level kind with dot leaders.
  • Lists of figures and lists of tables, usually right after the contents.
  • Back-of-book indexes, which are contents pages sorted alphabetically and worse, because the terms have lost even their order.
  • Cross-reference blocks — “See also sections 4, 7 and 11”, “Related procedures”, “Referenced standards.”
  • Revision history tables — a date, a revision letter, and “Updated section 4.2” repeated eleven times.
  • Cover and separator pages — a title, a document number, This page intentionally left blank.
  • Navigation and breadcrumb lists in HTML documentation, which are the same thing generated by a template and are best removed during extraction.
  • Agenda and section-divider slides, which is why a deck’s outline slides need handling separately from its content slides.

What they share: high density of the document’s own vocabulary, and zero propositions. A chunk of signpost text cannot answer a question because it does not contain a claim.

Why it is worse than ordinary noise

Most noise in an index is inert — a footer, a legal line, something nobody’s question resembles. Signpost text is not inert. It is written in the exact terminology of the document, so it is the piece of the document most likely to look like the answer to a question phrased in that terminology, while being the piece least able to answer it. It is the most term-dense text you own and it makes no statements.

There is a second, sneakier cost. A contents page lists sections that may or may not have survived extraction. Indexed as content, it papers over that: a query about section 4.3 finds something, and the fact that section 4.3 itself never made it into the corpus goes unnoticed for months.

The split, shown

A contents page as chunked, then as used.

=== INDEXED AS CONTENT ===
--- chunk 3 ---
Contents
1  Scope ......................................  3
2  Normative references .......................  4
3  Terms and definitions ......................  5
4  Inspection procedure .......................  8
4.1  Preparation ..............................  8
4.2  Findings .................................  11
4.3  Deferred items ...........................  14
5  Reporting ..................................  17

Everything a reader would search this manual for is in chunk 3. Nothing a reader would need is.

=== NOT INDEXED; USED AS THE OUTLINE ===
[document outline, held as metadata]
  §4.2 Findings → pages 11–13
  §4.3 Deferred items → pages 14–16

--- chunk 57  [§4.3 Deferred items · p.14] ---
Two items were deferred to the next inspection window.
The housing seal was not re-tested following the third
cycle and remains outstanding.

The contents page is now the thing that gave chunk 57 its heading path and its page range. It earned its keep without occupying a slot in the index.

The procedure

1. Detect it by shape, not by the word “Contents”. The reliable signals, scored together: a high ratio of short lines to complete sentences; lines ending in a bare number; runs of dots or tabs before that number; and the strongest one — a high proportion of the lines also appear verbatim elsewhere in the same document as headings. That last test is close to definitive, because signpost text is by construction a copy of other text.

2. Position is a weak signal, not a rule. Contents pages cluster at the front and indexes at the back, which is worth using as a tiebreak. It is not worth trusting alone: many real documents have a mid-document contents block per part, and some have none at all.

3. Exclude it from the index, and say so explicitly. Not silently dropped — recorded as excluded, with its class. Someone will ask later why a page is missing.

4. Use it as the outline. This is the point. A parsed contents page gives you a validated heading list, a hierarchy, and page ranges, which is exactly what a document with no parseable structure otherwise lacks. Where you inferred an outline from typography, the contents page is the thing you check it against.

5. Lift the metadata off the cover and revision pages before discarding them. Document title, number, revision, effective date, issuing body. On many documents this is the only place a date exists, and it belongs on every chunk.

6. Keep the lookup tables — as atomic chunks. An abbreviations table, a symbols key, a glossary of defined terms: these are signpost-shaped but they do contain statements, and people query them directly. Keep each one whole, give it a heading, and never split it. A glossary cut in half is a column of terms with no definitions.

7. Resolve the cross-reference blocks, don’t index them. “See also section 7” is useful as a link between chunks and useless as retrievable text. Store it as a relation on the chunk if you can use it; drop it if you can’t.

What it costs and what it still breaks

Sometimes the signpost is the only copy. Short documents occasionally list section titles in a contents block and then use no headings in the body. Discard the contents page and the section names are gone. Check that the outline is reproduced somewhere before you drop the only place it appears.

“Where is X documented” is a real question. In a large reference manual, the index genuinely is the answer to a locating question. If your users ask locating questions, keep one navigational chunk per document deliberately, labelled as an index, rather than keeping all of them by accident.

Genuine lists get caught. A parts list, a checklist, a set of numbered steps — all short-lined, some ending in numbers. The “also appears as a heading” test is what separates them, and where it is inconclusive, treat the block as a list to be split on its own terms rather than as furniture.

Revision histories are ambiguous. “Updated section 4.2 for clarity” is signpost text, but a question about what changed between revisions has nowhere else to go. Keep the table whole as one chunk if change history matters in your domain; drop it if it doesn’t. There is no universal answer and the decision belongs to whoever knows the corpus.

How to tell if it worked

Do the reverse test, because it is the valuable one. Take the terms from a document’s contents page and check that each one also appears in a chunk that is not the contents page. Every term that appears only in the signpost is a section that failed to extract or failed to split, and you have just found a hole in the corpus that no amount of retrieval tuning would have revealed.

Then sample chunks by the ratio of lines to sentences and read the top of that ranking. Signpost text sits at the extreme of that distribution along with tables and lists, so the list is short, and one pass through it tells you both what your detector missed and what it caught wrongly.

strategyboundaries