Blog
-
Chunking an HTML page
HTML nesting is presentational, not semantic. Why splitting on the DOM tree produces orphan headings, and how to split on the heading run instead.
-
Where a sentence actually ends
Splitting on a period produces chunks that start mid-thought. The abbreviations, decimals and quotes that break sentence segmentation, and what to do.
-
Deduplicating chunks before you index them
One warning in forty manuals becomes forty chunks. Which duplicate chunks to collapse, which to keep, and why a version difference is not a duplicate.
-
Choosing separators for a recursive splitter
The separator list is the only real knob on a recursive splitter. Order it wrong and it cuts inside tables; extend it and it respects your format.
-
Splitting text with no structure left to find
Scanned reports and plain-text exports arrive with no headings, hard-wrapped lines and running footers. Repair the text first, then infer the outline.
-
Chunking tables, code and transcripts
Three document types that break general splitters, and the handling each needs: repeated headers, syntax-aware cuts, and topic-grouped speaker turns.
-
Metadata is half the chunk
A chunk with no source, date or heading path can't be filtered or cited. What to attach, what to embed with the text, and what to keep out of the vector.
-
Overlap, and what it actually buys
Overlap insures against cutting through an idea. It also inflates the index and fills top-k with near-copies. When it earns its cost and when it doesn't.
-
Split on structure, not on characters
A character splitter cuts where the counter runs out. Documents already have boundaries — headings, clauses, functions. Use those; count as a fallback.
-
Why chunk size is a tradeoff, not a setting
Small chunks retrieve precisely and answer incompletely. Large chunks do the reverse. What each end actually breaks, and how to find your corpus's middle.
-
What to do with a table of contents
A contents page is the most term-dense chunk in your index and contains no statements. How to detect signpost text, what to do with it, and what to keep.
-
Documents you should not chunk
Some documents are one idea, and splitting them destroys the only thing they said. How to recognise them, and how a mixed corpus handles both kinds.
-
Chunking a spreadsheet
When the whole document is a table there is no prose to fall back on. Sheets, stacked headers, wide rows, and knowing when a retriever is the wrong tool.
-
Figures, captions, and the text that refers to them
An image is not indexable and its caption is. Keeping figure, caption and the paragraph saying see Figure 3 together, or at least connected.
-
Routing documents to the right splitter
One splitter for a mixed corpus guarantees some document class retrieves badly. A dispatcher, an honest default, and a way to see what went where.
-
The ceiling your embedding model puts on chunk size
Every embedding model truncates past some input length, silently. That limit is a hard cap on chunk size, and it is not the cap you should aim for.
-
Splitting text that has no spaces
Chinese, Japanese and Thai have no word delimiters, so character counts and sentence regexes both mislead. What to segment on when whitespace is absent.
-
Merging chunks that came out too small
Structural splitting produces one-line chunks. Merging fixes that, and merging across the wrong boundary makes a chunk about two unrelated things.
-
What your chunk-length distribution tells you
Histogram your chunk lengths. A spike at the maximum, a pile of near-empty chunks and a long right tail each name a specific bug in your splitter.
-
Read fifty chunks
Before building anything measurable, sample your index and read it. Most chunking bugs are visible to a person in twenty minutes and invisible to a metric.
-
When the document is already a chunk
FAQ entries, tickets, product records and short help articles arrive pre-split. Running a splitter over them is a downgrade — here is what to do instead.
-
Chunking contracts: the clause is the unit
Legal documents number their own chunks. Use the numbering, carry the definitions, and accept that cross-references are not solvable at the split.
-
Page boundaries are not chunk boundaries
Pagination is a printing artefact. Chunking per page cuts mid-sentence by design, and footnotes land in a chunk that never mentions their subject.
-
Chunking slide decks
A slide is six words and a chart, and the meaning was in the room. What to do with decks that must be indexed anyway, starting with the speaker notes.
-
Splitting a list without breaking the procedure
Step 7 retrieved alone is a sentence beginning with Then. Numbered procedures, heterogeneous bullets and definition lists each want a different cut.
-
Rewriting a chunk so it can stand alone
Chunks are full of pronouns, acronyms and see-above references pointing at text you cut away. What to resolve at index time, and what resolving it breaks.
-
Small to big: match on the fragment, return the section
Parent/child chunking splits the matching unit from the reading unit. It dissolves most of the size trade-off and adds a store to keep in sync.
-
Semantic splitting: cutting where the topic turns
Embed each sentence, then cut where consecutive sentences stop resembling each other. What that finds, what it invents, and what it costs at index time.
-
Characters, words, tokens: what your splitter is counting
A 1,000-character limit is not a 1,000-token limit, and the two disagree most on the content you care about most. Pick a unit and know its error bars.
-
When counting characters is the right answer
Fixed-size splitting is the fallback everyone apologises for. Four situations where it is genuinely the correct choice, and how to make it less bad.