Tagged “boundaries”
-
Chunking an HTML page
HTML nesting is presentational, not semantic. Why splitting on the DOM tree produces orphan headings, and how to split on the heading run instead.
-
Where a sentence actually ends
Splitting on a period produces chunks that start mid-thought. The abbreviations, decimals and quotes that break sentence segmentation, and what to do.
-
Choosing separators for a recursive splitter
The separator list is the only real knob on a recursive splitter. Order it wrong and it cuts inside tables; extend it and it respects your format.
-
Splitting text with no structure left to find
Scanned reports and plain-text exports arrive with no headings, hard-wrapped lines and running footers. Repair the text first, then infer the outline.
-
Chunking tables, code and transcripts
Three document types that break general splitters, and the handling each needs: repeated headers, syntax-aware cuts, and topic-grouped speaker turns.
-
Split on structure, not on characters
A character splitter cuts where the counter runs out. Documents already have boundaries — headings, clauses, functions. Use those; count as a fallback.
-
What to do with a table of contents
A contents page is the most term-dense chunk in your index and contains no statements. How to detect signpost text, what to do with it, and what to keep.
-
Splitting text that has no spaces
Chinese, Japanese and Thai have no word delimiters, so character counts and sentence regexes both mislead. What to segment on when whitespace is absent.
-
Merging chunks that came out too small
Structural splitting produces one-line chunks. Merging fixes that, and merging across the wrong boundary makes a chunk about two unrelated things.
-
Chunking contracts: the clause is the unit
Legal documents number their own chunks. Use the numbering, carry the definitions, and accept that cross-references are not solvable at the split.
-
Page boundaries are not chunk boundaries
Pagination is a printing artefact. Chunking per page cuts mid-sentence by design, and footnotes land in a chunk that never mentions their subject.
-
Splitting a list without breaking the procedure
Step 7 retrieved alone is a sentence beginning with Then. Numbered procedures, heterogeneous bullets and definition lists each want a different cut.
-
Semantic splitting: cutting where the topic turns
Embed each sentence, then cut where consecutive sentences stop resembling each other. What that finds, what it invents, and what it costs at index time.
-
Characters, words, tokens: what your splitter is counting
A 1,000-character limit is not a 1,000-token limit, and the two disagree most on the content you care about most. Pick a unit and know its error bars.
-
When counting characters is the right answer
Fixed-size splitting is the fallback everyone apologises for. Four situations where it is genuinely the correct choice, and how to make it less bad.