Tagged “strategy”
-
Deduplicating chunks before you index them
One warning in forty manuals becomes forty chunks. Which duplicate chunks to collapse, which to keep, and why a version difference is not a duplicate.
-
Metadata is half the chunk
A chunk with no source, date or heading path can't be filtered or cited. What to attach, what to embed with the text, and what to keep out of the vector.
-
Overlap, and what it actually buys
Overlap insures against cutting through an idea. It also inflates the index and fills top-k with near-copies. When it earns its cost and when it doesn't.
-
Split on structure, not on characters
A character splitter cuts where the counter runs out. Documents already have boundaries — headings, clauses, functions. Use those; count as a fallback.
-
Why chunk size is a tradeoff, not a setting
Small chunks retrieve precisely and answer incompletely. Large chunks do the reverse. What each end actually breaks, and how to find your corpus's middle.
-
What to do with a table of contents
A contents page is the most term-dense chunk in your index and contains no statements. How to detect signpost text, what to do with it, and what to keep.
-
Documents you should not chunk
Some documents are one idea, and splitting them destroys the only thing they said. How to recognise them, and how a mixed corpus handles both kinds.
-
Routing documents to the right splitter
One splitter for a mixed corpus guarantees some document class retrieves badly. A dispatcher, an honest default, and a way to see what went where.
-
When the document is already a chunk
FAQ entries, tickets, product records and short help articles arrive pre-split. Running a splitter over them is a downgrade — here is what to do instead.
-
Rewriting a chunk so it can stand alone
Chunks are full of pronouns, acronyms and see-above references pointing at text you cut away. What to resolve at index time, and what resolving it breaks.
-
Small to big: match on the fragment, return the section
Parent/child chunking splits the matching unit from the reading unit. It dissolves most of the size trade-off and adds a store to keep in sync.
-
Semantic splitting: cutting where the topic turns
Embed each sentence, then cut where consecutive sentences stop resembling each other. What that finds, what it invents, and what it costs at index time.
-
When counting characters is the right answer
Fixed-size splitting is the fallback everyone apologises for. Four situations where it is genuinely the correct choice, and how to make it less bad.