Deduplicating chunks before you index them
Somebody asks how to shut down the unit safely, and everything that comes back says the same sentence. It is the mandated warning block, and it appears verbatim in every one of your forty equipment manuals. The index contains forty copies of it and one copy of each actual procedure.
Two nearby problems are not this one. The same document arriving in your pipeline twice is an identity problem, solved upstream by giving documents stable IDs. Deciding which of several similar results to show is a query-time problem. This post is about chunks that are legitimately duplicated because genuinely different documents genuinely contain the same text.
Where duplicate chunks come from
Mandated text. Safety warnings, legal disclaimers, licence grants, privacy paragraphs, “this document is uncontrolled when printed.” Written once by someone in compliance, pasted into everything.
Versions of the same document. The v3 and v4 manuals are ninety per cent identical. Both are in the corpus because nobody deleted v3, and most of their chunks are byte-identical.
Templated sections. Every product page has a “Compatibility” paragraph generated from the same sentence stem. Every meeting note starts with the same agenda skeleton.
Quotation. A policy summary quotes three paragraphs of the policy. Now both documents contain them, and both are correct to.
Your own overlap. If you split with overlap, adjacent chunks share text by design. Those are not duplicates and must be excluded from any dedup pass before it runs, or it will collapse your corpus into every third chunk. Worth being clear about what overlap is buying before you interfere with it.
Duplicated markup. Responsive templates emit the same block twice, once hidden. That is a markup-level duplicate and it should be removed during extraction rather than deduped afterwards.
Why it costs you, stated carefully
The duplicated chunk is not wrong. Nothing about it is a defect in isolation. The damage is distributional: your index’s variety is lower than its size suggests. A corpus of ten thousand chunks in which four hundred are copies of nine unique boilerplate blocks has nine thousand six hundred distinct things in it, and the nine are the ones with the best chance of matching a vague query, because compliance language is written in the vocabulary people use when they don’t know what they’re looking for.
The second cost is worse and quieter. When two copies differ in one word, you have a contradiction in the index and no mechanism prefers either. The v3 chunk says the cooling period is thirty minutes; the v4 chunk says fifteen. Both are retrievable, neither says which manual it came from, and the answer is a coin flip.
The split, shown
The same clause reaching the index from two documents.
=== AS INDEXED, NO DEDUP ===
--- chunk 812 (manual-8842-v3.pdf) ---
Allow a cooling period of thirty minutes before removing
the housing. Verify with the surface probe.
--- chunk 1904 (manual-8842-v4.pdf) ---
Allow a cooling period of fifteen minutes before removing
the housing. Verify with the surface probe.
--- chunk 2231 (manual-9931-v4.pdf) ---
Allow a cooling period of fifteen minutes before removing
the housing. Verify with the surface probe.812 and 1904 are a version conflict wearing a duplicate’s clothes. 1904 and 2231 are a true
duplicate across two products.
=== AFTER DEDUP, PROVENANCE KEPT ===
--- chunk A (exact dup collapsed) ---
[applies to: 8842 v4, 9931 v4 · superseded text exists for 8842 v3]
Allow a cooling period of fifteen minutes before removing
the housing. Verify with the surface probe.
--- chunk B (retained, flagged) ---
[applies to: 8842 v3 · SUPERSEDED — see 8842 v4]
Allow a cooling period of thirty minutes before removing
the housing. Verify with the surface probe.One chunk where the text was identical, two where it wasn’t, and each one now says which documents it belongs to — metadata doing the work that collapsing would otherwise destroy.
The procedure
1. Normalise, then hash. Lowercase, collapse whitespace, strip punctuation-only differences, hash the result. Exact duplicates fall out immediately and cost nothing to find. Do this on the chunk after splitting, not on the document.
2. Exclude the overlap-neighbour pairs. Two chunks adjacent in the same document are expected to share text. Filter any candidate pair that comes from the same document at adjacent positions before you look at the rest.
3. Set a length floor. Short chunks are legitimately identical — a heading, a single-cell answer,
Not applicable. Below some length, duplication carries no information and collapsing it just breaks
provenance. Choose the floor by reading your shortest duplicates, not by picking a number.
4. Find near-duplicates with a cheap fingerprint. Overlapping word-shingles hashed down to a compact signature will surface pairs that differ by a word or a date without comparing every chunk to every other chunk. You are looking for candidates to inspect, not making a final decision.
5. Sort the candidates into three bins. Byte-identical across different documents: collapse to one chunk carrying a list of sources. Near-identical with a material difference — a number, a prohibition, a date: keep both and mark the superseded one. Near-identical with an immaterial difference — a trailing period, a product name substitution: collapse and note the variation.
6. Never collapse a material difference. This is the rule the whole procedure exists for. A version conflict resolved by “these are basically the same chunk” silently deletes the current answer half the time.
7. Keep the collapse reversible. Store the source list on the surviving chunk. Anything that later has to cite a location, apply a per-document permission, or delete one source document needs to know every document the chunk came from.
8. Re-run it on every ingest. Duplicates arrive over time. Document forty-one shows up next month carrying the same warning block, and a one-off cleanup pass will not see it.
What it costs and what it still breaks
Permissions. A collapsed chunk inherited from two documents is visible to the union of their audiences unless you are careful. If any document in your corpus is access-controlled, treat collapsing across documents as unsafe by default and keep one copy per permission scope.
Boilerplate that isn’t. The same paragraph can mean different things in two documents. “Contact your account manager” in the enterprise contract and in the free-tier terms is identical text with a different referent. Collapsing it produces a chunk that is true of nobody in particular.
Thresholds don’t travel. How similar is “the same” depends entirely on your corpus. A legal corpus is near-duplicate by nature; a set of engineering notes is not. Any similarity cut-off you adopt from somewhere else will be wrong here.
Partial containment. A chunk that is a subset of another — the summary quoting three of the policy’s paragraphs — is neither a duplicate nor distinct. Fingerprints handle it poorly, and the honest answer is usually to keep both and let the quoting document’s chunk carry a marker saying it is quoting.
How to tell if it worked
Count chunks by hash and read the top twenty by frequency. This takes minutes and it is the single most informative thing you can do to an index you have never inspected: the most-duplicated texts in any real corpus are always recognisable, and they are always either boilerplate you should collapse or overlap you should have excluded. It pairs naturally with reading fifty chunks — same posture, different sample.
Then check the near-duplicate bin by hand, at least once. Print every pair whose difference is a number or a negation. That list is your version-conflict inventory, and it exists whether or not you dedup — running the pass is just how you find out about it.