Chunking an HTML page

Someone asks what the refund window is for annual plans. The top chunk back is a heading and nothing else: Refund eligibility, thirty characters, no body. The answer is in the next chunk, which begins mid-sentence and never says what it is about.

The splitter was walking the DOM. That sounds like structural splitting and it isn’t, because HTML’s tree does not describe the document — it describes the layout.

Why the DOM tree is the wrong tree

A heading is a sibling of its content, not a parent. This is the one that surprises people. In almost all real HTML, <h2> and the paragraphs it introduces sit at the same level inside the same container. There is no <section> wrapping them. So a splitter that emits one chunk per DOM node, or per subtree at some depth, emits the heading as its own chunk and the paragraphs as their own chunks, and nothing connects them.

Nesting depth is arbitrary. The same paragraph might be six <div>s deep on one page and two on another, because one page went through a layout framework and the other didn’t. Depth is not hierarchy. Splitting at “depth 3” cuts at whatever the templating happened to produce.

Heading levels are chosen for size. A designer picks <h4> because it renders smaller. Pages routinely go <h1> → <h3> → <h2>, or use <h1> three times, or mark the section title as a <div class="section-title"> with no heading tag at all. Any code that builds a hierarchy by trusting the numbers will build a wrong one.

Some blocks must not be entered. <table>, <ul>, <pre> and <figure> are single units whose children are meaningless alone. A generic tree walk descends into them happily and produces chunks that are three <td>s.

Stripping navigation, footers and cookie banners is a separate, earlier problem — assume you have already extracted the content region before any of this. What follows is about where to cut what’s left.

The split, shown

The same fragment cut by DOM subtree, then by heading run.

=== ONE CHUNK PER BLOCK-LEVEL SUBTREE ===
--- chunk 1 ---
Refund eligibility
--- chunk 2 ---
Annual plans may be cancelled at any time. A refund is
issued for the unused whole months remaining.
--- chunk 3 ---
Plan            Window
Monthly         14 days
Annual          pro rata
--- chunk 4 ---
Requests are processed by the billing team.

Chunk 1 is a heading with no document. Chunk 3 is a table with no idea what it tabulates. Chunk 2 never uses the word “refund eligibility”, so the query’s strongest term appears only in the chunk that has nothing else in it.

=== ONE CHUNK PER HEADING RUN, ATOMIC BLOCKS KEPT WHOLE ===
--- chunk 1 ---
[Billing > Refunds > Refund eligibility]

Annual plans may be cancelled at any time. A refund is
issued for the unused whole months remaining.

| Plan    | Window   |
| ------- | -------- |
| Monthly | 14 days  |
| Annual  | pro rata |

Requests are processed by the billing team.

One chunk, one idea, the table intact and the heading path carried as text, doing the same job here it does in any structured format.

The procedure

1. Flatten to a linear stream of blocks. Discard the container <div>s entirely. What you want is an ordered list: heading, paragraph, list, table, code block, figure, paragraph. Depth information is noise; order is signal.

2. Build the hierarchy from the sequence of headings, not their numbers. Read the headings in document order and treat each one as opening a section that closes at the next heading of the same or larger size. Where the numbering is inconsistent, fall back to “the previous heading is my parent” rather than trusting the level. You are reconstructing an outline the markup only implies.

3. Cut at heading runs. A chunk is one heading plus everything until the next heading. Headings never become chunks of their own and never end up separated from what they introduce.

4. Treat <table>, <pre>, <ul>, <ol>, <blockquote> and <figure> as atomic. Never descend into them. If one of them alone exceeds your maximum, that is a table-splitting or list-splitting problem, handled on its own terms rather than by the HTML walk.

5. Convert the atomic blocks to a text form before embedding. Markdown-style tables and lists read far better to an embedding model than tag soup or than cells run together with no delimiter.

6. Expand what is hidden. <details> bodies, accordion panels and inactive tab panels are content that a reader can reach. If your extractor drops them you will lose the FAQ answers, which are the part people ask about.

7. Drop the duplicate copies. Responsive templates ship the same block twice — a desktop version and a mobile version, one of them hidden or aria-hidden. Keep one. This is a common source of chunks that duplicate each other.

What it costs and what it still breaks

You now need an HTML parser and a small amount of judgement code, where before you needed a string split. That is the cost, and it is worth it exactly when your corpus is HTML you did not author.

Client-rendered pages. The fetched HTML is an empty <div id="root">. Nothing here helps; the page has to be rendered before it can be split, and that is an extraction decision made upstream.

Pages with no headings. Long single-heading articles, marketing pages, anything built as a sequence of styled <div>s. The heading run degenerates into one enormous run, and you are back to splitting text with no structure left to find.

Pages that are one giant table. A pricing matrix or a spec sheet. The heading run is a single atomic block far over your maximum.

Anchor-linked reference pages. A long API page where every entry has an id and inbound links point at it. Heading runs and anchors usually agree, but where they don’t, the anchor is the thing people cite and it should decide the boundary.

How to tell if it worked

Sample chunks and read the heading paths first, not the bodies. If a path reads Billing > Refunds > Refund eligibility, the outline reconstruction worked. If half of them read Home > Documentation and nothing more, step 2 is trusting heading numbers that aren’t there.

Then look for two specific defects. Count chunks under about a line of text — those are orphan headings, and if you have any, step 3 is not being applied. And count chunks where most of the text is link anchors: a chunk that is mostly links is either surviving page furniture or a table of contents, and neither belongs in the index as content.