Splitting text that has no spaces — The Chunking Problem

A pipeline built and tuned on English documentation gets a Japanese corpus added to it. Nothing errors. Retrieval on the new corpus is poor in a way that does not match any of the usual diagnoses, and the chunk lengths look fine.

Almost every default in a splitter encodes an assumption about writing systems: that spaces separate words, that a period ends a sentence, that a character is roughly a fixed amount of information. In Chinese, Japanese, Thai, Khmer and several other scripts, all three are wrong.

Which assumptions break

Spaces do not delimit words. Chinese and Japanese are written continuously. Thai and Khmer put spaces at phrase or clause level, irregularly. A whitespace tokeniser applied to a Chinese paragraph returns one enormous “word,” and any splitter whose smallest fallback separator is a space has no fallback at all.

Sentence punctuation is different characters. Chinese and Japanese use 。 for a full stop, 、 for a comma-like pause, and full-width ！ and ？. A sentence regex looking for . ! ? finds nothing in a page of Japanese, and a recursive splitter that tries . before falling back to characters skips straight to characters. Thai traditionally uses no sentence-ending punctuation at all.

A character carries much more meaning. One Han character is a morpheme, sometimes a whole word. A fixed character count therefore corresponds to far more content in Chinese than in English, so the same splitter configuration produces chunks with very different information density across languages in the same corpus.

Characters are not code points. Emoji, combining marks, and characters outside the basic multilingual plane occupy more than one code unit in some string representations. Slicing at a code-unit offset can cut a single character in half and produce invalid text. Count grapheme clusters or code points, never code units.

The split, shown

The same passage in English and in Chinese, cut at the same nominal character count, with the Chinese version then cut on its own punctuation.

=== ENGLISH, cut at N characters, boundary nudged to a space ===
--- chunk 1 ---
Refund requests must be submitted within thirty days of
delivery. Requests submitted after that period will be
--- chunk 2 ---
declined automatically.

=== CHINESE, cut at N characters, no space to nudge to ===
--- chunk 1 ---
退款申请必须在收到货物后三十天内提交。超过该期限提交的申
--- chunk 2 ---
请将被自动拒绝。

Chunk 1 ends in the middle of the word 申请 (“request”), splitting a two-character word across the boundary. Neither half means what the whole does, and neither chunk contains the term a user would search for. The English version has a space to fall back to; the Chinese version has nothing, so the cut lands wherever the counter stopped.

=== CHINESE, cut on sentence punctuation ===
--- chunk 1 ---
退款申请必须在收到货物后三十天内提交。
--- chunk 2 ---
超过该期限提交的申请将被自动拒绝。

Two complete sentences, each containing 申请 intact. The only change is that the separator list knows 。 exists.

What to do

1. Detect the language before splitting, and store it on the chunk. You cannot choose a separator list without knowing the script. Language is also a filter worth having at query time, so it belongs in the metadata regardless.

2. Extend your separator list with CJK punctuation. 。, ！, ？, ；, ：, 、, and the full-width forms of Latin punctuation. This is the cheapest fix available and it converts a splitter that was falling through to raw characters into one that cuts at sentence ends. Order matters — see choosing separators.

3. Use paragraph and line breaks, which do exist. Newlines are script-independent. Even without word or sentence segmentation, cutting at paragraph boundaries is available in every language and is usually better than counting.

4. Use a word segmenter where you need word boundaries. Thai in particular has neither word spaces nor sentence punctuation, so a dictionary or model-based segmenter is the only way to find boundaries at all. Several exist for each of the affected languages; treat this as a per-language dependency, and check what your text-processing stack already provides before adding one.

5. Measure size in tokens, not characters, when the corpus is mixed. A single character budget across languages means the same number is generous in one script and stingy in another. The unit you count matters most exactly here, and tokenisers vary a great deal in how efficiently they encode non-Latin scripts.

6. Set size targets per language, if you keep counting characters. Less principled than step 5 and much easier to retrofit. If your corpus is 90% English with a Chinese subset, a per-language multiplier gets you most of the way.

7. Do not strip punctuation you do not recognise. Cleaning pipelines written for English sometimes remove non-ASCII characters, which deletes the sentence boundaries you need along with much of the text.

What still breaks

Mixed-script documents. A Japanese technical document containing English identifiers, code and product names. Language detection returns one answer for the document and the paragraph you are cutting may be the other language. Detect per paragraph where you can.

Vertical text and ruby annotations. Furigana and similar reading aids extract as interleaved characters, which corrupts the text before any splitting decision. Same class of problem as scrambled reading order in a paginated document — not fixable at the split.

Segmenters disagree. Word segmentation for Chinese and Japanese is genuinely ambiguous, and two segmenters will give different boundaries for the same string. This is a real limit, not a tooling gap.

Right-to-left scripts bring a different set of problems — directional marks, mixed-direction runs — that are about text handling rather than about the absence of delimiters, but they belong on the same checklist.

Traditional and simplified variants, or a corpus mixing them, will affect matching more than chunking. Worth knowing that it is not a chunking problem so you do not try to fix it here.

How to tell if it worked

The direct check: for a sample of chunks in each non-space-delimited language, look at the first and last few characters of each chunk and ask a reader of that language whether the boundary falls between words. If you have no reader available, a weaker proxy is to run a word segmenter over the concatenated document and count how many of your chunk boundaries fall inside a segmented word. That number should be near zero and, with a character splitter and no CJK punctuation, will not be.

Also plot the length distribution per language. A spike at the maximum for one language and not for others means the separator list has nothing to match in that script, and the splitter has quietly reverted to counting.