Characters, words, tokens: what your splitter is counting
Your splitter is configured with a maximum size. Your embedding model has a maximum input length. There is a good chance those two numbers are measured in different units, and nothing in the pipeline will tell you when they disagree.
The disagreement is not uniform. It is small on English prose and large on exactly the content that already retrieves badly — identifiers, tables, code, non-Latin scripts.
Three units, three things measured
Characters are what most splitters count by default, because counting them requires nothing. A character is cheap, exact, and unrelated to anything the embedding model cares about.
Words are what humans estimate in. Splitting on whitespace is nearly free and slightly more meaningful than characters, since a word is at least a unit of language. It also assumes whitespace delimits words, which is not true of every script.
Tokens are what the model actually consumes. A tokeniser maps text to a vocabulary of subword pieces; common words are usually one piece, rare words several, and unusual character sequences one piece per character or worse. This is the unit your model’s input limit is expressed in, and it is the only one that answers “will this chunk be truncated.”
The important property: the ratio between characters and tokens is not a constant. It is a property of the text.
Where the units diverge
Here is the same character budget spent on four kinds of content. All four fragments below are the same length in characters and wildly different in tokens.
--- fragment A: ordinary English prose ---
Customers on the annual plan may cancel at any time and will
receive a prorated refund for the unused portion of the term.
--- fragment B: identifiers and code ---
const rtrySchdlr = new RetryScheduler({maxAttempts: 5,
backoffFactorMs: 250, jitter: true, onExhausted: dropEvent});
--- fragment C: a table row with codes ---
| SKU-8842-XL | a3f9c1e7-4b2d | 2026-04-01T09:15:00Z | 14.99 |
--- fragment D: base64 and hashes ---
sha256:9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6cA runs at close to the tokeniser’s best case: mostly whole words, mostly one piece each. B splits
rtrySchdlr and backoffFactorMs into several pieces apiece and spends a token on every brace,
colon and comma. C is almost all identifiers, timestamps and punctuation. D is close to the worst
case a tokeniser has — an arbitrary character sequence with no vocabulary entries to match, so it
degrades toward one token per character or two.
So a splitter set to a fixed character count produces chunks that consume dramatically different amounts of the model’s input allowance depending on what happens to be in them. The ones that consume most are the ones densest in the identifiers a user is most likely to search for.
The split, shown
One section of a configuration reference, cut at a fixed character count and then cut against a token budget. The token figures are illustrative and tokeniser-specific; the point is that the two boundaries land in different places on the same text.
=== SPLIT AT A FIXED CHARACTER COUNT ===
--- chunk 1 (chars 0–300 · well under the token cap) ---
## Connection settings
Connections are pooled per process. The pool is created
lazily on the first query and is not shared across forks.
--- chunk 2 (chars 300–600 · OVER the token cap) ---
pool_max_conns=32 pool_min_idle=4
pool_acquire_timeout_ms=2500
pool_recycle_after_s=3600 tls_min_version=1.3
tls_verify_peer_name=true keepalive_probe_interval_s=15
stmt_cache_entries=512 stmt_cache_evict=lru
[continues — tail silently discarded before embedding]Both chunks are the same length in characters. Chunk 1 is prose and sits comfortably inside the model’s input limit. Chunk 2 is nothing but identifiers, underscores, digits and equals signs, each of which the tokeniser splits into several pieces — so the same character count is far more tokens, and the tail of the chunk never reaches the model.
=== SPLIT AGAINST A TOKEN BUDGET ===
--- chunk 1 (prose, under the token cap) ---
[Configuration → Connection settings]
Connections are pooled per process. The pool is created
lazily on the first query and is not shared across forks.
--- chunk 2 (settings, under the token cap) ---
[Configuration → Connection settings → pool]
pool_max_conns=32 pool_min_idle=4
pool_acquire_timeout_ms=2500
pool_recycle_after_s=3600
--- chunk 3 (settings, under the token cap) ---
[Configuration → Connection settings → tls and cache]
tls_min_version=1.3 tls_verify_peer_name=true
keepalive_probe_interval_s=15
stmt_cache_entries=512 stmt_cache_evict=lruCounting in tokens produced three chunks where counting in characters produced two, and the identifier-dense material was split into pieces small enough to survive. Nothing about the character version looked wrong: the chunk lengths were even, no boundary fell mid-word, and a third of chunk 2 was invisible to the index.
What goes wrong when the units disagree
Silent truncation. If your character limit maps to more tokens than the model accepts, the tail is discarded. No error, no warning; the chunk is embedded as though it ended early. This is the failure worth caring about most, and it has its own post.
Wasted headroom in the other direction. Set the character limit conservatively enough that the worst content fits, and your ordinary prose chunks come out far smaller than intended. You have optimised for base64 and paid for it on every English paragraph — which pushes you toward the small end of the chunk-size trade-off without deciding to.
Chunk-length statistics stop meaning anything. A distribution measured in characters cannot tell you whether any chunk is near the model’s limit, so the one diagnostic that would have caught the truncation is blind to it. Measure the distribution in the unit of your constraint.
Cross-corpus surprises. A pipeline tuned on English documentation behaves differently the day somebody adds a corpus in another language, and the difference shows up as retrieval quality rather than as an error.
Which unit to count in
Count in the unit of the constraint you are actually bound by. If the binding constraint is your embedding model’s input length, count tokens with that model’s tokeniser. Not a different model’s tokeniser, and not an estimate — different model families segment text differently, and the estimate is worst on the content where you need it to be best.
Use characters as a fast pre-filter, tokens as the check. Tokenising every candidate boundary is slower than counting characters, and on a large corpus that cost is real. A workable arrangement: use characters to find the approximate boundary, then tokenise the resulting chunk once to verify it is under the limit, and split it further only if it is not. You pay for one tokenisation per chunk rather than one per candidate offset.
Keep the measured length on the chunk. Store the token count as a field alongside the text, the way you store everything else a chunk needs to carry. It costs a field, it makes the distribution auditable, and it lets a downstream stage know the size of what it is handling without recomputing it.
Do not tune two limits at once. If you switch from characters to tokens, keep the effective size roughly where it was, verify nothing regressed, and only then change the size. Changing the unit and the number in one step means you cannot attribute the result.
Token counts are approximate, and that is fine
Two honest caveats.
Any number you quote for a token count is specific to one tokeniser. It is not portable across model families, and a claim like “this paragraph is about sixty tokens” is a statement about one vocabulary. Say so once in your documentation and move on; the alternative is pretending a universal ratio exists.
And the tokeniser your embedding model uses may not be the one your generation model uses. They are different models with different vocabularies. The count that matters for whether a chunk gets truncated is the embedding model’s. How much of a generation model’s window a set of chunks consumes is a separate question with a separate answer, and conflating them is how people end up sizing chunks against the wrong ceiling entirely.
How to check your corpus
Take a few hundred chunks spanning your document types. For each, compute the character length and the token length with your embedding model’s tokeniser, and look at the ratio.
You are looking for two things. First, the maximum: if any chunk is at or above the model’s input limit, you have been truncating and did not know. Second, the spread: if the ratio is roughly constant across your corpus, characters are a fine proxy and you can stop thinking about this. If it varies by a factor of two or more between document types, the unit is a real problem and the fix is to count in tokens for the types where it bites.
Either way you will have learned something concrete about your corpus in about an hour, which is better than adopting a conversion factor from someone else’s.