Tagged “evaluation”
-
What your chunk-length distribution tells you
Histogram your chunk lengths. A spike at the maximum, a pile of near-empty chunks and a long right tail each name a specific bug in your splitter.
-
Read fifty chunks
Before building anything measurable, sample your index and read it. Most chunking bugs are visible to a person in twenty minutes and invisible to a metric.