Normalizing Text for Measurement

Any pipeline that reduces a document to a number — word count, reading time, readability, a token budget, a dedup key, a checksum, a search index — has two halves: a normalizer that turns markup into something comparable, and a measurer that consumes it. Reviews concentrate on the measurer. Almost every bug in this shape lives in the normalizer, and none of them raise.

The failure mode is uniform: the output is still a plausible string, the metric is still a plausible number, and nothing anywhere reports a problem. See reporting-derived-metrics for the other end of the same pipeline — what the measurer does when the sample is too small.

A missing separator fuses two blocks into one token

A token-walking stripper emits text for content tokens and separators for close tokens. Miss one close token and two blocks become one word.