Measurement Validity

Most product metrics are designed to be moved, not to be believed. This skill is about the second kind — a number an outsider with no reason to trust you can be asked to accept, and the inference errors that quietly turn a measurement into a story.

Start with the instrument. Then check what you are allowed to conclude from it.

Design the instrument

• [[references/benchmark-on-an-instrument-you-did-not-build]] — an internal completion metric proves users moved through your product; an external scale proves the product did something. Includes funding the field that will adjudicate your claim.
• [[references/ship-a-subscore-only-if-it-adds-information]] — the three gates a component score must pass before it earns a place on the dashboard, and why the noisiest component is often the most valuable one.
• [[references/measure-willingness-alongside-capability]] — for anything users decline rather than fail at, nerve is the binding constraint and it moves independently of skill.

Report it honestly

• [[references/publish-the-component-where-you-lose]] — naming your worst dimension is what makes the good numbers legible as measurement rather than marketing, and what it costs you.
• [[references/clean-cohort-buys-attribution-costs-generality]] — a filtered cohort is the only way to isolate what you caused, and the same filter stops the result generalising. Also covers disclosing the stages of a funded study you were walled off from.
• [[references/audit-bias-at-intersections]] — bias can live at a combination of attributes and vanish when either is checked alone, so a one-row-per-attribute fairness table proves almost nothing.

Don't let the instrument lie