Efficacy Measurement

Engagement is the metric that arrives free in your event stream, which is why most products prove only that people came back. This skill is about the other three quarters of the claim — did they learn it, can they use it, and does anyone outside your company agree — plus the instrument design that makes those answers survive contact with a skeptic.

The organizing idea: a number you can move at will cannot support a claim about the world. Every node here is a way to hand part of the grading to something you do not control — an external standard, an independently authored item pool, your own population in aggregate, the shape of a distribution.

Decide what to measure

• [[references/four-axes-that-do-not-substitute]] — enjoyment, mastery, real-task application and external standard are four independent questions, each needing its own method; and none of them is one number.
• [[references/borrow-the-instrument-you-do-not-control]] — run your users through the incumbent's own test, compare against a published study of the incumbent's users, and report the axis where you lose.

Design the instrument

• [[references/build-the-item-pool-independently-of-the-content]] — questions drawn from the material you shipped measure exposure; an independently authored pool is your holdout set.
• [[references/embed-a-pre-test-and-post-test-in-the-product]] — put the measurement in the path as a gate, ask about the next segment before you teach it, and keep the scores private.
• [[references/thin-sample-many-people-to-grade-the-curriculum]] — a handful of questions per person, aggregated across everyone, grades your content instead of your users and hands you a ranked build list.

Score it well