evaluating-systems
Installation
SKILL.md
Evaluating systems
Evaluate a system by what it produces. A system can operate exactly as designed and still fail its intended outcome.
State what should happen
Express the intervention as a causal hypothesis:
Changing X should improve Y under Z conditions because of M.
Establish what happens before the intervention. Choose an observation window long enough for its proposed mechanism to produce evidence.
Separate operation from outcome
Verification determines whether the system behaves as designed. Validation determines whether that behavior improves the intended outcome.
An apparent improvement can still fail when its cost is unacceptable or it cannot be sustained. Treat those as distinct claims rather than folding them into a single success metric.