Instrumenting for Observability

Overview

Once code is running in production nobody can watch it directly. The only thing knowable about it is what it records about itself, so a failure that was never instrumented is a failure nobody can explain — and unlike most engineering debt, this one can't be paid off later. The moment the data is needed is after the incident, and by then the window to collect it has closed.

The core discipline of this skill is one inversion:

Instrumentation is designed backwards from the questions it must answer, never forwards from the code.

Walking a diff and adding a log line wherever something interesting happens produces volume proportional to code complexity rather than to operational risk. Writing the failure questions first, then adding one signal per question, produces the opposite — and gives a pruning rule in both directions: a question with no signal is a gap, and a signal that answers no question gets deleted.

When to Use

• A new endpoint, job, consumer, or user-facing flow is about to merge
• An existing path is being changed in a way that alters how it can fail
• Reviewing a diff for whether it could be debugged at 3am
• A retro concluded that an outage took too long to diagnose because the data wasn't there
• An engineer asks for "logging" or "metrics" without saying which, or which fields