Observability

Overview

This skill turns modern observability practice into actionable guidance. The aim is not a survey of the field but to help you instrument code, debug production, and design reliability mechanisms well.

The throughline: observability is the ability to understand any state your system can reach -- including ones you have never seen before -- by interrogating its outputs, without shipping new code to capture them first. In the AI era the bottleneck is no longer writing code; it is validating and understanding the code you (or an agent) just shipped. Observability is the feedback loop that closes that gap.

When to use this skill

Use it when instrumenting code, reviewing someone's telemetry, debugging a production problem from first principles, designing SLOs/alerts, configuring sampling or pipelines, choosing where to store telemetry, or making a system legible to AI agents. For a one-line "add a log here", you don't need the skill; for anything that shapes what data exists or how you'll answer questions later, you do.

The core mental model

• Observability is a property of dependable software, not a product you buy. It sits alongside reliability and availability. You can't recover state you never emitted -- if the information wasn't captured, it's gone forever. So the design question is always "what will I need to ask later?"
• Monitoring answers known-unknowns; observability answers unknown-unknowns. Threshold dashboards tell you that something is wrong and roughly where. They cannot tell you why a novel, multi-causal failure happened. Distributed systems fail in a zoo of partial-degradation states, so the rare "inexplicably slow, no smoking gun" case is now the common one.
• The litmus test: can any engineer, regardless of seniority, diagnose a complex issue purely by interrogating emitted data? If it needs institutional intuition held in a few senior heads, your observability is low.
• Systems are sociotechnical. The humans operating the system are part of it. Diagnosis must be teachable and transferable, not folklore.
• Only production is production. Tests prove theory; production proves reality. You are not done building until you have observed your change working in production. Test in prod, or live a lie.