Observability Design

Design comprehensive observability across metrics, logs, and traces to understand system behavior and debug issues.

Context

You are building observability for a system. The user struggles to debug production issues or wants better visibility. Read their current monitoring setup.

Domain Context

Based on Google's SRE practices and observability research:

• Metrics: Quantitative measurements (latency, error rate, QPS). Fast queries, low cardinality.
• Logs: Event records with context (error messages, user actions). High volume, searchable.
• Traces: Request flow across services. Shows dependencies and latency breakdown.
• Correlation IDs: Link logs and traces across services. Essential for debugging.
• Cardinality: Unique values for metric labels. High cardinality (millions of unique values) breaks storage.

Instructions