Observability Instrumentation

Use when scale-readiness-review/resilience-patterns says "add a metric"; use debug-runtime to diagnose live failures.

Workflow

Pick the path (payment, payout, settlement, reconciliation, hot endpoint); list the questions ops will ask (did X succeed, failure rate/p95, queue depth, reconciliation drift).
Log: structured key-value/JSON, correlation id, stable business keys (transaction, account, idempotency key, provider ref, tenant) — never secrets/PANs/credentials; log decisions/transitions and call boundaries.
Metrics via the repo's client:
- RED: Rate, Errors, Duration (histogram) for request-driven paths.
- USE: Utilization, Saturation, Errors for resources (pools, workers, connections).
- Domain signals: queue depth/age, settlement lag, retry counts, breaker state, reconciliation drift, idempotency-conflict rate.
- Label by operation/outcome; bound cardinality.
Trace: propagate context through jobs/queues/calls; span each external call with duration/status.
Alert on symptoms users feel (error rate, latency, backlog, drift); SLO thresholds/burn rates; name cause and next step; delete unused alerts.
Verify: trigger the path, confirm logs/metrics/traces appear and alerts fire on failure.