Read docs/public/broker-operator/README.md § Observability for the full reference.
Enable Prometheus by adding [observability] prometheus_bind = "HOST:PORT" to the broker TOML. The broker spawns an axum listener serving text/plain; version=0.0.4 exposition.
Scrape the listener with your Prometheus / VictoriaMetrics / etc. setup. Expect these 11 metric families (some are dormant today — registered but not yet observed):
- broker_rpc_seconds{op} — RPC latency. Dormant.
- broker_cache_metadata_hits_total — metadata-cache hits.
- broker_cache_object_hits_total — object-byte-cache hits.
- broker_cache_object_fills_total — object-byte-cache fills.
- broker_cache_evictions_total — dormant.
- broker_authz_decisions_total{outcome} — labels are allow, deny, error.
- broker_watch_fanout (gauge) — dormant.
- broker_policy_epoch_advances_total — increments per successful SIGHUP reload.
- broker_redirect_emissions_total{kind} — labels are read, write. Increments at every fixture-driven and plugin-driven redirect emission site.
- broker_lifecycle_events_total{event} — labels are reload_ok, reload_failed, drain_start, drain_complete.
- broker_uptime_seconds.
Pipe grpc.health.v1.Health/Check to your health-check probe. Reports Serving whenever backend-kind introspection on the active Stack succeeds. Readiness is not flipped during drain (the gRPC server stops accepting new connections via serve_with_incoming_shutdown but Health/Check keeps the last reported state until the server thread exits).
Wire tracing. The broker uses init_tracing_from_env so the standard RUST_LOG / OTEL_* env vars apply. OTLP push as a [observability] otlp_endpoint field is reserved (surfaces Unsupported if set); layer opentelemetry_otlp onto init_tracing_from_env manually if you need it.

Tracing fields to collect

Per the broker-operator persona doc:

principal.id — on every object-IO span (broker.stat, broker.read, broker.write, broker.list, broker.list_versions, broker.list_address_roots).
policy_epoch — on every object-IO span and every pb::ErrorDetail.
object.address — redacted via RedactedUrl (scheme + host + port + path only, no query / fragment / userinfo).
audit_id — on ReadRedirect and WriteRedirect envelopes and on every pb::ErrorDetail. Freshly minted when the host's RequestContext.audit_id is None.
cache.hit, redirect.kind — when applicable.
outcome ∈ {allow, deny, error} — on the broker_authz_decisions_total counter.

Fields not stamped today: route.id and backend.id on per-RPC spans. Closing the gap is tracked.

Audit log shape (what you don't get)

There is no AuditRecord / AuditEvent type, durable sink, or operator-facing explain-decision <audit-id> tool. Diagnostic fields land in tracing spans, error details, and redirect envelopes only. Operators that need an audit log point a log aggregator at the broker's tracing output and filter on audit_id + policy_epoch + principal.id.

Sinks must redact physical URLs before logging query strings or signed headers. The pipeline must never log raw bearer tokens, credential bytes, or usable signed URLs.

Suggested alerts

rate(broker_lifecycle_events_total{event="reload_failed"}[5m]) > 0 — a SIGHUP reload failed validation; the old broker is still live on stale config.
rate(ovstorage_auth_decisions_total{outcome="error"}[5m]) > 0 — the built-in auth Layer returned evaluation errors (not denies). Check the active policy and listener authentication configuration.
broker_uptime_seconds reset implies a restart — pair with broker_policy_epoch_advances_total to distinguish a clean reload-induced advance from a process crash.
rate(broker_redirect_emissions_total{kind="write"}[5m]) falling to zero on a route that previously emitted them suggests the plugin can no longer mint redirects (and the broker is falling back to inline upload, which has the 64 MiB WRITE_BODY_BYTE_CAP).

References

docs/public/broker-operator/README.md § Observability — the full reference.
ovstorage-operator-debug-broker — using these surfaces to triage runtime issues.
ovstorage-operator-configure-broker-policy — interpreting broker_authz_decisions_total{outcome="deny"}.

ovstorage-operator-monitor-broker

Monitor the Broker

Goal

Recipe