Databricks Observability

Overview

Monitor Databricks job runs, cluster utilization, query performance, and costs using system tables and the Databricks SDK. Databricks exposes observability data through system tables in the system catalog (audit logs, billing, compute, query history) and real-time Ganglia metrics on clusters.

Prerequisites

• Databricks Premium or Enterprise with Unity Catalog enabled
• Access to system.billing, system.compute, and system.access catalogs
• SQL warehouse or cluster for running monitoring queries

Instructions