Kafka engine health

Answers "are the Kafka consumers polling, committing and keeping up, and is anything starving them?" from system.kafka_consumers, system.metric_log, system.query_views_log and system.text_log. Run altinity-expert-clickhouse-connection first if the connection mode, cluster and time window are not yet established.

Query packs

• checks.sql — 7 checks: consumption health per consumer (kafka-01), average rows per commit (kafka-02), rebalances and assignments (kafka-03), consumers versus message broker pool size (kafka-04), pool utilization over 12 hours (kafka-05, needs system.metric_log), slow materialized views on Kafka tables (kafka-06, needs system.query_views_log), Kafka messages in the log (kafka-07, needs system.text_log).
• advanced_checks.sql — 4 checks: consumer exception drill-down for one table (needs {db} and {kafka_table}), total lag per table, lag per partition, broker connection health. The last three parse rdkafka_stat. The file also contains a three-statement consumption-speed recipe marked @skip-matrix: it creates a temporary table, sleeps, then compares snapshots, so all three statements must run in ONE clickhouse-client session. It cannot run over MCP or as separate --query calls.
• troubleshooting.md — common Kafka errors and configuration fixes (ACL errors, poll interval, dead letter queue, offset rewind, parallel consumption tuning).
• references/rack-awareness.md — cross-AZ and client.rack investigation; load it when the question involves AWS MSK, KAFKA_CLIENT_RACK, cross-AZ cost, VPC endpoints, or a Kafka cluster migration.
• reference.md — background (Kafka engine and librdkafka settings); read only when you need to explain a recommendation.

How to run the query packs