Apache Kafka Best Practices

Core Principles

• Use acks=all with min.insync.replicas=2 and replication.factor=3 for durability
• Enable idempotent producers (default since Kafka 3.0) to eliminate duplicates
• Commit offsets manually after processing — never rely on auto-commit for at-least-once
• Use Schema Registry with Avro or Protobuf — never StringSerializer for structured data
• Deploy in KRaft mode — ZooKeeper is removed since Kafka 4.0
• Monitor consumer lag, under-replicated partitions, and offline partitions continuously
• Right-size partitions: too many wastes resources, too few caps throughput

Topic Design (CRITICAL)

topic-naming

Use a hierarchical naming scheme with dots or hyphens. Never use underscores.