Apache Kafka Best Practices

Version: Kafka 4.2.0. ZooKeeper is fully removed in Kafka 4.x — use KRaft mode for all cluster deployments.

Producer Patterns

Idempotent Producer (CRITICAL)

• Enable enable.idempotence=true
• Prevents duplicate messages
• Requires acks=all, retries > 0, max.in.flight.requests.per.connection <= 5

Exactly-Once Semantics

• Use transactional API: initTransactions(), beginTransaction(), commitTransaction()
• For exactly-once end-to-end processing

Performance

• Batching: linger.ms (wait for batch to fill)
• Compression: compression.type=snappy or lz4
• batch.size: 16KB default, tune based on message size