kafka

Purpose

Apache Kafka is a distributed event streaming platform used for building real-time data pipelines and streaming apps, enabling high-throughput, fault-tolerant messaging.

When to Use

Use Kafka for scenarios requiring real-time data ingestion and processing, such as log aggregation, event-driven architectures, or microservices communication; avoid it for simple queueing needs where lighter tools like RabbitMQ suffice.

Key Capabilities

Supports distributed streaming with topics, partitions, and replicas for scalability and durability.
Offers exactly-once semantics via transactional APIs to prevent data loss or duplication.
Handles high volumes with configurable retention policies, e.g., retaining messages for 7 days using log.retention.hours=168 in broker config.
Provides consumer groups for load balancing, where multiple consumers share a group ID to partition topic consumption.
Integrates streaming processing via Kafka Streams API for stateful transformations, like aggregating events with KTable objects.

Usage Patterns

To produce messages, create a topic first, then use a producer client; for consumption, subscribe to a topic and process messages in a loop. Always handle offsets manually or via auto-commit to avoid reprocessing. For batch processing, use Kafka Connect to ingest data from sources like databases. Pattern: Use idempotent producers for at-least-once delivery by setting enable.idempotence=true in producer configs.