Karafka

Apache Kafka for Ruby with high-performance multi-threaded processing.

Iron Laws

1. Process in batches when possible - Better throughput
2. Handle dead letter queues - Don't lose failed messages
3. Monitor lag and throughput - Key operational metrics
4. Use idempotent consumers - Safe to reprocess
5. Commit offsets after processing - At-least-once delivery
6. Partition by business key - Preserve ordering

Installation

gem 'karafka', '~> 2.5'
gem 'karafka-web'  # For Web UI