Kafka in Rails

Kafka is the right tool when you have multiple consumers of the same event stream, need replay, or need to retain events for days. Wrong tool for "I want to send a job" — use Solid Queue / Sidekiq. This skill covers production patterns for Rails apps using karafka, the maintained Ruby Kafka client.

The opinion

Use karafka 2.x for both producing and consuming. Use the Confluent Schema Registry with Avro or Protobuf — never raw JSON in production. Pick partition keys by entity (e.g., account_id) so per-entity events are ordered. Consumers are at-least-once by default — make them idempotent. For transactional publish, use the outbox pattern (write to DB + outbox in one transaction, separate process flushes to Kafka).

Counter-positions:

• racecar (Heroku) — simpler, but less actively maintained; lacks karafka's web UI, swarm mode, and DLQ support.
• ruby-kafka — the underlying low-level client; rarely used directly anymore.
• rdkafka-ruby — karafka's underlying high-perf binding to librdkafka. Use directly only if you need very low-latency.

Setup