Apache Kafka Streaming AI Skill Guide

Overview & Engine Architecture

Kafka is a distributed commit log: producers append to topics partitioned for parallelism; consumer groups divide partitions among consumers. Ordering is per partition (key-dependent). Brokers replicate partitions (ISR); consumers track offsets. Delivery is at-least-once by default; idempotent producers and transactions narrow failure modes but do not replace application idempotency across external systems.

Producers (idempotence?, acks, compression)
    -> brokers (leader/followers, ISR)
        -> consumer group (poll, process, commit offsets)
            -> side effects (DB, HTTP) — must be idempotent

When to use / when not to

Use when