data-engineering-streaming
Installation
SKILL.md
Streaming Data Systems
Real-time data ingestion and stream processing with Apache Kafka, MQTT, and NATS JetStream. Covers producers, consumers, and stream processing patterns for data engineering pipelines.
Quick Comparison
| Feature | Apache Kafka | MQTT | NATS JetStream |
|---|---|---|---|
| Use Case | High-throughput event streaming | IoT, mobile, constrained devices | Cloud-native, microservices |
| Throughput | Millions/sec | Thousands/sec | Hundreds of thousands/sec |
| Durability | Disk-based log, replayable | Ephemeral (configurable) | Disk-based persistence |
| Ordering | Per-partition | N/A (topic-based) | Per-subject |
| Python Client | confluent-kafka | paho-mqtt | nats-py |
| Best For | Event sourcing, CDC, log aggregation | Sensor data, telemetry | Service-to-service messaging |