Data Streaming

Modern Best Practices: choose the event backbone and stream processor separately, treat schemas and replay as product interfaces, default to event-time processing for stateful analytics, and verify managed-service behavior from primary docs before making vendor-specific recommendations.

Primary sources live in data/sources.json. Refresh time-sensitive claims against official docs before making definitive recommendations about managed services, version-specific features, limits, or pricing.

This skill covers the data platform side of streaming: event backbones, CDC, stateful processing, schema governance, and real-time delivery into lakes, warehouses, search, or serving systems.

When to Use

• Choose between Kafka, Redpanda, Pulsar, Kinesis, or managed Kafka offerings
• Design topic strategy, partitioning, retention, replay, and ordering guarantees
• Build or fix CDC pipelines with Debezium, Flink CDC, or managed database-streaming tools
• Choose between Flink, Kafka Streams, Spark Structured Streaming, or lighter transformation paths
• Define schema registry, compatibility, contract, and tombstone handling rules
• Deliver streams into Iceberg, Hudi, Delta, ClickHouse, warehouses, caches, or search systems
• Review streaming SLOs, lag, checkpointing, reprocessing, and operational failure modes

When NOT to Use