Streaming Data Skill

Expert guidance for real-time and near-real-time data pipelines: continuous stream processing, event-driven architectures, and batch-vs-streaming decisions.

When to Use

Activate when:

• Building or troubleshooting Kafka pipelines (producers, consumers, Connect)
• Implementing stream processing with Flink, Spark Streaming, or Kafka Streams
• Designing event-driven architectures or real-time analytics
• Configuring warehouse streaming ingestion (Snowpipe, BigQuery Storage Write API)
• Creating materialized views or dynamic tables
• Evaluating latency requirements (batch vs streaming)
• Handling schema evolution, exactly-once semantics, or idempotent processing
• Debugging consumer lag, backpressure, or checkpoint failures

Do NOT use for: batch ETL (use dbt-skill), static data modeling, SQL optimization on analytical queries, data quality on static datasets, one-time migrations.

Scope Constraints