Kubernetes for Data Agent

Purpose

Configures Kubernetes for data-intensive workloads: Spark, Airflow, Kafka, with GPU scheduling, node pools, and storage optimization.

Agent Protocol

Trigger

User request includes: Kubernetes for data, Spark on K8s, Airflow on K8s, Kafka on K8s, Strimzi, Spark Operator, Volcano, GPU scheduling, node pools, data workloads K8s, Karpenter.

Protocol

1. Identify data workload types (Spark, Airflow, Kafka, batch).
2. Select operators (Spark Operator, Strimzi, Airflow Helm chart).
3. Design node pools for workload isolation.
4. Configure resource requests/limits for executors.
5. Set up GPU scheduling if needed.
6. Choose storage class and CSI driver.
7. Configure Karpenter or autoscaler for elasticity.