Terraform for Data Infrastructure

When to Use

Load this skill when the user needs to:

• Provision S3 or MinIO buckets for a data lake (versioning, lifecycle transitions, SSE-KMS encryption, bucket policies)
• Create IAM roles and policies for Spark, Airflow, or other data workloads with least-privilege access; configure IRSA on EKS
• Create or manage Amazon MSK (Managed Streaming for Kafka) clusters or Confluent Cloud Kafka clusters
• Deploy Airflow, Spark History Server, or other data tools on Kubernetes using helm_release
• Structure a Terraform monorepo with reusable modules and per-environment tfvars
• Manage Terraform state with S3 + DynamoDB locking or Terraform Cloud
• Set up GitHub Actions CI/CD for terraform plan / apply, or adopt Terragrunt for DRY multi-environment configs
• Write typed variables with validation, sensitive outputs, and locals for computed values

Project Structure

A scalable layout separates reusable modules from environment-specific entry points: