GenAI on Amazon EKS

End-to-end opinionated guidance for running generative AI / LLM workloads on Amazon EKS, structured as a 6-layer stack from compute hardware up through the AI gateway. This skill is opinionated: it recommends one AWS-canonical reference stack — the JARK stack (JupyterHub + Argo + Ray + Karpenter) extended with vLLM serving and a LiteLLM gateway — and surfaces the alternatives plus the customer-context flags that justify deviating.

Two sources are the canonical foundation and every recommendation must align with one or both: the EKS AI/ML Best Practices guide and the awslabs/ai-on-eks reference implementation. For "how do I run a single EKS cluster well" (compute, networking, upgrades) use eks-best-practices; for designing/building the cluster itself use eks-design / eks-build. This skill is the GenAI workload layer on top.

When to Use This Skill

Activate when the user wants to:

• Choose GenAI hardware — NVIDIA GPU (p5/g6/g6e) vs AWS Neuron (Trainium/Inferentia)
• Serve an LLM (vLLM, Ray Serve, Triton, KServe) or run distributed training/fine-tuning on EKS
• Design GPU/Neuron scheduling (Karpenter NodePools, device plugins, EFA, Capacity Blocks, Spot)
• Pick ML storage (FSx for Lustre, Mountpoint S3 CSI, EFS, S3 Vectors) or wire GPU/Neuron observability
• Stand up an AI gateway (LiteLLM/Envoy), RAG pipeline, or agentic platform on EKS
• Optimize GenAI cost (Neuron migration, Spot+checkpoint, KV-cache tiering, consolidation)