Model Serving on Kubernetes

Production ML model serving with KServe and Triton — canary deployments, autoscaling, and GPU-aware scheduling.

When to Use This Skill

Use this skill when:

• Serving scikit-learn, PyTorch, TensorFlow, or ONNX models at scale
• Implementing canary deployments and A/B testing for ML models
• Autoscaling inference pods based on request rate or GPU metrics
• Deploying LLMs with Triton or KServe on Kubernetes
• Managing multiple model versions with traffic splitting

Prerequisites

These are version-dependent sketches, not a complete cluster installation. Pin chart/image versions and check installed CRD schemas and serving mode (Knative versus standard). Canary fields and KEDA targets depend on that mode; target an actual scalable workload supported by the chosen operator, not an assumed InferenceService scale subresource. Configure ingress authentication/TLS, least-privilege model-store access and resource/cost bounds. Remove or implement the placeholder transformer; arbitrary models and “-v2” IDs must exist and be licensed.