setup-gpu-finops
Installation
SKILL.md
Setup GPU FinOps
Use this skill to implement cost transparency for self-hosted AI models and GPU workloads in Kubernetes clusters.
Core Rules
- DCGM Integration: Deploy the NVIDIA DCGM Exporter to expose low-level GPU metrics (utilization, memory, power) to Prometheus.
- Namespace Attribution: Use Kubecost (or OpenCost) custom pricing sheets to map GPU node costs down to the pod and namespace level.
- Idle Optimization: Implement KEDA (Kubernetes Event-driven Autoscaling) or Karpenter to scale GPU nodes to zero when inference queues are empty.
- Alerting: Configure Slack/PagerDuty alerts for teams exceeding their daily GPU compute budget.
- Metric Standardization: Normalize GPU metric names with
hw.gpu.*prefixes via OpenTelemetry Collector processors. - MIG-MULTI-TENANT-ISOLATION: For H100/A100/B200, configure NVIDIA MIG (Multi-Instance GPU) partitions to provide hardware-level isolation between teams sharing the same physical GPU — enforces memory and compute boundaries and prevents one tenant's OOM from affecting another's inference workload.
- KEDA-INFERENCE-AUTOSCALING: Deploy KEDA ScaledObjects driven by vLLM or Triton inference queue depth metrics to scale model replicas to zero when queues are empty for >15 minutes — configure
cooldownPeriod: 900sto prevent flapping. - VLLM-APC-COST: Enable vLLM v0.6+ Automatic Prefix Caching (APC) to reuse KV cache for repeated system prompts; structure prompts with static content first and dynamic user content last for maximum cache hit rate — reduces GPU compute cost by 20-60% for high-traffic repeated-context workloads.
- KV-CACHE-OFFLOAD-COST (2026): for agentic and long-context workloads, extend KV cache into CPU RAM / storage tiers (vLLM-SGLang offloading, NVIDIA ICMS, storage-backed KV) before scaling GPU count — offloaded KV turns idle GPU VRAM back into serving capacity; price the tiers separately in cost dashboards.
- SPECULATIVE-DECODING-THROUGHPUT (2026): EAGLE-3 / MTP draft models deliver up to 6.5x decode throughput on supported stacks (vLLM, SGLang; TPU support since 2026-10) — model the throughput-per-dollar gain using measured draft acceptance rate, and treat acceptance < ~70% as a net loss.
- RACK-SCALE-PRICING (2026): maintain pricing sheets for rack-scale Blackwell systems (GB300 NVL72: 72 GPUs, ~278GB HBM3e each, NVLink-pooled domain) with per-GPU fractional allocation math; track the Vera Rubin roadmap for refresh depreciation windows.