AKS GPU & Inference (Day-2)

Operate and troubleshoot GPU and model-serving workloads on AKS after they exist. This is the failure → check → fix loop, not a setup guide — for enabling GPUs, KAITO, or a model runway, use azure-skills airunway-aks-setup.

Operating rules

• Read-only by default. Diagnose and propose; do not scale, cordon, delete node pools, or delete KAITO Workspaces unless the user explicitly asks — deleting a Workspace has a cost trap (below).
• Evidence before conclusion. GPU symptoms have layered causes (quota → capacity → scheduling → driver). Quote the event, node capacity, or condition that supports the root cause.

The four walls (check in this order)

Most "my GPU workload won't run" incidents are one of these, and they stack — clearing one exposes the next:

1. GPU quota is 0 by default. Every N-series VM family starts at 0 vCPUs in every region, and quota is measured in vCPUs, not GPUs. Check: az vm list-usage --location <region> -o table and look for the family row (e.g. Standard NC Family vCPUs, Standard NCADSA100v4 Family vCPUs). Error signature: "Insufficient vCPU quota for Standard NCASv3_T4 Family…".
2. Quota ≠ capacity ≠ SKU-eligibility. Even with quota, a region/zone can lack physical GPU capacity, and AKS gates some GPU SKUs behind a separate eligibility ticket. Check what's actually creatable: az vm list-skus --location <region> --resource-type virtualMachines -o table.
3. The taint/toleration/nodeSelector triad. GPU pools use the convention taint sku=gpu:NoSchedule. A GPU pod must carry the matching toleration (key=sku, value=gpu, effect=NoSchedule), request nvidia.com/gpu: 1, and (by convention) select accelerator: nvidia. A missing toleration is the most common Pending cause after quota.
4. No nvidia.com/gpu advertised on the node → a driver / device-plugin model mismatch. See the driver decision matrix in references/gpu-scheduling.md.

Symptom → reference