gke-ai-troubleshooting-node-unresponsive-timeout
Installation
SKILL.md
Troubleshoot unresponsive GKE TPU and GPU nodes (NodeStatusUnknown)
When the Compute Engine host of a TPU or GPU node has a fatal hardware error,
kernel panic, or non-maskable interrupt (NMI) lockup, the guest OS stops
responding. The kubelet can no longer send heartbeats, so the node Ready
condition becomes Unknown with Reason: NodeStatusUnknown (Kubelet stopped posting node status.). If node auto-repair is disabled on the node pool, GKE
doesn't repair the node. The node can stay NotReady, and pods on it can stay
in Terminating, which blocks multi-host JobSet workloads from recovering.