hyperpod-cluster-debugger
HyperPod Cluster Debugger
Operating policy. Run read-only diagnostics yourself. Never run a command that changes cluster, node, or workload state — present each one as a Suggested command (run this yourself) block and wait for the customer to run it. Destructive order: investigate → reboot → replace (replace destroys root + secondary volumes; not supported on Slurm controller nodes).
Before any state-changing CLI: ask if it's IaC-managed. HyperPod clusters, SGs, EKS access entries, and IAM are usually provisioned via CloudFormation / CDK / Terraform. If yes, the fix belongs in IaC — running the CLI will drift and the next deploy reverts it. Use the CLI only when IaC is unavailable (locked out, predates IaC, mid-review).
scripts/diagnose-cluster.sh is read-only: it collects state via AWS APIs (and SSM for Slurm controller health) and prints each issue as [FAIL] ... → references/<file>.md § <section>.
| Reference | Open when |
|---|---|
| cluster-diagnostics-detail.md | Per-finding remediation runbook (§ A–L) |
| cluster-operations.md | Operational deep-dives (EFA SG, EKS access, SSM, Slurm, filesystem) |
| cloudformation-errors.md | § H needs the full per-resource CFN error catalog |
| capacity-planning.md | § B or --validate flags capacity / subnet sizing |
| lifecycle-scripts.md | § C points at a specific lifecycle failure |
| iam-permissions.md | Full IAM policy for the diagnostic |