Huawei Cloud CCE AZ Power Outage Chaos Experiment

Overview

End-to-end CCE AZ power outage chaos experiment in three phases:

Phase 1: Prepare          Phase 2: Execute           Phase 3: Log Analysis
─────────────             ──────────────             ──────────────────
Discover clusters    →    Pre-check (nodes Ready)  → Load experiment context
Select cluster            Shutdown (BatchStop)       Discover dependencies
Discover node AZs         Monitor (Node/Pod)         Collect logs (LTS + Pod)
Select target AZ          Wait (duration)            Analyze error patterns
Validate compatibility    Rollback (BatchStart)      Generate analysis report
Generate + deploy config  Verify + report

Scenario: Shut down all CCE nodes in one AZ → Kubernetes marks nodes NotReady → Pods evicted and rescheduled to other AZs → start nodes back → verify recovery. Tests cluster HA under AZ-level failure.