troubleshoot-spark-job
Installation
SKILL.md
Diagnose failed or degraded Apache Spark workloads on AWS Glue, Amazon EMR on EC2, Amazon EMR Serverless, and Amazon EMR on EKS. Uses the SageMaker Unified Studio Spark Troubleshooting MCP server (spark-troubleshooting) to analyze job logs, error traces, and resource metrics. Optionally follows up with the Spark Code Recommendation MCP server (spark-code-recommendation) to generate targeted code fixes - but only when the diagnosis identifies a code-level fix.
Reference Documentation
- error-handling.md -- MCP connection issues and setup guidance
Workflow
Constraints:
- You MUST call
analyze_spark_workload(thespark-troubleshootingserver) FIRST as the primary diagnostic tool for every troubleshooting request. It provides deeper analysis than AWS job run APIs alone. - You MUST NOT call
spark_code_recommendation(thespark-code-recommendationserver) standalone or as a first step. It is a FOLLOW-UP ONLY: invoke it afteranalyze_spark_workloadhas produced a diagnosis AND that diagnosis identifies a code-level fix AND the user wants code changes. Calling it without a prioranalyze_spark_workloaddiagnosis produces lower-quality root cause analysis. - You MUST NOT call
spark_code_recommendationwhen the diagnosed fix is configuration-only (e.g. memory, partition, or resource tuning with no code change). Present the configuration fix directly. - Prefer
analyze_spark_workload's natural chaining into code recommendation over an explicit standalonespark_code_recommendationcall. - Do NOT call
aws emr describe-step,aws emr describe-cluster,aws glue get-job-run,aws emr-serverless get-job-run,aws emr-containers describe-job-run, or any AWS CLI command as a diagnostic step. The MCP tool provides all the information needed. - Do NOT call
aws emr list-clusters,aws emr-serverless list-applications,aws emr-containers list-virtual-clusters,aws glue list-jobs,aws glue get-job-runs, or any discovery/list API. Ask the user for identifiers instead.