troubleshoot-spark-job

Installation
SKILL.md

Diagnose failed or degraded Apache Spark workloads on AWS Glue, Amazon EMR on EC2, Amazon EMR Serverless, and Amazon EMR on EKS. Uses the SageMaker Unified Studio Spark Troubleshooting MCP server (spark-troubleshooting) to analyze job logs, error traces, and resource metrics. Optionally follows up with the Spark Code Recommendation MCP server (spark-code-recommendation) to generate targeted code fixes - but only when the diagnosis identifies a code-level fix.

Reference Documentation

Workflow

Constraints:

  • You MUST call analyze_spark_workload (the spark-troubleshooting server) FIRST as the primary diagnostic tool for every troubleshooting request. It provides deeper analysis than AWS job run APIs alone.
  • You MUST NOT call spark_code_recommendation (the spark-code-recommendation server) standalone or as a first step. It is a FOLLOW-UP ONLY: invoke it after analyze_spark_workload has produced a diagnosis AND that diagnosis identifies a code-level fix AND the user wants code changes. Calling it without a prior analyze_spark_workload diagnosis produces lower-quality root cause analysis.
  • You MUST NOT call spark_code_recommendation when the diagnosed fix is configuration-only (e.g. memory, partition, or resource tuning with no code change). Present the configuration fix directly.
  • Prefer analyze_spark_workload's natural chaining into code recommendation over an explicit standalone spark_code_recommendation call.
  • Do NOT call aws emr describe-step, aws emr describe-cluster, aws glue get-job-run, aws emr-serverless get-job-run, aws emr-containers describe-job-run, or any AWS CLI command as a diagnostic step. The MCP tool provides all the information needed.
  • Do NOT call aws emr list-clusters, aws emr-serverless list-applications, aws emr-containers list-virtual-clusters, aws glue list-jobs, aws glue get-job-runs, or any discovery/list API. Ask the user for identifiers instead.

Phase 0: Gather Job Identifiers

Installs
26
GitHub Stars
3.5K
First Seen
1 day ago