Managed Spark on Google Cloud

[!IMPORTANT]

You MUST ALWAYS follow the Task Execution Workflow when writing spark code.

Task Execution Workflow

Understand schemas: ALWAYS use @skill:discovering-gcp-data-assets skill or references/schema_direct_inspection.md to understand input and output schemas. Include the schema in your thought process BEFORE generating any code. Do NOT guess column names. Unless explicitly specified, assume that the assets are located in the same project. Avoid scanning for assets across other projects as it can take a long time. If an expected dataset or table does not exist, use @skill:discovering-gcp-data-assets to discover all similar tables in the namespace or project.

MINOR TYPO RULE: If there is a minor typo (e.g. employees vs employee), you can fix the error and proceed.