Data Platform Pipeline (AWS Serverless)

This skill builds the ingestion → storage → catalog → query layers of a serverless data platform. The output is a working CDK TypeScript project plus the Glue scripts and Athena DDL to run it. Downstream consumption (Quick Sight, dashboards, chat agents) is out of scope — this skill stops at "data is queryable in Athena."

The skill is opinionated: best practices are baked in, not presented as options. If a choice has a clear winner for serverless analytics on AWS, the skill picks it.

One deliberate exception. If the catalog + query layers must live in a different region from storage (follow-up #8), the choice between the two mechanisms is presented to the user, not decided — it turns on a legal question (is data residency a mandate or a preference?) and a cost question that depends on customer-specific volume. See reference/cross-region-query.md §2.

🔴 CRITICAL RULES (never violate)

1. S3 Tables catalog needs --extra-jars + --user-jars-first: 'true' — Glue hard-fails (Cannot find constructor for interface org.apache.iceberg.catalog.Catalog) without the s3-tables-catalog-for-iceberg-runtime JAR. --datalake-formats iceberg alone is NOT enough.
2. spark.sql.extensions is STATIC in Glue 5 — set ALL Spark/Iceberg config via the job's --conf in defaultArguments, NEVER via spark.conf.set() at runtime (fails with Cannot modify the value of a static config).
3. No views on the S3 Tables catalog — CREATE VIEW is unsupported (incl. cross-catalog refs from AwsDataCatalog). Use mart_* CTAS tables instead of v_* views on the Iceberg path.
4. DROP TABLE (purge=false) is unsupported on S3 Tables — use DROP TABLE ... PURGE or the S3 Tables API (aws s3tables delete-table).
5. NEVER switch architecture due to tool versions — upgrade the CLI/CDK instead. Tool versions are fixable in 2 minutes; architecture is permanent. Fall back to Hive only when the user explicitly opts in, or S3 Tables is genuinely unavailable in-region.

Full failure modes, the supported/unsupported Athena DDL list, encoding rules, CDK gotchas, and the known-issues table → reference/gotchas.md.