Palantir Python Transform Pipeline

Overview

Create a pipeline whose inputs, outputs, engine, write behavior, and quality checks are explicit before the first production build. Prefer the simplest supported compute engine that satisfies data scale and feature requirements, then prove the choice with Foundry metrics.

Prerequisites

• Identify the owning project, Code Repository, input and output datasets, schema contract, data classification, and build schedule.
• Confirm whether the workload requires Spark or can use a single-node engine such as Polars, pandas, or DuckDB.
• Read references/official-docs.md and inspect current input transaction history before selecting incremental semantics.
• Develop on a sandbox branch with representative but appropriately protected data.

Current Contract

• Python transforms support batch and incremental pipelines, reusable libraries, expectations, and single-node or distributed engines.
• Input and output datasets must differ; using the same dataset creates a cyclic dependency.
• Incremental input modes and output write modes have precise transaction semantics; modify and replace are not interchangeable.
• A snapshot build may be needed when incremental transaction history becomes progressively slow or invalid.