Parallel Geospatial

Reference for designing and debugging large-scale parallel geospatial processing on a single fat node. You do not write geospatial code at this scale single-threaded. The defaults below come from real continental-scale (15M polygons, 6T-pixel rasters) failures.

The shape of every problem here is the same: partition along an axis where work is naturally independent, then run an embarrassingly parallel pool over the partitions. The skill is identifying that axis cheaply and avoiding the operation-specific footguns that crash GEOS or DuckDB at scale.

The decision tree (start here)

For a new geospatial operation at >1M-feature or >1B-pixel scale, ask in this order:

1. Is there a natural disjoint partition? state, county, tile, hilbert-curve cell, raster window, time slice. If yes → ProcessPool over partitions. Stop, you're done thinking.
2. Is the work output-disjoint but input-overlapping? (e.g. spatial join: each output row needs a small bbox query against an input). Build an STRtree once on inputs, partition over outputs. Workers share a read-only tree via fork-inherited memory or rebuild small per worker.
3. Is it a global aggregation that can't be partitioned? Decompose into a tree-reduce: partition → reduce-per-partition → small final merge. Almost every real "global" op decomposes this way (union, dissolve, count-distinct, hilbert sort).
4. Is it truly serial? Rare. Consider whether you actually need the operation, or whether a different formulation (e.g. raster mask AND vs. polygon difference) sidesteps it.

If you find yourself running anything covering a continent in a single thread, stop and reconsider — single-threaded for >1M features is almost always a design bug, not a "we'll optimize later" one.

Footguns that cost whole batch jobs (real failures)