staff-engineering-skills-hot-partitions

Installation
SKILL.md

Hot Partitions Trap

You distributed the data evenly. But traffic isn't even -- one partition is on fire. Before choosing a partition key, ask: does this key distribute access patterns, not just data?

The Core Problem

Data distribution and access distribution are different things. Ten million users across 100 partitions is 100,000 users per partition. But if one user generates 50% of traffic, that user's partition handles 50% of total load. You've built a distributed system that behaves like a single node.

Real traffic follows power laws. A small number of entities generate most of the activity. Your partition scheme must account for this.

Detection: When You're Creating a Hot Partition

Stop and fix if you see:

  1. Partitioning by date/timestamp for write-heavy data -- today's partition receives ALL writes. Yesterday's is idle. This is the most common hot partition pattern for time-series, events, and logs.

  2. Partitioning by a low-cardinality key -- country code (US gets 50%), status field (90% are "active"), boolean flags. Low cardinality means few partitions, and the most common value dominates.

  3. A single global key in DynamoDB -- pk = "global-leaderboard" or pk = "config". Every request for that item hits the same partition. DynamoDB partitions can handle ~3,000 RCU / ~1,000 WCU per second per partition key.

Installs
1
First Seen
2 days ago
staff-engineering-skills-hot-partitions — triggerdotdev/staff-engineering-skills