data-partitioning
Installation
SKILL.md
Data Partitioning
Design partitioning strategies (sharding, bucketing) to distribute data across multiple databases or nodes, enabling horizontal scaling.
Context
You are designing data partitioning for large-scale systems. The user faces single database bottlenecks (throughput or storage). Read their data volume, access patterns, and consistency requirements.
Domain Context
Based on designing data-intensive applications and partition strategies from Dynamo/Cassandra:
- Range Partitioning: Partition by key range (UserID 1-1000 on DB1, 1001-2000 on DB2). Simple but uneven load.
- Hash Partitioning: Partition by hash(key). Even distribution, but range queries require scatter-gather.
- Directory-Based: Maintain mapping of key → partition. Flexible, adds lookup cost.
- Rebalancing: When adding nodes, some partitions migrate. Strategy affects availability.
- Hot Partitions: Some keys receive disproportionate traffic. Mitigate by splitting hot partition or using caching.