Data Pipelines, Scheduled Jobs and Data Integrity

Data integrity: what users need is restores, not backups

Users judge you by whether their data is there and correct when they ask for it. A backup that has never been restored is a hope, not a plan.

Threat model

Data loss comes from many combinations of:

• Root cause: user error, application bugs, operator/automation error, infrastructure failure, malicious action, migration bugs.
• Scope: wide (many entities) vs. narrow (one user/object).
• Rate: sudden (big bang) vs. slow and creeping (a bug corrupting a little data each day for months).

Slow, creeping corruption is the most dangerous: by the time it's noticed, it's in every recent backup and replica.

Defense in depth