Operating safely

REQUIRED BACKGROUND: the principal-engineering skill.

Overview

Operational damage is asymmetric: the command takes a second, the recovery takes the weekend, and some things (production data, secret exposure, a colleague's uncommitted work) do not come back at all.

Destructive operations

• Look at the target first. Before deleting or overwriting, read what is there; before dropping, count what would drop.
• Targeted over bulk. Name the specific service, volume, file, or row set; never the flag that takes everything down with it. Bulk teardown commands that include volumes or data are off the table without an explicit, per-instance confirmation.
• The operator owns live process lifecycles. Ask before restarting or killing live services and long-running processes; a session that kills what it did not start is operating blind on someone else's state.
• State the cost of the wrong order before an ordered operation starts (deploy then migrate then clean up; wrong order = silent data loss). Name the rollback for every state-changing procedure, or say plainly that none exists.
• Database changes need their own confirmation. Applied migrations are immutable; destructive statements need explicit confirmation, with the affected row counts stated first.

Secrets