threat-list
Threat List
Outcome: a short memo that says, for each item on the fixed list, whether it applies to this system, why (which interface, access level, or data flow makes it possible), how practical the most realistic version would be, and whether it is worth a research look. Zero leads is a legitimate memo.
The habit comes from Carlini's description on the Latent Space podcast (around 00:53) of keeping a list of bad things that could happen and checking every new system against all of it, accepting that most checks come back empty. The list lives in references/checklist.md; read it before the pass.
Inputs
Before running the list, establish from the user or from available documentation:
- What the system is: model, dataset, serving API, agent with tools, training pipeline, or retrieval stack.
- Who controls what: who trains, who serves, who supplies data, who can query, and with what access (query outputs only, logits or probabilities, weights, training data).
- Exposed interfaces and their limits: rate limits, logged fields, returned fields, file or URL ingestion, tool calls.
- Assets that matter: weights, training data, user data, system prompt, downstream actions.
If these are unknown, say which are unknown and run the list anyway, marking items whose applicability depends on the missing fact.