llm-output-privacy-risk
LLM Output Privacy Risk Assessment
Overview
Large language models (LLMs) present unique privacy risks that go beyond traditional ML systems. Because LLMs are trained on massive corpora that may contain personal data, they can memorise and reproduce verbatim training data — including names, email addresses, phone numbers, and other PII. Additionally, LLMs can hallucinate plausible but false personal data, creating defamation and accuracy risks. Prompt injection attacks can bypass safety guardrails to extract training data or system prompts containing confidential information. This skill provides a structured framework for assessing, mitigating, and monitoring privacy risks in LLM-generated outputs at Cerebrum AI Labs.
LLM Output Privacy Risk Categories
Risk 1: Training Data Memorisation
LLMs memorise training data, particularly sequences that appear multiple times or are distinctive. Extractable memorisation occurs when a model, given a prefix, completes the text with verbatim training data.
| Factor | Impact on Memorisation Risk |
|---|---|
| Model size | Larger models memorise more (Carlini et al., 2023) |
| Data duplication | Repeated sequences are memorised at higher rates |
| Training epochs | More passes over data increase memorisation |
| Data distinctiveness | Unique sequences (names, numbers) are more extractable |
| Temperature at inference | Lower temperature increases verbatim reproduction |