llm-red-team
Skill: LLM Red Team
Supplementary Files:
payloads.md— Prompt injection payloads (direct/indirect/encoded), jailbreak corpora (DAN, roleplay, many-shot, encoding), model-extraction probes, RAG-poisoning document templates, membership-inference scripts, agentic tool-abuse payloads, safety-filter bypass techniques (multi-language, tokenizer tricks, homoglyphs), hallucination induction patterns, model DoS vectors, perimeter recon (model fingerprinting), reporting templates, and defense-evasion obfuscation patternstest-cases.md— 12 structured test cases (TC-LR-001 .. TC-LR-012) covering recon, direct injection, indirect injection, jailbreak, extraction, RAG poisoning, agentic abuse, safety bypass, hallucination, DoS, defense bypass, and full red-team reportguides/llm-red-team-playbook.md— End-to-end red-team playbook (pre-flight authorization, six-phase methodology, tool deep-dives on promptfoo/garak/PyRIT/PurpleLlama, multi-turn attack-chain design, evidence collection, OWASP LLM Top 10 mapping, and the purple-team feedback loop)guides/llm-jailbreak-arsenal-playbook.md— State-of-the-art jailbreak arsenal (DAN/STAN/AIM persona patterns, Anthropic many-shot jailbreaking, Microsoft Crescendo multi-turn escalation, PAIR/TAP automated adversarial rewriting, GCG suffix optimization, prefix injection, payload smuggling via base64/ROT13, defense layers — Constitutional AI, Llama Guard, NeMo Guardrails, Azure AI Content Filter — plus battery construction and ASR reporting)
Summary
LLM red team skill domain covering offensive testing of large language model systems, generative AI applications, RAG pipelines, and agentic tool-using assistants. The skill equips the operator to discover and demonstrate semantic-layer vulnerabilities — prompt injection, jailbreaks, system-prompt extraction, training-data leakage, RAG poisoning, agentic tool abuse, and safety policy bypass — using the current open-source toolkit (promptfoo, garak, PyRIT, PurpleLlama, AI-Infra-Guard, llm-guard). Findings are mapped to the OWASP LLM Top 10 (LLM01-LLM10) and the emerging LLM-ATT&CK taxonomy, and reported with reproducible payloads, response evidence, and concrete remediation guidance.
Tools: promptfoo, garak, PyRIT, PurpleLlama (Llama-Guard / CyberSecEval), AI-Infra-Guard, llm-guard, picklescan, Burp Suite, custom Python (requests/openai), ffuf, mitmproxy
Domain: ai-red-team