Interpreting and Controlling Model Behavior via Constitutions for Atomic Concept Edits

This skill enables Claude to apply the ACE (Atomic Concept Edits) constitution framework from Kalibhat et al. (2026) to real tasks. The core idea: systematically mutate prompts using targeted add/remove/replace operations on individual concepts, observe how a model's behavior changes, then distill those observations into a constitution -- a structured natural-language document of "Good Strategies" and "Bad Strategies" that predicts and controls how prompt edits affect model output. This is a black-box method: it requires no model internals, only the ability to query a model and evaluate its output.

When to Use This Skill

• When a user wants to understand why a model succeeds or fails on certain prompts and needs a systematic explanation, not anecdotes.
• When a user needs to steer model behavior (e.g., force shorter outputs, increase accuracy, improve image-text alignment) by learning which prompt modifications reliably work.
• When a user is comparing multiple models and wants to discover behavioral differences (e.g., "Does GPT-5 handle distractors differently than Gemini?").
• When a user asks to "red-team" or robustness-test a model by finding prompt perturbations that degrade performance.
• When a user wants to optimize prompt templates at scale by learning generalizable rewrite rules rather than hand-tuning individual prompts.
• When a user is building an autorater or evaluation pipeline and needs to identify which input features most affect a quality metric.

Key Technique

Atomic Concept Edits (ACEs) are the primitive operations. Given an input prompt, you first extract its explicit concepts (entities, attributes, constraints present in the text) and implicit concepts (plausible additions). Then you apply one of three operations: remove(c) deletes a concept, add(c) introduces a new concept, or replace(c, c') swaps one concept for another. Each ACE produces a mutated prompt that is sent to the target model and scored by a task-specific autorater (a binary classifier that judges whether the output meets the task goal).

Constitution learning is an evolutionary optimization loop. You start by collecting a labeled dataset of (original prompt, ACE, autorater score) triples. An LLM then drafts an initial constitution -- a structured list of mutation strategies labeled as "Good" (achieve the task goal) or "Bad" (fail). A surrogate classifier (another LLM call using the constitution as instructions) predicts autorater scores on held-out data. Misclassified examples become feedback: the constitution is iteratively refined over 5-20 epochs until the surrogate's predictions align with actual autorater scores. The result is a compact, human-readable document describing which types of edits reliably produce which behavioral outcomes.