Constitutional AI - Harmlessness from AI Feedback

Quick start

Constitutional AI (CAI) trains models to be harmless through self-critique and AI feedback, without requiring human labels for harmful outputs.

Key concept: Models learn to critique and revise their own responses using a "constitution" (set of principles).

Two phases:

1. Supervised Learning (SL): Self-critique + revision
2. Reinforcement Learning (RL): RLAIF (RL from AI Feedback)