Validate Guidelines

When to use

• User proposes or has made changes to runner/models/guidelines.ts and wants to ensure they don't regress other models
• User says "validate the guideline changes" or "run the guideline validation"
• Before committing guideline edits, to confirm improvements or no-regression across sonnet, opus, gemini, chatgpt (or a subset)

Overview

Guideline changes are validated by running evals twice per model: once with the current (before) guidelines and once with the proposed (after) guidelines. Results are compared; any eval that passed before and fails after is a regression. The goal is to ensure changes improve or at least do not regress scores across multiple models.

Step 1: Identify the change

Determine which guideline sections were modified in runner/models/guidelines.ts (e.g. function_guidelines, query_guidelines, file_storage_guidelines) and the intent (new rule, clarification, token compaction).

Step 2: Build before and after guideline files