Model Eval

Run a side-by-side model evaluation for an agent, comparing two Bedrock models on identical queries.

When to Run

• Before switching an agent's model (e.g., Nova Lite → Sonnet)
• After prompt changes that might affect model-specific behavior
• When evaluating a new model release for an existing agent
• Periodically to verify model quality hasn't regressed

Input

• Agent name (e.g., "keeper", "captain", "scout")
• Model A ID (current model, e.g., us.amazon.nova-2-lite-v1:0)
• Model B ID (candidate model, e.g., us.anthropic.claude-sonnet-4-6)
• Optional: custom eval queries (defaults to Athena traces, then generated)

Process