Add Scorable LLM-as-a-Judge to Your Application

These instructions guide you through creating LLM evaluation judges with Scorable and integrating them into your codebase. Scorable is a tool for creating LLM-as-a-Judge based evaluators for safeguarding applications. Judge is the Scorable term for grouping evaluations from different metrics (Helpfulness, Policy Adherence, etc...)

Execution Contract

You are responsible for completing Scorable setup and integration end-to-end in as few turns as possible.

You MUST analyze the codebase for LLM interaction points.
You MUST install and use Scorable CLI directly.
You MUST execute judge generation commands yourself.
You MUST integrate judge execution into code yourself.
You MUST run verification checks after changes.
You MUST update project documentation for usage.
You MUST NOT delegate technical steps to the user except where explicitly required like setting up the API key if not using a temporary key.
You MUST continue until implementation is complete or a hard blocker is reached.