Prompt Eval Builder

Domain knowledge for building evaluation artifacts from prompt specifications. Provides extraction patterns, output templates, validation logic, and invocation prompt generation rules.

When to Use

• Generating promptfoo eval configs from command/agent prompts
• Extracting testable assertions from instruction text (primarily LLM rubrics)
• Creating test invocation prompts (user inputs that test the command)
• Validating generated YAML output

Assertion Types

LLM Rubrics (Default): Generated for most behavioral requirements. Rubrics contain REQUIRED/PROHIBITED/EDGE CASES sections and evaluate behavior holistically by checking output text AND Metadata JSON.

Programmatic Assertions (Complex Cases): Generated when requirements need exact counting, strict sequencing, or complex conditional logic that cannot be expressed in natural language rubrics. Uses type: javascript with file:// references to shared assertion functions.

Skill Files