Eval System

A clean, minimal benchmark system that runs production code against test cases and writes results to a co-located markdown artifact (eval.md). The eval.md is a judgment surface: it exists so a strong external model or a human can read the real outputs and judge quality, not so the eval can auto-declare a winner.

Core Principles & Rules

1. Import Production Logic Directly

• ALWAYS import and invoke the actual production function, prompt constants, and schemas from the feature file.
• NEVER duplicate prompts, system messages, or JSON schemas inside eval.ts.
• Why: Re-implementing prompts in the eval causes prompt drift. When production changes, a duplicated eval becomes a false benchmark testing obsolete behavior.

2. Benchmark Real Runs — Never Estimate

• ALWAYS execute every candidate model to capture ground-truth latency, tokens, and cost.
• NEVER estimate or hallucinate latency, token counts, or pricing in a report without running the models.
• Why: Models cannot predict real streaming latency, throughput, or cost. Only direct execution records the truth.