Running With/Without Evals

Overview

A with/without eval captures real opencode run transcripts on the same task, once with a plugin loaded and once without, to show what the plugin actually changes. The output is recorded sessions, not a benchmark.

Core principle: the result is only worth publishing if it would survive a hostile reader who has the transcripts. That means pre-registering the bar before you run, keeping a clean baseline, including the control that could disprove your thesis, and conceding what the baseline already does well.

The reusable harness lives at tests/manual/with-without-eval/run-arm.sh. Reuse it; don't reinvent the isolation.

When to use

• Building or refreshing the with-without-systematic demo page
• Validating that a plugin/skill change actually moves model behavior
• Producing a credibility artifact for public promotion

When NOT to use