scenario-model-comparison
Scenario Model Comparison
Overview
A comparison is one brief run unchanged across several models, then judged on criteria written down before the first paid call. Every candidate has its own contract, so the comparison lives in the normalization: what is held fixed, what each schema forces to differ, and where each number came from. recommend shortlists, dry_run prices, the jobs_wait rows carry the billed cost, the grid tool puts the results side by side, and a table delivers. Connection and the core loop: see the scenario skill; a rubric for judging output against a brief: scenario-refine-loop. If a sibling skill named here is missing from your available skills, ask the user to install it (npx skills add scenario-labs/skills --skill <name>); unattended, proceed from tool schemas and flag the gap.
Quick reference
| Step | Do |
|---|---|
| 1. Frame | One brief, its inputs (prompt text, reference asset ids), three to five pass/fail criteria, all written before running |
| 2. Shortlist | recommend with the brief as prompt (limit up to 10); search for candidates the user named; keep three to five |
| 3. Normalize | model_schema_get each candidate: the shared fields, the caps that differ, the flags that rewrite prompts |
| 4. Price | model_run with dry_run: true (top-level, beside model_id, never inside parameters) per candidate and utility model; apply the Budget gate |
| 5. Run | model_run with top-level wait: false per candidate, launched back to back, then one jobs_wait over all the job ids |
| 6. Measure | cuCost from the jobs_wait rows; createdAt to updatedAt from job_get for the seconds |
| 7. Judge | The sheet from model_scenario-grid-maker, then each asset at full size, scored against the criteria |
| 8. Deliver | One table row per candidate; the assets filed in one collection and tagged by model id |