Sikaru evals

An eval runs every eligible example in an environment dataset against an agent's live version, then reports pass rates for each agent version. Publish the agent first (see author-sikaru-agent), because a saved draft isn't what gets tested.

1. Build the dataset

Each example needs an input and can have expected, tags and metadata.

{"input": "Which pandas function reads a CSV file?", "expected": "pandas.read_csv", "tags": ["io"]}
{"input": [{"role": "user", "content": "Drop rows with nulls"}], "expected": "df.dropna()"}

• input is a string (one user message) or a list of messages.
• expected is optional. A string becomes the expected assistant reply.
• In a CSV, the columns are input, expected and tags, with tags separated by commas.
• The limits are 10 MB and 5,000 examples per file.