Eval Engineering

Create one Harbor task at a time, run it, inspect the result, and repeat with the user.

map harness + environment -> propose directions -> user chooses
-> draft specs -> user approves -> build + run + audit -> repeat

Use the latest Harbor release. Put task source under evals/.

Boundaries

• Task: instruction.md plus an Environment and Verifier.
• Harness: the complete agent Harbor runs: model, prompts, loop, repository-defined tools, middleware/hooks, memory/session behavior, and Harbor adapter. Harbor calls this the Agent.
• Environment: the container/world around the Harness: OS, files, backing data, services, identity, permissions, network, clock, and mutable state.
• Verifier: the test script that independently scores the response, trajectory, or resulting Environment state.

Repository-defined tool code belongs to the Harness. The data or service behind it belongs to the Environment. Example: a docs agent's search_docs definition and result parsing stay in the Harness; the frozen search index and its error behavior live in the Environment. If production supplies a tool server dynamically, keep that server in the Environment and preserve how the Harness discovers and calls it.