Vally eval suites

Skills in the azure-skills plugin are required to have integration tests that run prompts against an LLM agent to evaluate whether they help the agent accomplish goals in target scenarios. Such integration tests are written as vally eval suites, using vally as the underlying tool for running tests and grading the agent outcome.

Write vally eval suites

Vally eval suites are written as yaml documents. All eval suites share eval spec.

Refer to the official documentation on the schema of the spec and the schema of the eval suites writing-eval-specs.

Vally eval suites for azure-skills plugin have the following file layout. The shared eval spec is located at <repo-root>/.vally.yaml. The eval suites are categorized by plugin and skills. The eval suites for each skill are located at <repo-root>/evals/<plugin-dirname>/<skill-name>/*.yaml, e.g. <repo-root>/evals/azure-skills/azure-ai/eval.yaml.

Use meaningful file names to categorize tests. If a skill needs fixture files for its eval suites, it should organize such fixture files in a fixture directory under its directory, e.g. <repo-root>/evals/azure-skills/azure-ai/fixture/. The vally test runner and stimulus validation script will load all *.yaml files except for those under a fixture/ directory. Make sure to put all fixture files under the fixture/ directory.

Why is there a custom executor

Our custom executor implemented features that vally doesn't support yet, such as early termination, system prompt modification, screenshot taking, etc. Besides, test-all-integration runs automated integration tests, collects its exported data and feeds the data to a dashboard web app under <repo-root>/dashboard/ to monitor skill integration test results.

If you intend to have your vally suites use any of the extended features or have their results be consumed by the dashboard, you MUST use the custom executor in your vally suites.