Vally eval suites

Skills in the azure-skills plugin are required to have integration tests that run prompts against an LLM agent to evaluate whether they help the agent accomplish goals in target scenarios. Such integration tests are written as vally eval suites, using vally as the underlying tool for running tests and grading the agent outcome.

Write vally eval suites

Vally eval suites are written as yaml documents. All eval suites share eval spec.

Refer to the official documentation on the schema of the spec and the schema of the eval suites writing-eval-specs.

Vally eval suites for azure-skills plugin have the following file layout. The shared eval spec is located at <repo-root>/.vally.yaml. The eval suites are categorized by skills. The eval suites for each skill are located at <repo-root>/evals/<skill-name>/eval.yaml, e.g. <repo-root>/evals/azure-ai/eval.yaml. If a skill needs fixture files for its eval suites, it should organize such fixture files in a fixture directory under its directory, e.g. <repo-root>/evals/azure-ai/fixture/.

The eval suites can be organized into separate files under a skill's eval directory. For example, you can have <repo-root>/evals/<skill-name>/evalA.yaml and <repo-root>/evals/<skill-name>/evalB.yaml. When running the vally suites using npm run test:vally command, the test script will run suites from all these .yaml files.

Migrate integration tests

azure-skills plugin have implemented JavaScript integration test using Jest as the underlying test runner. All such integration tests are under tests/**/integration.test.ts files.

To migrate integration test for a skill to vally suites, create its eval suite spec at <repo-root>/evals/<skill-name>/eval.yaml, add a suite that runs the same prompt and uses vally's built-in graders to grade the trajectory of the agent run. If the integration test grades the agent run in a way that vally's built-in graders don't support, refer to the official documentation on how to create a custom grader writing-custom-grader.