evals-create-suite
Create an Eval Suite
Overview
Eval suites live in dedicated kbn-evals-suite-<name> packages. Each suite is a self-contained Playwright project that uses the evaluate fixture from @kbn/evals to run LLM experiments with datasets, tasks, and evaluators.
Inputs to Collect
- Suite name (kebab-case, e.g.
my-feature) - Parent directory under
x-pack/(e.g.x-pack/platform/packages/shared/ai-infra/orx-pack/solutions/security/test/) - Owner GitHub team handle (e.g.
@elastic/appex-ai-infra) - Group (
platform,security,observability,search) - Visibility (
sharedorprivate) - Whether custom fixtures are needed (chat client, esArchiver, supertest, etc.)
Do NOT Use node scripts/scout.js generate
Eval suites are not standard Scout test configs. The Scout generator creates test/scout/ directories that are picked up by Scout's CI discovery glob -- this will break because evals configs use createPlaywrightEvalsConfig (not createPlaywrightConfig) and contain non-JS files (like .text prompt files) that Playwright cannot parse.
More from elastic/kibana
codeql
Work with CodeQL in Kibana — write, test, and debug custom queries locally, fetch scan results from GitHub, and validate inline suppression comments. Use when writing or debugging CodeQL queries, running CodeQL unit tests, analyzing SARIF results, fetching scan results, or checking codeql suppression justifications.
1optimize-bundle-size
Reduce plugin `page load bundle size` and avoid unnecessary increases in `packages/kbn-optimizer/limits.yml`. Use when proactively optimizing bundles, investigating CI page-load overages, or reviewing PRs that change bundle limits.
1kibana-api
Shared utilities for interacting with a local Kibana instance. Provides auto-detection of Kibana URL and auth, and a kibana_curl wrapper.
1api-authz
Kibana API route authorization patterns. Use when configuring route security, working with requiredPrivileges, using authzResult for privilege-based branching, opting out of authorization, or naming custom privileges.
1ftr-testing
Deep reference for the Kibana Functional Test Runner (FTR). Use when reading, analyzing, debugging, or reviewing FTR tests, including config anatomy, services, page objects, loadTestFile patterns, data loading, tags, CI wiring, and common FTR idioms.
1validate-oas
Use when you need a quick VALID or NOT VALID result for a scoped Kibana OAS area, and first ensure the generated `oas_docs` inputs are up to date so validation runs against the current environment rather than stale snapshots.
1