vally-eval

Pass

Audited by Gen Agent Trust Hub on Oct 15, 2026

Risk Level: SAFEDYNAMIC_EXECUTIONINDIRECT_PROMPT_INJECTIONCOMMAND_EXECUTIONEXTERNAL_DOWNLOADS
Full Analysis
  • Dynamic Execution via Grader Plugins: The skill provides instructions for using vally-cli to load external TypeScript modules as grader plugins (e.g., tests/vally/vally-graders.ts). This allows for the runtime execution of code based on relative file paths within the repository, which is a standard pattern for extensible testing frameworks but warrants oversight of the loaded scripts.
  • Indirect Prompt Injection Surface: The evaluation framework is designed to modify the agent's system prompt and auto-inject context into user prompts based on data defined in eval.yaml and fixture files. While necessary for simulating test scenarios, this creates a surface where the agent's behavior is influenced by external configuration files.
  • Ingestion points: Evaluation suites read from eval.yaml files, fixture/ directories, and trajectory data in results.jsonl format.
  • Boundary markers: The documentation does not specify explicit delimiters for preventing instruction injection within these files, relying on the test runner's configuration.
  • Capability inventory: The system can provision Azure resources, execute shell commands through the test runner (e.g., npm run test:vally), and modify agent system prompts.
  • Sanitization: The skill includes a validation script (validate-stimulus.ts) that enforces conventions and validates special tag values before execution.
  • Command Execution for Testing: The instructions involve executing shell commands via npm and npx to run tests and validate evaluation suites. These commands trigger local scripts and CLI tools that interact with the development environment.
  • Automated Resource Provisioning: The "Azure fixture" feature is designed to provision Azure resources to serve as context for integration tests. This involves interacting with cloud infrastructure and requires associated credentials and cost management, which are documented as expected behaviors for this utility.
Audit Metadata
Risk Level
SAFE
Analyzed
Oct 15, 2026, 02:11 AM