The Agent Skills Directory

Local System Interaction via Harbor CLI: The skill utilizes the Harbor command-line interface to initialize and run evaluation tasks. This interaction involves the use of Docker to provide an isolated environment for executing agent code, ensuring that evaluations do not directly impact the host system.
External Service Integration for Trace Sourcing: The skill incorporates integration with LangSmith to retrieve and analyze execution traces. This involves the use of the langsmith CLI and requires an API key, which the skill correctly instructs the user to provide via environment variables for secure handling.
Management of Sensitive Credentials: The instructions provide clear guidelines for managing API keys and other secrets, emphasizing the use of environment variable references and avoiding hardcoded credentials in prompts, images, or logs. This aligns with standard security best practices for secret management.
Dynamic Execution Surface: To perform its primary function, the skill generates and executes Python scripts (such as adapters and verifiers) and configuration files. This dynamic execution is scoped to the evals/ directory and is designed to facilitate controlled simulations and automated scoring.
Indirect Prompt Injection Risks: The skill ingests untrusted data in the form of execution traces and agent outputs. The instructions include specific guidance for 'Verifier Design' that recommends treating agent content as untrusted and using explicit boundary instructions to prevent embedded directions from influencing the automated judging process.

eval-engineering