The Agent Skills Directory

Dynamic Code Execution: The scripts/run_metrics.py utility uses Python's importlib to load and execute metric definitions from a user-provided file. This allows the skill to perform complex, custom evaluation logic.
Evidence: spec.loader.exec_module(mod) in scripts/run_metrics.py loads code from the file path provided via the --metrics_file argument.
Context: This is a standard pattern for extending the functionality of evaluation frameworks with custom logic, but it means the agent will execute any code contained within the specified file.
Indirect Prompt Injection Surface: The skill is designed to ingest and analyze agent trajectories (conversations, tool calls, and results). If these trajectories originate from untrusted interactions, they could contain adversarial content designed to manipulate the LLM judge's verdict.
Evidence Chain:
Ingestion points: scripts/run_metrics.py reads trajectory data from JSONL files provided by the user.
Boundary markers: Prompts in references/metric_templates.md use Markdown headers (e.g., # User request, # Agent trajectory) to delineate data types.
Capability inventory: The skill makes network calls to the Google Cloud AI Platform API and executes local Python code.
Sanitization: The skill provides templates and best practices in references/writing_checks.md to help users write robust evaluation rubrics that resist bias.
Command Execution & Cloud Interaction: The skill instructs the agent to execute various gcloud commands for authentication, project configuration, and API management.
Evidence: Multiple gcloud commands in the Setup section of SKILL.md (e.g., gcloud auth login, gcloud services enable).
Context: The skill adheres to the principle of least privilege by recommending the roles/aiplatform.user role instead of broader administrative permissions.
Usage Attribution: The skill requires the agent to add attribution tags to CLI commands and HTTP headers for telemetry purposes.
Evidence: references/attribution.md specifies the use of CLOUDSDK_METRICS_REQUEST_ATTRIBUTION and custom User-Agent strings.
Context: This is a standard practice for platform-integrated tools to monitor usage and does not involve the collection of sensitive user data.

agent-platform-autorater-customization