The Agent Skills Directory

[INDIRECT_PROMPT_INJECTION]: The skill processes untrusted external data, including evaluation datasets stored in S3 and custom metric definitions provided by the user. This creates an attack surface for indirect prompt injection where malicious instructions embedded in the data could attempt to influence the judge LLM's scoring or the agent's behavior.
Ingestion points: Evaluation datasets are accessed via S3 (references/llmaaj-evaluation.md) and custom metrics are provided as JSON input (references/llmaaj-custom-evaluation.md).
Boundary markers: The skill utilizes specific placeholders ({{prompt}}, {{prediction}}) for metric definitions, though raw evaluation data lacks explicit boundary markers.
Capability inventory: The skill can execute local scripts, call AWS APIs (SageMaker, Bedrock, S3), and generate/write executable Python and Jupyter Notebook files.
Sanitization: The skill implements a validation script (scripts/validate_custom_metrics.py) that uses the Pydantic library to enforce schema compliance and a 5,000-character limit on instructions.
[DYNAMIC_EXECUTION]: The skill generates executable content by populating code templates with user-supplied variables at runtime.
Evidence: Templates in code_templates/custom_scorer_evaluator.py and code_templates/llmaaj_evaluator.py are filled with ARNs, S3 URIs, and JSON blobs before being written to the filesystem as .py or .ipynb files.
[COMMAND_EXECUTION]: The agent is instructed to execute a local Python script to validate the integrity and format of user-provided metric configurations.
Evidence: references/llmaaj-custom-evaluation.md specifies the execution of python scripts/validate_custom_metrics.py on user-provided JSON artifacts.
[EXTERNAL_DOWNLOADS]: The generated code includes commands to install the SageMaker Python SDK from the official registry.
Evidence: Templates include %pip install --upgrade sagemaker>=3.7.1, which targets a well-known and official package maintained by the vendor.

model-evaluation