The Agent Skills Directory

[COMMAND_EXECUTION]: The skill's documentation and workflow examples in SKILL.md suggest using os.system() to invoke shell scripts and the lm_eval CLI. These f-string-based commands (e.g., in Workflow 2) use variables like checkpoint_path and step, which could be vulnerable to command injection if an attacker can influence those values. Additionally, references/custom-tasks.md provides examples using subprocess for executing code logic.
[DYNAMIC_EXECUTION]: The skill provides instructions for benchmarks such as HumanEval and MBPP which involve executing Python code generated by the model. This capability is gated by the --allow_code_execution flag. Furthermore, references/custom-tasks.md demonstrates how to implement custom evaluation metrics that use subprocess to execute generated code snippets.
[INDIRECT_PROMPT_INJECTION]: The skill is designed to ingest and process untrusted data from external benchmarks and model generations, creating a surface for indirect prompt injection.
Ingestion points: Untrusted content enters via dataset_path (supporting HuggingFace datasets, local JSONL, and CSV files) and the resulting model predictions analyzed during evaluation.
Boundary markers: The skill uses Jinja2 templates (e.g., doc_to_text) to format prompts, but these do not sanitize or escape model-generated content that may be executed later.
Capability inventory: The skill relies on os.system (SKILL.md), subprocess (references/custom-tasks.md), and file-writing capabilities to automate the evaluation pipeline.
Sanitization: No explicit validation or sanitization is documented for model-generated code before it is executed by the evaluation harness or custom metrics.
[CREDENTIALS_UNSAFE]: The documentation in references/api-evaluation.md provides instructions for setting up API keys for OpenAI and Anthropic using environment variables. While the examples use standard placeholders (e.g., sk-...), the skill facilitates the management and use of sensitive API credentials within the agent's environment.

evaluating-llms-harness