The Agent Skills Directory

[DYNAMIC_EXECUTION]: The documentation describes the use of the --allow_code_execution flag, which is required for benchmarks like HumanEval and MBPP. This results in the execution of code generated by the model under test to verify functional correctness.
[INDIRECT_PROMPT_INJECTION]: The skill is designed to ingest and process outputs from various LLMs. When evaluating code-generation models, there is a risk that a malicious or poorly aligned model could generate harmful code that the harness then executes.
Ingestion points: Model-generated responses to benchmark prompts are processed by the evaluation scripts in SKILL.md and references/custom-tasks.md.
Boundary markers: Not explicitly detailed in the provided instructions, though standard for the underlying tools.
Capability inventory: The skill utilizes os.system in automation examples (SKILL.md) and subprocess in custom task utilities (references/custom-tasks.md), combined with the capability to execute generated code via --allow_code_execution.
Sanitization: The guide provides examples of code execution for benchmarking purposes without emphasizing sandboxing or restricted environments.
[COMMAND_EXECUTION]: The skill provides Python code snippets in SKILL.md that use os.system() to automate evaluations during training loops. While presented as a convenience for developers, these calls execute shell commands based on script variables.

evaluating-llms-harness