The Agent Skills Directory

[REMOTE_CODE_EXECUTION]: The documentation instructs users to clone the BigCode Evaluation Harness repository from GitHub and install it using pip install -e ., which involves downloading and installing third-party code from an external repository.
[COMMAND_EXECUTION]: The tool relies on an --allow_code_execution flag that enables the execution of model-generated code on the host system to perform unit testing. This capability allows for arbitrary code execution if the model output contains malicious logic.
[EXTERNAL_DOWNLOADS]: The skill references and suggests downloading resources from several external sources, including GitHub (bigcode-project), the GitHub Container Registry (ghcr.io), and Hugging Face datasets. These are well-known services and organizations in the AI research community.
[INDIRECT_PROMPT_INJECTION]: The evaluation workflow is designed to ingest and execute untrusted data in the form of code generated by large language models, creating a vulnerability surface for injection attacks.
Ingestion points: Model-generated code solutions for benchmarks such as HumanEval, MBPP, and MultiPL-E (documented in SKILL.md and references/benchmarks.md).
Boundary markers: The instructions suggest using Docker containers (ghcr.io/bigcode-project/evaluation-harness-multiple) as a security boundary to isolate the execution of untrusted code.
Capability inventory: The system can execute arbitrary code on the host or inside a container via the evaluate library's code_eval metric and subprocess calls.
Sanitization: The skill relies on external isolation mechanisms (Docker) rather than performing static analysis or sanitization on the model-generated output before execution.

evaluating-code-models