The Agent Skills Directory

[EXTERNAL_DOWNLOADS]: Fetches the BigCode Evaluation Harness from its official GitHub repository and retrieves Docker images from the GitHub Container Registry.
[REMOTE_CODE_EXECUTION]: Executes code generated by AI models to assess functional correctness (pass@k metrics) when the --allow_code_execution flag is enabled in the harness.
[INDIRECT_PROMPT_INJECTION]: The skill processes and executes untrusted code generated by external AI models as part of the benchmarking process.
Ingestion points: Model-generated code solutions loaded from generations.json or generated during evaluation in SKILL.md and references/benchmarks.md.
Boundary markers: None; model outputs are treated as executable code snippets.
Capability inventory: Full code execution capability via accelerate launch main.py and the code_eval metric as documented in references/custom-tasks.md.
Sanitization: None within the skill scripts; however, the skill documentation strongly advises using Docker for isolated execution to mitigate risks.
[DYNAMIC_EXECUTION]: Employs dynamic execution (exec) to run generated code snippets against benchmark unit tests.
[COMMAND_EXECUTION]: Includes instructions for using the --trust_remote_code flag, which allows the execution of custom scripts embedded in Hugging Face model repositories.

evaluating-code-models