evaluating-code-models

MEDIUM W011: Third-party content exposure detected (indirect prompt injection risk).

• Third-party content exposure detected (high risk: 0.85). Runtime evaluation loads benchmark problems from external HuggingFace datasets (e.g., openai_humaneval, mbpp, evalplus/*) and formats their text into prompts that are fed into the agent LLM context, so the prompt content is outsider-authored.

MEDIUM W012: Unverifiable external dependency detected (runtime URL that controls agent).

• Potentially malicious external URL detected (high risk: 0.80). The skill explicitly instructs fetching and running remote code — e.g., git clone https://github.com/bigcode-project/bigcode-evaluation-harness.git (followed by pip install -e .) and docker pull ghcr.io/bigcode-project/evaluation-harness-multiple (then docker run) — which downloads external content that will be executed at runtime and is presented as a required workflow dependency.