The Agent Skills Directory

[DYNAMIC_EXECUTION]: The helper scripts scripts/run_accuracy_test.sh and scripts/run_performance_test.sh use the sed utility to inject user-supplied shell variables into Python configuration files.
Evidence: In scripts/run_accuracy_test.sh, variables like $HOST_IP, $HOST_PORT, and $BATCH_SIZE are injected into Python dictionary structures using sed -i.bak. A maliciously crafted input (e.g., containing quotes and Python code) could result in arbitrary code execution when the modified Python file is subsequently imported by the ais_bench tool.
[COMMAND_EXECUTION]: The benchmarking scripts execute shell commands built from unvalidated user arguments.
Evidence: Both scripts/run_accuracy_test.sh and scripts/run_performance_test.sh construct a command string $CMD using positional arguments and then execute it via exec $CMD. This lack of argument sanitization could lead to shell injection if the agent passes untrusted user input to these scripts.
[INDIRECT_PROMPT_INJECTION]: The skill is designed to ingest and process external datasets for model evaluation, creating an attack surface for indirect prompt injection.
Ingestion points: Datasets are loaded from the ais_bench/datasets/ directory and external URLs provided in documentation (e.g., Alibaba Cloud OSS).
Boundary markers: No explicit boundary markers or instructions to ignore embedded prompts in dataset content are documented.
Capability inventory: The skill possesses file-write capabilities (to the outputs/ directory), network request capabilities (via model API classes like VLLMCustomAPIChat), and subprocess execution.
Sanitization: There is no evidence of sanitization or filtering of the content within the processed datasets before they are sent to the models or stored in results.
[EXTERNAL_DOWNLOADS]: The skill documentation encourages downloading benchmark datasets from external sources.
Evidence: SKILL.md provides instructions to download the GSM8K dataset from an Alibaba Cloud OSS bucket (opencompass.oss-cn-shanghai.aliyuncs.com). While OpenCompass is a recognized benchmark provider, downloading files from external URLs at runtime carries inherent risk if the source is compromised.

ais-bench