The Agent Skills Directory

[INDIRECT_PROMPT_INJECTION]: The skill is susceptible to indirect prompt injection in its grading and analysis components. In scripts/grade_answers.py, the _llm_grade function takes predicted outputs and ground_truth answers and interpolates them directly into a prompt for a secondary AI instance to verify correctness. Malicious content within the benchmark data or agent predictions could potentially influence the grader's decision or attempt to override instructions.
Ingestion points: The predicted and ground_truth fields from benchmark results in scripts/grade_answers.py, and question text in scripts/run_eval.py.
Boundary markers: None. The content is directly embedded into the prompt string without delimiters or instructions to ignore nested commands.
Capability inventory: Use of subprocess.run to call the claude CLI in scripts/grade_answers.py, scripts/run_eval.py, scripts/skill_routing_test.py, and scripts/skill_routing_test_v2.py.
Sanitization: No escaping or validation is performed on the ingested content before prompt interpolation.
[DYNAMIC_EXECUTION]: The harness uses dynamic execution to run specialized analysis scripts. In scripts/run_eval.py, the precompute_for_capsule function scans data capsules for specific file patterns and executes corresponding Python scripts located in other skill directories (e.g., tooluniverse-phylogenetics/scripts/scogs_paired_compare.py). This allows for the execution of scripts within the local environment based on the structure of the data being processed.
[COMMAND_EXECUTION]: Multiple Python and Bash scripts (scripts/run_eval.py, scripts/grade_answers.py, scripts/run_harness_loop.sh) utilize subprocess.run or direct shell execution to coordinate the benchmark runner, grading, and environment setup. This includes invoking the claude CLI with various flags to simulate agent interactions and running local utility scripts for scientific data analysis.

devtu-benchmark-harness