The Agent Skills Directory

[COMMAND_EXECUTION]: The evaluation runner in scripts/run_evals.py uses subprocess.run to execute an isolated Python worker process.
Evidence: The script spawns a new process using sys.executable with the -I (isolated) flag to evaluate regular expressions.
Context: This is a security feature designed to prevent Regular Expression Denial of Service (ReDoS) attacks by enforcing a 1-second timeout and resource isolation for potentially complex regex patterns supplied in evaluation specifications.
[DYNAMIC_EXECUTION]: The skill employs runtime execution of a hardcoded Python snippet for isolated data processing.
Evidence: In scripts/run_evals.py, a worker string worker = "import json,re,sys; ..." is executed via subprocess.
Context: The executed code is static and internal to the script. It is used to separate the regex engine from the main process, ensuring that even if a pathological regex is encountered in an eval spec, it cannot hang the main runner.
[INDIRECT_PROMPT_INJECTION]: The skill defines a surface for processing potentially adversarial data during the evaluation of LLM outputs.
Ingestion points: scripts/run_evals.py ingests untrusted data from JSONL dataset files via the --dataset argument.
Boundary markers: The script implements a "fail-closed" contract where it rejects datasets that do not match the sealed sample_manifest_id or that contain holdout rows in optimization mode.
Capability inventory: The runner can perform regex matches, JSON validation, and numeric comparisons. It has no network capabilities and restricted file writing (to the --out path only).
Sanitization: The script includes robust sanitization, such as limiting regex pattern length to 4096 characters, limiting input size to 1,000,000 characters, and using a strict JSON parser that rejects non-standard constants (like NaN or Infinity) and duplicate keys.

ai-evals