qa-generating-e2e-tests

Fail

Audited by Gen Agent Trust Hub on Apr 20, 2026

Risk Level: CRITICALREMOTE_CODE_EXECUTIONCOMMAND_EXECUTIONPROMPT_INJECTION
Full Analysis
  • [REMOTE_CODE_EXECUTION]: The script scripts/generate_playwright.py creates executable Python code by directly interpolating journey definition data into string templates without sanitization.\n
  • Evidence: In generate_playwright.py, the generate_step_code and generate_test_file functions use f-strings to insert variables like value, target, and base_url into the generated Python script (e.g., lines.append(f' await {locator}.fill("{value}")')).\n
  • Risk: Maliciously crafted input in the journeys.json file (which could be derived from untrusted requirements or application discovery) can escape the string literals and inject arbitrary Python code into the generated test files.\n- [COMMAND_EXECUTION]: The skill facilitates the execution of the generated Python scripts through the run_e2e.py script as described in the workflow.\n
  • Evidence: The SKILL.md workflow encourages running the generated tests immediately after generation (python scripts/run_e2e.py --tests tests/ ...).\n
  • Risk: Since the generation phase is vulnerable to code injection, the subsequent execution phase provides a direct path for an attacker to achieve command execution on the host system.\n- [PROMPT_INJECTION]: The skill processes potentially untrusted data from external sources to define user journeys, creating an attack surface for indirect injection.\n
  • Ingestion points: Data enters via journey_from_requirements.py (which parses Jira tickets or requirements files) and discover_journeys.py (which crawls a target URL).\n
  • Boundary markers: None identified in the provided code; input is treated as trusted data during the code generation process.\n
  • Capability inventory: The skill has the ability to write files (generate_playwright.py using Path.write_text) and execute shell commands as shown in the workflow.\n
  • Sanitization: There is no evidence of input validation, escaping, or sanitization of the data retrieved from external sources before it is used in code generation.
Recommendations
  • AI detected serious security threats
Audit Metadata
Risk Level
CRITICAL
Analyzed
Apr 20, 2026, 03:29 AM
Security Audit — agent-trust-hub — qa-generating-e2e-tests