creating-online-evaluations
Pass
Audited by Gen Agent Trust Hub on Sep 10, 2026
Risk Level: SAFEINDIRECT_PROMPT_INJECTIONDYNAMIC_EXECUTIONCOMMAND_EXECUTION
Full Analysis
- [INDIRECT_PROMPT_INJECTION]: The skill is designed to ingest and process data from production traffic ($ai_generation events), which contains untrusted external LLM outputs and user-supplied content. This creates a surface for indirect prompt injection where malicious data could influence evaluation results.
- Ingestion points: Production traces and generation events are accessed and evaluated by Hog scripts or LLM judges as described in SKILL.md and references/evaluation-payload.md.
- Capability inventory: The skill can create persistent evaluation rules via posthog:llma-evaluation-create, execute SQL queries via posthog:execute-sql, and trigger network requests to LLM providers for judging.
- Boundary markers: The instructions emphasize defining clear, crisp criteria for evaluations but do not specify mandatory delimiters or sanitization steps for the untrusted content being evaluated within the scripts or prompts.
- Sanitization: The skill implements a security best practice by recommending deterministic Hog scripts (evaluation_type: hog) where possible, which reduces the risk of the evaluator following instructions embedded in the data compared to LLM-based judges.
- [DYNAMIC_EXECUTION]: The skill involves the runtime generation and execution of Hog source code and SQL queries to implement and verify evaluation logic.
- Script generation: It provides templates for Hog source code that iterate over evaluation_events to apply boolean logic, as shown in the examples in SKILL.md and references/evaluation-payload.md.
- Database interaction: It utilizes posthog:execute-sql to verify the volume and properties of events matching evaluation conditions, ensuring the filters target the intended data.
- [COMMAND_EXECUTION]: The skill uses several PostHog-specific tools to manage the lifecycle of evaluations within the platform.
- Tooling usage: Tools such as posthog:llma-evaluation-create, posthog:llma-evaluation-run, and posthog:llma-provider-key-list are used according to the vendor's intended design for AI observability and management.
Audit Metadata