creating-online-evaluations
Pass
Audited by Gen Agent Trust Hub on Nov 1, 2026
Risk Level: SAFEINDIRECT_PROMPT_INJECTIONDYNAMIC_EXECUTION
Full Analysis
- [INDIRECT_PROMPT_INJECTION]: The skill is designed to ingest and process untrusted data (AI-generated text and traces) to perform automated evaluations, creating a surface for indirect prompt injection.
- Ingestion points: Untrusted data enters the evaluation context through
$ai_generationevents and execution traces, which are then processed by Hog scripts or LLM judges. - Boundary markers: The instructions do not mandate the use of delimiters or 'ignore' instructions when the agent constructs prompts for the
llm_judgethat include untrusted generation content. - Capability inventory: The skill utilizes
posthog:llma-evaluation-createto deploy logic that triggers LLM calls or executes domain-specific scripts on the ingested data. - Sanitization: No explicit sanitization or escaping of the external AI-generated content is required before it is interpolated into evaluation prompts or scripts.
- [DYNAMIC_EXECUTION]: The skill involves generating and deploying Hog source code, a domain-specific language, to be executed on the PostHog server at runtime.
- Evidence: Phase 2.4 and the
HogEvalConfigsection describe the creation of custom Hog scripts (viaevaluation_config.source) that are executed to score events. While this is a controlled DSL environment, it constitutes dynamic logic generation based on agent-authored templates.
Audit Metadata