agento11y
Agent Observability
Agent Observability records what LLM-powered applications do in production and scores the quality of their output.
Applications send generations (individual LLM API calls — request, response, model, tokens, tool calls) to Agent Observability. Generations belonging to the same user session are grouped into a conversation.
Evaluators are scoring functions (LLM judge, regex, heuristic, JSON schema, etc.) that assess generation quality. Rules bind evaluators to production traffic, they select which generations to evaluate (e.g. only user-visible turns), filter by agent/model, and control sampling rate. When a rule matches a generation, Agent Observability runs the bound evaluators and writes scores.
All commands live under gcx agento11y. Use gcx agento11y <subcommand> --help for flags and usage.