Answer probe

Runs a fixed query set against AI search engines and assistants and scores each answer on three questions: does it name the entity, does it link one of the entity's domains, and does it state each expected fact correctly. The script aieo/probe.py writes one dated JSON file per run. A trend compares runs of the same query set version only, because results from a changed query set are not comparable.

The probe records what one engine answered on one day. The cause of a citation is not in the data.

Inputs

• The entity: the names the pages use, and its domains.
• The facts an answer should state, each with a pattern for a correct statement and, where known, a pattern for a wrong one.
• The engines to probe. Manual mode works for any engine with a chat or search page. The adapter mode needs an engine with an API and a key.

Steps

1. Copy the templates without overwriting: