Agent readiness audit

Audit how well a website serves AI agents by probing its standard discovery surface. Everything here is plain HTTP GETs against well-known paths; no tooling beyond fetch/curl is needed. Run the probes, then report what exists, what is missing, and the fix for each gap.

The probes, in order of importance

For the target domain https://example.com, check each of these and note status code, content type, and whether the content is real or a stub:

1. /llms.txt: the site guide for agents. Good ones are a curated navigation index under ~30k characters with markdown links that resolve. A missing or bloated llms.txt is the single most common gap.
2. /robots.txt: look for explicit AI-crawler sections (GPTBot, ClaudeBot, PerplexityBot, Google-Extended), a Content-Signal: line, and whether user-triggered agents (ChatGPT-User, Claude-User) are allowed. Open-by-default scores worse than a considered, tiered policy.
3. Markdown availability: fetch a content page with Accept: text/markdown, try the page URL plus index.md, and look for rel="alternate" type="text/markdown" in the HTML head. Agents parse markdown far more reliably than a JS-heavy DOM.
4. /openapi.json (also /api, /docs, /developers): a real, reachable API with a spec is the difference between "agents can read about you" and "agents can use you". Check error responses are JSON, not HTML.
5. MCP surface: /.well-known/mcp.json, /.well-known/mcp/server-card.json, and a POST of a JSON-RPC initialize to any advertised endpoint. Also search the official registry: https://registry.modelcontextprotocol.io/v0/servers?search=<domain>.
6. Identity and trust: homepage JSON-LD (Organization with sameAs and contactPoint, plus extended types like Service), /.well-known/ard.json, /.well-known/agent-card.json (A2A), /auth.md.
7. Sitemap hygiene: /sitemap.xml exists, entries carry lastmod, and the newest entry is recent.

Reporting

For each gap, state: what is missing, why an agent trips on it, and the concrete fix (the file to publish or the endpoint to stand up). Rank by impact: a missing API or MCP server outweighs a missing A2A card. Note what already works; an audit that only lists faults misreads sites that are half way there.