Blocked scrape triage

A scrape came back wrong. Find out why before changing anything, then escalate in cost order. Most "blocks" are not bot detection, some are not even a live server, and the browser is the most expensive answer rather than the first one.

Work the steps in order. Step 0 ends more investigations than the whole ladder does.

Prerequisites

• Apify account (sign up)
• Authentication via one of:
  • apify login (OAuth, if using the Apify CLI)
  • APIFY_TOKEN environment variable
  • Token from Apify Console → Settings → Integrations
• A one-time permission approval, before the first diagnostic run. apify/cheerio-scraper requires full account access and refuses to start until that is approved in the Console. Both paths refuse, with different wording: MCP call-actor answers requires full access to your account. You must approve its permissions before running it, while the CLI prints Error: Actor apify/cheerio-scraper requires full access to your Apify account and has not been approved yet. on stdout with exit code 0 — so an unattended CLI flow reads it as a normal result and stops without saying why. Approve it once at the URL the message prints, before you need a diagnosis. The same applies to apify/web-scraper and the browser Actors in Rung 5; apify/website-content-crawler does not require it.

What a diagnosis is allowed to cost

One DNS pair, one navigation per egress per host, then stop and conclude. The unit is the host, not the organisation: one county published its records across three hostnames that refused a request in three different ways, and a budget spent on "the target" would have bought one of those three answers. The budget counts requests to hosts the target owns; looking something up on a third-party platform, a data catalogue or your own IP check is free. If you are four requests into one host and still cannot name the cause, you are no longer diagnosing, you are probing, and you should say what you know and what it would cost to learn more. One navigation means one page: a browser pulling forty subresources for that page is still one navigation, and a burst of repeated probes is not. The one exception worth spending on is the residential probe in Step 3, a single run that separates "your network" from "actually defended".