Web scraping blocked: status codes and protection errors explained
If a scrape suddenly returns an error instead of page content, the code tells you where the problem is likely to sit.
Some responses point to how often you are requesting, some to whether the request is allowed at all, and others to a bot-protection layer in front of the site. This page collects those cases in one place, with a practical checklist for narrowing down the cause before you change your setup.
Start with the code, not a generic workaround
Use the error you actually receive as your starting point. A standard HTTP status code and a protection-provider error can look similar in a log, but they usually require different responses. Work through this short triage before retrying the same request unchanged:
- Note the exact code, the target domain, and whether it appears on every request or only under load.
- Separate rate-related failures from access-related failures and from browser or bot-detection challenges.
- Change one variable at a time, then retest with a small sample so you can tell what helped.
- If the same code persists, open the matching guide below rather than applying a generic workaround.
HTTP status codes
These codes cover the most common HTTP-level blocks encountered during scraping. Each one has its own linked guide explaining what it means and how to avoid it:
- 403 status code - what is it and how to avoid it?
- 429 status code - what is it and how to avoid it?
- 444 status code - what is it and how to avoid it?
- 499 status code - what is it and how to avoid it?
- 503 status code - what is it and how to avoid it?
- 520 status code - what is it and how to avoid it?
Cloudflare errors
When a site sits behind Cloudflare, the error number identifies the specific protection or configuration problem. The guides below cover the errors listed for this topic:
- Cloudflare Error 1009: what is it and how to avoid it?
- Cloudflare Error 1010: what is it and how to avoid it?
- Cloudflare Error 1015: what is it and how to avoid it?
- Cloudflare Error 1020: what is it and how to avoid it?
- Cloudflare Errors 1006, 1007, 1008: how to avoid them?
Human-verification challenges
Some blocks do not return a familiar status code at all. Instead, the page asks you to prove you are human. Perimeter X is one such protection provider, and its verification prompt is covered separately:
- How to scrape Perimeter X: Please verify you are human?
When infrastructure becomes the bottleneck
When blocks keep recurring across sites, the bottleneck is often infrastructure rather than a single setting: managing headless browsers and rotating proxies yourself takes ongoing effort. A managed scraping API can take over that layer. If you are evaluating one, check what the plan actually includes before committing — for example, whether a trial allowance exists, how credits are counted per request, and what happens when a target site escalates its defences. Treat any advertised free allowance as something to confirm on the provider's current pricing page rather than a fixed entitlement.
Frequently asked questions
How should I diagnose a web scraping block?
Start with the exact code or error number, then check whether the block is tied to request rate, access permissions, browser behaviour, or a protection provider such as Cloudflare or Perimeter X. The guides below separate those cases so you can apply the relevant fix instead of treating every block the same way.