520 Status Code: What It Means and How to Stop It Blocking Your Scraper
A 520 response is tied to Cloudflare.
Cloudflare uses it as a catch-all when the origin server sends back something unexpected. The practical question is whether the fault is on the origin side or in your request. This guide separates the two, gives a fix order, and keeps the diagnosis tied to what the response can actually tell you.
What a 520 response actually is
Cloudflare returns a 520 as a catch-all when the origin server sends something unexpected. Because it is a catch-all, the code alone does not tell you which specific failure happened. It only tells you that the exchange between Cloudflare and the origin did not go as expected.
That has a direct consequence for scraping: a 520 can appear even when your request was perfectly valid, because the origin itself is having technical issues. It can also appear when your request is missing something the site expects, and the origin responds in a way Cloudflare cannot interpret. Both paths lead to the same status, so diagnosis has to come from context, not from the code.
Two causes, one status
Treat the possible causes as two branches and test them in order.
Branch A — technical issues on the website. The source notes that a 520 might be caused by some technical issues on the website. In that case the origin is the place to look, and request changes may not be what resolves it.
Branch B — request-side fault. The source also notes that a 520 can be caused if your requests do not contain the required data that the website is expecting. The signal is that a browser request to the same URL succeeds while your client's request fails, and the gap narrows when you add the expected headers.
A quick decision checklist:
- Does the URL load in a normal browser right now? If no → lean toward Branch A.
- Does it fail only from your client? If yes → lean toward Branch B.
- Does it fail intermittently from both? → the origin may be having technical issues.
- Does adding the headers below change the outcome? If yes → Branch B confirmed.
Step one: send the headers the site expects
Before touching tooling, make sure your request carries the data a normal browser would send. The source names three header families worth checking first: CORS, Referrer and Auth. A missing or malformed member of any of these can push the origin into an unexpected response path, which Cloudflare then reports as 520.
A useful field map for the request:
- CORS — the cross-origin header family the source lists as required.
- Referrer — the referrer header the source lists as required.
- Auth — the authentication header family the source lists as required.
If adding these headers resolves the 520, you were in Branch B and you are done. If the response is unchanged, move to step two.
Step two: reduce automation signals
When headers are complete and the origin is healthy, the remaining likely cause is that the server has flagged your client as automated. A headless browser driven by default automation flags is a common trigger. The practical remedy is to use a driver that suppresses those flags — undetected-chromedriver is the option referenced here — and to pair it with proxies from a provider you trust, so requests do not arrive from a single obvious source.
That pairing matters more than either half alone. A stealth driver from one IP still looks like a bot farm; rotating proxies behind a default driver still leak automation flags. Use both, then re-test the same URL that failed.
Frequently asked questions
What does a 520 status code mean?
It is a Cloudflare-issued catch-all response for when the origin server sends something unexpected. The code alone does not identify the specific failure, so you have to look at context to work out the cause.
What should I check first when I get a 520 while scraping?
Check whether the URL loads in a normal browser. If it does, your request is the likely cause — start with the CORS, Referrer and Auth headers. If it does not, the site may be having technical issues and request changes may not help.
Which headers are most likely to matter for a 520?
CORS, Referrer and Auth are the three families the source lists as required. A missing or malformed member of any of them can push the origin into an unexpected response that Cloudflare reports as 520.
What if my headers are complete and I still get 520?
The server may be recognising your client as automated. Use undetected-chromedriver to suppress default automation flags, and pair it with proxies from a reliable provider so requests do not arrive from a single obvious source.