Skip to content

Web Scraping Without Getting Blocked: A Layer-by-Layer Guide

Modern anti-bot stacks do not rely on a single check. They score the IP, the TLS handshake, the HTTP/2 parameters, the browser fingerprint, and the session's behavior, then decide whether to serve the page, throw a challenge, or return an error. The practical consequence is that a fix applied to one layer often fails because a different layer is still flagging the request. This guide walks through each layer, shows how to read the block you are getting, and helps you choose the smallest change that actually solves the problem.

Read the block before you fix it

The type of failure you see is a strong hint about which layer rejected you. Use it as a starting diagnostic rather than guessing.

  • Instant 403 on the very first request. The request was rejected before any page content loaded. The usual suspects are request headers and the TLS fingerprint. A browser-impersonating client is often enough here.
  • Page renders in the browser but fails from a Python requests script. This commonly points to a TLS fingerprint mismatch: the handshake does not resemble a real browser's.
  • Blocks that appear mid-session. These usually indicate rate-limiting or IP reputation problems rather than a fingerprint issue.
  • Endless CAPTCHA loop. The combined signals have pushed the trust score negative. Solving the puzzle does not clear the underlying signals.

A useful rule: fix the earliest layer that could be rejecting you first, then re-test. Changing user agents when the TLS handshake is the problem wastes time.

CAPTCHA challenges have also moved beyond the classic image grid. Modern systems include Cloudflare Turnstile, which runs invisibly in the background, and reCAPTCHA v3's background 0-1 scoring. These are effectively impossible to solve manually, so avoidance is the better strategy. When a challenge is unavoidable, basic visual CAPTCHAs can sometimes be handled with OCR, while services such as 2Captcha and Death by Captcha use real people to solve harder ones for a fee. Solving a challenge does not erase the signals that triggered it, so expect repeat challenges if the session still looks suspicious.

As reported in the captured 2026 guide, Imperva's 2025 Bad Bot report put automated traffic at 51% of all web activity — a dated third-party figure that helps explain why sites now score requests across multiple layers rather than relying on a single check.

The five detection layers, in order

Anti-bot providers implement these differently, but the core layers are broadly consistent:

  1. IP and ASN reputation. Datacenter ranges, known proxy ranges, and addresses with a history of bot activity are flagged early.
  2. TLS fingerprinting (JA3 and JA4). TLS parameters exposed during the handshake are hashed. Real browsers and HTTP clients produce distinct fingerprints, so bots can be identified before any HTML is served. JA3 is the most common format; JA4 is its successor and handles TLS 1.3 and QUIC. As reported in the captured 2026 guide, Akamai rolled out JA4 fingerprinting in 2026, having pioneered commercial TLS fingerprinting years earlier.
  3. HTTP/2 fingerprinting. The order of HTTP/2 parameters carries a signature that is easy to compare against a real browser. It is often paired with TLS fingerprinting and analyzes protocol-level parameters such as SETTINGS frames and header order.
  4. Browser fingerprinting. Dozens of client-side properties — canvas and WebGL rendering, fonts, screen dimensions, plugin lists — are correlated. The navigator.webdriver property is a classic automation marker; in real browsers it often has an undefined value, and setting it to false is itself a flag. Mismatched attributes are another red flag: a client that advertises a macOS user agent but exposes Windows-only fonts, or one that pairs a mobile viewport with desktop-class hardware, tends to be scored as automated.
  5. Behavioral analysis. Mouse movement, scroll patterns, click accuracy, and navigation sequence are tracked across the session.

Hard targets run several of these at once. Passing one layer only moves you to the next.

Vendor behavior varies. As reported in the captured 2026 guide, Cloudflare has shifted toward an economic model: since July 2025 it has blocked AI crawlers by default on new domains, its Pay Per Crawl program returns an HTTP 402 (Payment Required) response to crawlers that have not paid for access, and it runs AI Labyrinth, a honeypot that serves fabricated content to bots it catches, alongside standard Turnstile challenges. DataDome leans almost entirely on machine learning, running hundreds of foundational models plus more than 85,000 customer-specific models, and is especially common on European sites partly due to its emphasis on EU data compliance. Akamai has invested more heavily in TLS-layer detection than other major vendors. These are dated vendor-behavior observations from the captured guide; verify current behavior before relying on them.

Match the fix to the layer

Use this checklist to choose a proportionate response. Start with the cheapest change that addresses the layer you believe is failing.

  • IP layer. Rotate IPs and prefer residential or mobile proxies for defended targets. Datacenter proxies are fine for open sites but are commonly blocked on protected ones because they rarely carry human traffic. {{OWNER_PROXY_PRICING}}
  • TLS layer only. A browser-impersonating client such as curl_cffi (or curl-impersonate) can replicate a real browser's TLS and HTTP/2 fingerprint; these tools have supported HTTP/3 since v0.15. They have no JavaScript engine, so they will not pass browser fingerprinting or JS challenges.
  • Browser fingerprint layer. Use a stealth browser. nodriver is the official successor to undetected-chromedriver, developed by the same author; it communicates directly with Chrome via CDP and masks the Runtime.Enable leak. Camoufox is a hardened Firefox fork with built-in fingerprint spoofing, stealth patches, and proxy support; it is fully compatible with the Playwright API, so transitioning existing Playwright scripts takes minimal effort, and it is one of the more effective options for bypassing fingerprinting tools like CreepJS. Note that undetected-chromedriver itself is now a legacy tool that modern anti-bot systems detect reliably, because it never plugs the Runtime.Enable CDP leak.
  • Behavior layer. Randomize request order and timing, vary user agents, and occasionally follow links or linger on pages.
  • All layers at once. A managed scraping API handles proxies, rendering, and fingerprint management behind a single call.

If you are unsure which layer is failing, test with a TLS-impersonating client first. If that still fails, the target is likely also checking the browser fingerprint or requiring JavaScript.

Headers are part of the fingerprint too. Beyond the User-Agent, send a plausible Referrer (the previous page a link was followed from), an Accept-Language matching the target locale, and Client Hints such as sec-ch-ua and sec-ch-ua-platform that are consistent with your User-Agent string. For generating User-Agent strings, libraries such as Fake-Useragent (Python), User Agents (JS), Faker (Ruby), and Fake User Agent (Rust) can help.

When a site exposes an internal API, scraping it can be simpler than scraping HTML. Watch the network tab while interacting — for example, clicking a "Load more comments" button often fires an XHR that returns JSON. Replicating that request requires matching its headers and parameters, and some APIs use single-use tokens to prevent simple replays. Exporting requests as a HAR file lets you inspect them in tools like Paw or Postman. Mobile apps hide their requests more effectively; intercepting them often requires a MITM proxy such as Charles. Two illustrative cases from the captured guide: Pokémon Go's Niantic banned players whose cheat tools overlooked hidden API parameters, and Starbucks' unofficial API employed device fingerprinting, encryption, and single-use requests.

Open-source headless browsers: what each one still gets through

The table below is a dated reference from the captured 2026 guide. It reflects the state of each tool at that time; check current project status before relying on it. The strongest open-source option noted was Camoufox, while Playwright and Puppeteer without stealth patches were described as detected at the fingerprint layer. Cloudscraper, which is not a real browser, was described as failing against modern bot management; for Cloudflare-protected sites, curl_cffi or a managed scraping API was presented as a better alternative.

A worked pacing example

Suppose you need product data from a URL pattern like https://www.myshop.com/product/[ID] for IDs 1 through 10,000. Walking the IDs at a constant one-second interval is an obvious scraping signature. A more human-like plan:

  • Shuffle the ID order so access is not sequential.
  • Insert random delays ranging from a few seconds to several minutes.
  • Rotate user agents across requests.
  • Occasionally follow an unrelated link or pause as if reading.

This does not defeat a fingerprint check, but it reduces the behavioral signal that triggers rate limits and CAPTCHAs. If you are driving a real browser, Selenium's mouse-interaction API can also simulate cursor movement, drag-and-drop, and complex click sequences, which helps with pages that require hover states or mouse-triggered JavaScript.

Respect the server and its rate limits

Bombarding a server degrades service for others and increases the chance your scraper is blocked. Practical steps:

  • Read the site's terms of service and API documentation; rate limits are often stated there.
  • Watch response headers such as RateLimit-Limit, RateLimit-Remaining, and RateLimit-Reset.
  • On HTTP 429, back off progressively rather than retrying immediately.
  • Monitor latency; a sudden increase suggests the server is struggling.
  • Prefer off-peak hours, starting just after midnight in the server's local time.

If you are unsure of acceptable rates, contact the site administrators directly.

Location, honeypots, and other traps

Geography matters. A site serving a local audience may treat out-of-region IPs as suspicious, so use proxies that match the site's primary user base. The captured guide's example: a Brazilian food delivery service mainly serving customers within Brazil would likely flag scraping attempts from U.S. or Vietnam proxies. A documented case involved a company blocking an entire country's IP ranges after detecting heavy scraping from there.

Honeypots are links invisible to humans but detectable by bots. Common patterns include CSS display: none or visibility: hidden, off-screen positioning such as position: absolute; left: -9999px, and links camouflaged against the background (for example, color: #ffffff on a white background). Avoid following links that are irrelevant to your objective, respect robots.txt, and validate collected data against trusted sources because some honeypots serve false information.

Tor is an anonymity tool, not a practical evasion method: exit nodes are public and commonly blocklisted, and relay routing is slow.

When to use a managed scraping API

If maintaining proxies, stealth browsers, and fingerprint patches is not the core of your project, a managed API can collapse the layers into one request. The captured guide describes a service that combines premium and stealth proxies, JavaScript rendering, and fingerprint management behind a single call, returning HTML or JSON. Whether that trade-off is right for you depends on volume, budget, and how much control you need over the browser session.

Before choosing a plan, confirm the current pricing, included usage, trial conditions, billing schedule and cancellation terms.

Before choosing a plan, confirm the current pricing, included usage, trial conditions, billing schedule and cancellation terms.

Open-source headless browsers and the protection tiers they still get through (2026 reference)
ToolEngineProtection tier it still gets through
NodriverChromium, via direct CDP (patched)Fingerprinting and basic JS challenges; struggles with forced, interactive Turnstile challenges
CamoufoxFirefox (hardened fork)Fingerprinting, JS challenges, and moderate behavioral checks. The strongest open-source option in 2026
PlaywrightChromium, Firefox, WebKitBasic JS rendering only. Detected at the fingerprint layer without added stealth patches
PuppeteerChromium/ChromeBasic JS rendering only. The stealth-plugin ecosystem has been largely unmaintained since early 2025
SeleniumChrome, Firefox, Edge, SafariDetected at the browser fingerprinting layer without stealth patches
CloudscraperNo real browser: HTTP client + JS interpreterLegacy/basic JS challenges only. Fails against modern bot management, Turnstile, and fingerprinting checks

Dated 2026 reference from the captured guide; tool capabilities change frequently — verify current project status.

Frequently asked questions

Why does my scraper get a 403 when the page works in my browser?

Most often it is a TLS fingerprint mismatch. Python requests and similar HTTP clients send a TLS and header signature that no real browser produces, so the request is rejected before any HTML is served. If the site relies only on network-layer fingerprinting (TLS and HTTP/2), a browser-impersonating client like curl_cffi can get through. If it also runs browser fingerprinting or requires JavaScript, you will need a stealth browser paired with residential proxies.

Can websites detect Selenium and Playwright?

Yes. Vanilla Selenium and Playwright leak automation signals such as navigator.webdriver, missing browser APIs, and the CDP Runtime.Enable calls that anti-bot systems flag instantly. Fortified versions such as nodriver, SeleniumBase UC Mode, Patchright, or Camoufox patch these leaks and help you survive longer.

Do I need residential proxies or are datacenter proxies enough?

Use residential proxies for reliable scraping of defended targets. Datacenter proxies work on unprotected sites but are commonly blocked on protected ones because they rarely carry human traffic. Residential proxies appear to be real ISP users and pass reputation checks. Match the proxy type to the target: datacenter for open sites, residential or mobile for protected ones.

How do I check if a website allows scraping?

Read the site's robots.txt for disallowed paths and crawl-delay rules, check the Terms of Service for scraping clauses, and look for an official API. If an API exists, use it. Scraping publicly available data is generally lawful, but logged-in areas and personal data are subject to much stricter rules.

How do I bypass Cloudflare when scraping?

You need every layer right: browser-grade TLS and browser fingerprints, residential IPs, and consistent sessions. Cloudflare checks TLS and HTTP/2 fingerprints, runs JavaScript challenges, scores IP reputation, and serves Turnstile when in doubt. A managed scraping API can handle all of this in one call.

Why do CAPTCHAs keep appearing even after I solve them?

Because CAPTCHAs are a symptom, not the root problem. If your IP, fingerprint, or behavior still scores as automated, the site keeps challenging you regardless of how many puzzles you pass. Fix the upstream signals — a clean residential IP, consistent cookies, and human-like pacing — and most CAPTCHAs stop appearing.

curl -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36" \
https://example.com
curl -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36" \
https://example.com
curl \
  -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36" \
  -H "Referer: https://www.google.com/" \
  https://example.com