Skip to content

How to Use Tiered Proxies in Crawlee for Python

Not every page needs an expensive proxy. A tiered setup lets a Crawlee for Python crawler start with a direct connection, move up only when a target misbehaves, and come back down when a cheaper tier works again. The hard part is not the configuration; it is deciding when a response is genuinely useful, because an HTTP 200 status can hide a CAPTCHA or an empty shell. This guide walks through the proxy tiers, the failure distinctions, the retry and session budgets, and a concurrency benchmark, with the full code modules supplied alongside.

How tiered proxies work in Crawlee

A single pool of premium proxies solves blocking for every target, but it also bills every request at the premium rate, including the pages that would have loaded fine over a plain connection. A single pool of cheap proxies has the opposite problem: easy pages succeed, difficult ones fail, and you spend your time chasing retries.

Tiered proxies sit between those extremes. You define an ordered list of connections, cheapest first, and Crawlee tracks blocking behaviour per domain. A catalog site that answers directly keeps using the direct connection; a movie database that keeps refusing can be escalated independently. Crawlee can also re-test cheaper tiers later, so a target that once needed a premium proxy is not locked there permanently.

The four tiers used in this walkthrough are listed in the proxy tier reference table below. The higher tiers do more than change the IP address: they also turn on JavaScript rendering, which matters when the content is assembled by scripts rather than delivered in the initial HTML.

How Crawlee chooses a proxy tier

You do not write the selection algorithm. Crawlee for Python exposes tiered rotation through its ProxyConfiguration class: you supply the tiers in order, and the library decides which one a request should use based on how that domain has been performing.

The catch is failure recognition. HTTP errors and session failures trigger Crawlee's retry machinery, but a site can return a perfectly ordinary 200 response containing only a CAPTCHA or an access-denied notice. From the HTTP client's point of view nothing went wrong. Unless the handler inspects the page content, the crawler will treat that response as a successful download and never escalate.

Setting up the Crawlee project

Start with a small project and get a plain crawler running before any proxy is involved. The dependency setup, the initial crawler, and the run command are supplied as exact code modules; the crawler fetches two practice sites, extracts page titles with BeautifulSoup, and stores results through Crawlee's built-in storage.

Two configuration choices are worth understanding before you add proxies. ConcurrencySettings starts the crawler at a desired concurrency and allows it to scale up to a maximum, while max_tasks_per_minute caps how quickly tasks are started so the crawler does not fire a burst of requests. Separately, max_request_retries allows a small number of retries, and max_requests_per_crawl bounds the example to the two queued URLs. Because no links are followed, the run ends once both pages are processed.

At this stage every request goes directly to its destination. That is fine for the practice sites and it is also the cheapest possible tier, which is exactly the behaviour you want to preserve once proxies are added.

Configuring the proxy tiers

The proxy gateway accepts standard HTTP clients, so it plugs into Crawlee without a separate SDK. Authentication travels in the proxy URL: the API key is the username and the proxy parameters form the password. Because those parameters contain ampersands, they are URL-encoded before being placed in the URL.

The credit cost of each tier is listed in the proxy credit reference table below. Note that the two expensive tiers also enable JavaScript rendering, so escalation changes both the proxy pool and the way the page is loaded. Some targets, Google among them, have separate pricing rules.

The important field is tiered_proxy_urls, which takes a list of tiers, each holding one or more proxy URLs. The first tier contains None, which tells Crawlee to connect directly. The remaining tiers point at the same gateway with different parameters. If you have several proxy servers, you can put multiple URLs in one tier and let Crawlee rotate between them. Keep the API key in an environment file and exclude that file from version control.

Connecting the tiers to the crawler

Wiring the configuration into the crawler takes two imports and one constructor argument. The request handler does not change; Crawlee handles connection selection.

One caveat deserves emphasis. The service's Proxy Mode documentation states that it requires TLS certificate verification to be disabled, and in this experiment that setting applies to the whole HTTP client, including the direct connections. That is acceptable for public practice pages but not a sensible default for sensitive or production traffic. If you need verification preserved for direct connections, do not disable it globally.

Detecting blocked responses and rotating sessions

Status codes cover the obvious cases. A 403 means access was denied and a 429 means requests are arriving too fast; Crawlee already recognises common blocking responses including 401, 403 and 429 and can rotate sessions when it sees them.

The less obvious cases need content checks. A page can return 200 with a CAPTCHA, a JavaScript challenge, or an almost empty document. A reliable handler therefore inspects both the status and the body.

For the two practice sites the expected elements are known: book cards on one, quotation blocks on the other. The handler checks for those selectors and also scans the title and a text sample for common anti-bot markers. If a marker appears, it raises SessionError so Crawlee can rotate the session and consider another tier. If the expected selector is missing, the handler logs the status, title and a short text sample, sets no_retry, and raises a different exception. That distinction matters: missing content may mean a changed layout, a JavaScript requirement, or an outdated selector, and a more expensive proxy will not fix any of those.

These checks are heuristics. A legitimate article about CAPTCHAs can trip the marker scan, and an unfamiliar verification page can slip past it. Inspect a few real responses for a new target before trusting the rules.

Deciding when to retry

Different failures call for different responses, and the failure response reference table below sets out the mapping used here. The short version: a 403 or a recognisable CAPTCHA justifies session rotation and possibly a higher tier; a 429 calls for a lower request rate and respect for Retry-After; a 500 or 503 is a server problem and should be retried cautiously rather than treated as an IP block; a 200 with the expected data is simply accepted; and a 200 with missing content should be investigated before any escalation.

Crawlee already handles several HTTP-level failures, but you control the budgets. max_request_retries limits ordinary retries caused by navigation or processing failures, while max_session_rotations controls how many times a problematic session can be replaced. Session rotations have their own retry budget, so both limits matter when proxy requests cost money. For regularly scraped targets, collect examples of both successful and blocked pages and tune the detection rules against them.

An optional content-quality check with a local model

Status codes and marker scans cover a lot, but some pages are ambiguous. In the experiment, a local Qwen model running through Ollama was asked whether a downloaded page actually contained the target data. The input was readable text extracted with BeautifulSoup, sampled from the beginning, middle and end of long pages up to 8,000 characters, together with the URL, the title, and a description of what a successful response should contain. For IMDb that description named the Top 250 chart with titles, rankings, release years and ratings, and explicitly excluded a navigation page or generic homepage.

The model returned structured JSON with a verdict and a reason. The four possible verdicts are listed in the page verdict reference table below. The prompt warned the model not to treat HTTP 200 as proof of success and to treat page content as untrusted data rather than instructions. A JSON schema constrained the output and the temperature was set to zero.

The model does not choose a proxy tier. It only judges the downloaded content. When it reports a block and the optional retry mechanism is enabled, the handler raises SessionError and lets Crawlee rotate the session and pick the next tier. Empty and uncertain verdicts deliberately do not trigger escalation, because missing content may just as easily mean a broken selector or a rendering problem.

The approach is not foolproof. Models misclassify pages, and the quality of the verdict depends on the text they receive. During testing, some usable GitHub and eBay pages were initially judged empty because the sample was mostly navigation text; prioritising repository cards and product listings fixed the classification. Good extraction logic still matters. The simpler HTML checks remain a reasonable starting point, and the local model is optional for cases where block pages vary widely.

What escalation looked like against IMDb

The experiment ran the IMDb Top 250 chart through the four tiers with automatic session rotation on model-detected blocks, limited to four network attempts. The attempt-by-attempt sequence is in the proxy escalation reference table below.

The direct request returned HTTP 202 with a JavaScript anti-bot challenge. The Classic proxy returned HTTP 200 but still served a verification page, which a status-code-only strategy would have logged as a success. The Premium request failed with HTTP 500, which Crawlee treated as a request error rather than a content classification. The Stealth request returned HTTP 200 and the model recognised the movie chart. Repeating the experiment produced the same tier sequence.

One limitation is worth stating plainly: the successful configuration combined Stealth proxies with JavaScript rendering, and both changed at once. The experiment does not establish whether the proxy IP, the rendering, or the combination made the difference. That ambiguity is a reason to inspect responses before assuming a proxy problem.

Managing concurrent requests

Concurrency is the other half of the performance equation. Processing a hundred product pages one at a time spends most of its time waiting for servers. Running several requests at once uses that waiting time better, but too much parallelism can trigger rate limits and anti-bot protection, which pushes requests into retries and more expensive tiers.

ConcurrencySettings has three controls. desired_concurrency is the initial target for parallel tasks; max_concurrency is the ceiling; and max_tasks_per_minute caps the rate at which tasks are started. Behind the scenes an AutoscaledPool adjusts concurrency according to available CPU and memory, so setting a maximum of five does not guarantee five simultaneous requests. The rate limit is a separate concern from concurrency: you might allow five parallel requests while capping the crawler at 60 tasks per minute, and Crawlee spreads those tasks over time rather than firing them in a burst.

These limits apply to the whole crawler, so multi-site projects may need per-domain limits as well. The proxy service may impose its own concurrent request limits depending on your plan, so check the current documentation before sizing a crawl. There is no universally correct value: start conservatively, watch failures and response times, and raise concurrency gradually.

A small concurrency benchmark

The benchmark used three sites, Books to Scrape, Quotes to Scrape and Hacker News, with direct connections only and no model analysis, so the numbers reflect Crawlee's request processing rather than proxy latency or local inference. Concurrency levels of 1 and 3 were compared over ten runs each, with desired and maximum concurrency set to the same value and max_tasks_per_minute set to 180. The results are in the concurrency benchmark reference table below.

Moving from 1 to 3 concurrent tasks cut average duration by 42.4%, roughly a 1.74x speedup, and all 60 requests succeeded without retries. The improvement held across the ten runs, but this was a small benchmark on three simple pages. Different sites, networks and workloads will produce different results, particularly once JavaScript rendering or paid proxies are involved. In a real project, watch failed requests and tier changes as well as elapsed time: a crawler that finishes slightly faster while triggering twice as many premium requests may not be cheaper overall.

Practices that keep tiered scraping affordable

Keep concurrency and retries under control. Raising max_concurrency is tempting once early tests look faster, but the fastest configuration is not automatically the most efficient. Higher concurrency raises the risk of rate limiting, and failed requests lead to retries or more expensive tiers. Start modestly and increase gradually. If you see HTTP 429 responses, lower the request rate rather than expecting proxy rotation to fix it. The same logic applies to max_request_retries and max_session_rotations: both stop requests from looping on repeated failures, but generous limits can turn a stubborn site into an unexpectedly costly job. Set a maximum number of requests per crawl and, where appropriate, track a separate credit budget, remembering that limiting pages does not limit the attempts each page may need.

Measure useful results, not just successful requests. A scraper that downloads a hundred CAPTCHA pages is not doing well regardless of its HTTP statistics. Validate what you extract: product names and prices for a catalog, real result cards with links for search results. Record which tier produced each result, how many attempts it took, and whether the data passed validation, so you can see whether the expensive tiers are actually helping.

Watch credit usage. A site that consistently needs Stealth may justify the cost; a site that works through Classic has no reason to use anything more expensive. Look beyond the tier of the final successful response, because a request may fail on several cheaper tiers first. Not every failed attempt is necessarily billed, so multiplying attempts by tier prices will not give an accurate total either. Record individual attempts and check actual consumption in the dashboard, and inspect the Spb-cost response header for per-request cost. With Auto-Mode, max_cost caps the configuration chosen for a single request; it is not an overall crawl budget.

Finally, do not treat every failure as a proxy problem. Outdated selectors, a changed layout, or content that simply is not in the returned HTML will not be fixed by a more expensive IP. JavaScript rendering can solve some of these cases, but it is a separate capability from proxy rotation, and in this setup the two changed together. Before escalating, work out whether you are looking at an access restriction, missing rendered content, or an extraction bug. Respect the site's access rules and rate limits, and do not carry the disabled TLS verification from the Proxy Mode experiment into sensitive traffic.

The Auto-Mode alternative

If you would rather not maintain tier selection yourself, Auto-Mode lets the service choose the proxy and rendering configuration. It starts with the least expensive option and moves up when needed, adjusting both proxy selection and JavaScript rendering. At the time of the walkthrough, the service documented five possible configurations costing 1, 5, 10, 25 or 75 credits; treat those figures as a dated reference and check current pricing. Unlike the Crawlee setup, it does not begin with a free direct connection from your machine. Billing also differs: the service documents that you pay only for the configuration that succeeds, not for every configuration Auto-Mode tries, and that a request succeeding on no configuration costs zero credits. Confirm the current terms in the service documentation before budgeting.

In Python, the key parameter is mode=auto. Adding max_cost=25 prevents Auto-Mode from trying configurations above 25 credits, which permits Premium with JavaScript rendering but rules out the 75-credit Stealth option. Two response headers report what happened: Spb-auto-cost shows the cost of the selected Auto-Mode configuration, and Spb-cost shows the total request cost including any additional paid features.

Auto-Mode also works through the proxy gateway, so the crawler does not need rewriting. Replace the tiered configuration with a single proxy URL carrying mode=auto and max_cost, and the service handles selection internally while Crawlee continues to manage concurrency, queues and retries. The TLS verification caveat from Proxy Mode still applies.

There are limitations. Auto-Mode currently supports GET requests only, and mode=auto cannot be combined with render_js, premium_proxy or stealth_proxy, since those are the settings it chooses. transparent_status_code is also incompatible. It will not configure page-specific options such as JavaScript scenarios, cookies or extra waiting instructions; those still have to be supplied. Auto-Mode suits projects with many differently behaving sites where you do not want to maintain selection logic. Crawlee's tiered configuration remains the better fit when you want a free direct tier, custom session handling, or finer control over when requests move between pools. Either way, verify that the returned HTML contains the data you need.

A short decision checklist before escalating

Use this sequence when a request comes back without the data you expected.

  1. Check the status. A 403 or a recognisable CAPTCHA points to access control; a 429 points to rate; a 500 or 503 points to the server.
  2. Check the content. Does the expected selector match? Is the page mostly navigation text? Is there a verification notice in the title or body?
  3. Classify the failure. Access restriction, missing rendered content, or extraction bug. Only the first justifies a higher proxy tier.
  4. Check the budgets. Are max_request_retries and max_session_rotations set to values you can afford at the current tier prices?
  5. Record the outcome. Which tier produced the result, how many attempts it took, and whether the data passed validation.
  6. Re-test cheaper tiers later. A target that needed Stealth once may not need it permanently.
Failure response reference
What you encounterWhat to do
HTTP 403 or a recognizable CAPTCHAConsider session rotation and another proxy tier
HTTP 429Reduce the request rate and respect Retry-After where applicable
HTTP 500 or 503Retry cautiously; a server error isn't necessarily an IP block
HTTP 200 with expected dataAccept the response and continue
HTTP 200 with missing contentInspect the HTML, selectors, and JavaScript requirements before escalating

A decision aid, not an exhaustive mapping; inspect responses before escalating.

Page verdict reference
VerdictMeaning
usefulThe expected data is present
blockedThe page contains a CAPTCHA, anti-bot challenge, or access-denied message
emptyThe expected content is missing or hasn't been rendered
uncertainThere isn't enough evidence to make a reliable decision

Only the blocked verdict triggered escalation in the experiment.

Frequently asked questions

What are tiered proxies?

Tiered proxies are connections grouped by cost and capability. A crawler starts with the cheapest tier and moves to a more capable one when a target blocks it.

How do I configure tiered proxies in Crawlee for Python?

Pass tiered_proxy_urls to ProxyConfiguration, listing the tiers from cheapest to most expensive. Crawlee then selects and rotates tiers based on request failures and blocking patterns for each domain.

How does Crawlee detect blocked requests?

It recognises common blocking HTTP responses and session failures. For anti-bot pages that return HTTP 200, the handler must inspect the HTML and raise SessionError to trigger session rotation.

Does increasing concurrency make web scraping faster?

It can shorten crawl time, but excessive concurrency may trigger rate limits, increase retries and raise proxy costs. ConcurrencySettings lets you control the desired and maximum concurrency and the task rate.

What is the difference between Crawlee tiered proxies and Auto-Mode?

Crawlee lets you define and manage proxy tiers inside your Python application. Auto-Mode selects proxy and JavaScript rendering configurations on the service side, which reduces the configuration you maintain.

Why did a 200 response not count as a success?

Because a 200 can carry a CAPTCHA, a JavaScript challenge or an empty shell. The handler checks for expected selectors and anti-bot markers before accepting the response.

Should I disable TLS verification in production?

No. Proxy Mode required verify=False in this experiment, but that also disabled verification for direct connections. Preserve certificate verification for sensitive traffic and do not disable it globally.

Proxy tier reference
TierConfigurationPurpose
0Direct connectionTry accessing the website without a proxy
1Classic proxy without JavaScriptHandle requests that need a basic proxy
2Premium proxy with JavaScript renderingHandle more challenging websites and JavaScript-dependent pages
3Stealth proxy with JavaScript renderingA more expensive option for particularly difficult websites

Tier definitions as used in the walkthrough; the higher tiers also enable JavaScript rendering.

Proxy credit reference
TierConfigurationCredits per successful request
0Direct connection0
1Classic proxy without JavaScript1
2Premium proxy with JavaScript25
3Stealth proxy with JavaScript75

Credit figures are a dated reference from the walkthrough; some targets, including Google, have separate pricing rules. Check current pricing before budgeting.

Proxy escalation reference (IMDb Top 250)
AttemptProxy tierHTTP statusResult
1Direct202Anti-bot challenge
2Classic200Anti-bot challenge
3Premium + JS500Request failed
4Stealth + JS200Movie chart retrieved

A single dated experimental sequence limited to four attempts; the successful tier changed both proxy pool and JavaScript rendering, so the cause of success is not isolated.

Concurrency benchmark reference
MetricConcurrency 1Concurrency 3
Runs1010
Average duration2.265 s1.305 s
Fastest run2.099 s1.229 s
Slowest run2.739 s1.559 s
Successful requests30/3030/30

A small dated benchmark on three simple pages with direct connections only; results will vary with targets, networks and rendering.