curl_cffi for Python Web Scraping: Browser Handshakes, Sessions, and Escalation
curl_cffi is a Python HTTP client built on libcurl through cffi. Its distinctive job is to make a request negotiate TLS and HTTP/2 the way a named browser does, rather than the way stock Python clients do. That changes the part of the connection an anti-bot system reads first. This guide covers installation, the impersonate control, browser targets, sessions, proxies, async use, a complete 403-to-200 example, and the point where a JavaScript challenge needs a different tool.
What curl_cffi changes, and what it does not
A TLS fingerprint is a signature of how a client opens its encrypted connection: which cipher suites it offers, in what order, with which extensions. A JA3 hash is a common way to express that signature. Stock Python clients produce a recognizable fixed signature. curl_cffi can produce Chrome's, Safari's, Firefox's, Edge's, or Tor's instead, and it can also order HTTP/2 frames like the impersonated browser.
That is a network-layer change. It does not execute JavaScript, and it does not record redirect history. Those limits matter later, when a block is a challenge page rather than a handshake gate.
The API is deliberately close to requests. Most migrations change the import line and add one argument. The response object still exposes status_code, text, json(), and headers.
How curl_cffi compares with requests, aiohttp, and httpx
The comparison table below is a narrative summary of client capabilities. The decisive column for protected sites is the TLS fingerprint each client sends. requests, aiohttp, and httpx all send a fixed Python signature. curl_cffi can send a real browser profile.
For sites that do not inspect the handshake, any of these clients is fine. When a target rejects the connection before HTML arrives, curl_cffi is the one to reach for.
Why a stock Python fingerprint gets blocked
Anti-bot systems can read the TLS handshake before a single byte of HTML is sent. A fixed Python fingerprint is a known quantity. The User-Agent header arrives later, so rotating it does not change the first signal the server sees.
A client that claims to be Chrome in its User-Agent but connects over HTTP/1.1 with a Python cipher order is easy to flag. curl_cffi changes the earlier part of the conversation: the cipher order, extensions, and HTTP/2 settings.
Install curl_cffi and make a first impersonated request
Install with the hyphenated package name, then import with the underscore. The mismatch is a common first stumble: a hyphen is not legal in a Python identifier, so import curl-cffi is read as subtraction.
curl_cffi needs Python 3.10 or newer and ships prebuilt binaries, so there is no compiler step on a normal machine. The async client is included in the base package.
The smallest useful request adds impersonate="chrome" to a normal GET. That one argument configures libcurl to offer Chrome's TLS cipher suites in Chrome's order, send Chrome's extensions, and order HTTP/2 frames the way Chrome does. HTTP/3 over QUIC is available by opting in with http_version="v3"; the default is HTTP/2.
Choose a browser target: rolling alias or pinned version
The impersonate argument accepts a rolling alias such as "chrome" or a pinned label such as "chrome131". Both produce a genuine browser handshake, but they are not the same handshake. Pinning is reproducible and can go stale; an alias stays current but can change after a routine package upgrade.
A practical pattern is to use the alias in development and a pin in production when a run must be reproducible, with a reminder to bump the pin when the library is updated. Keeping curl_cffi current matters because profiles are added when browsers change their fingerprints.
For non-browser targets, the named profiles can be skipped. curl_cffi accepts raw ja3=, akamai=, and extra_fp= arguments on the request call. This is an advanced escape hatch for matching a mobile app or API client that does not correspond to a shipping browser.
Sessions, proxies, and async concurrency
A Session keeps cookies, headers, and the impersonation profile across calls. Set impersonate once at construction. Consistency is itself a signal: a real browser does not switch its TLS fingerprint between page one and page two.
A proxies dict routes impersonated traffic through HTTP, HTTPS, or SOCKS5 proxies. The proxy and the fingerprint solve different problems. curl_cffi fixes how the connection looks; a proxy fixes where it comes from. A perfect Chrome fingerprint from a low-trust datacenter IP can still be stopped, so pair impersonation with residential proxies for targets that take blocking seriously.
For a list of URLs, AsyncSession with asyncio.gather runs many impersonated requests concurrently. The async with block closes connections cleanly. Concurrency does not grant permission to exceed the target's tolerance; cap fan-out as you would with any client.
curl_cffi also speaks WebSockets at the impersonated fingerprint and can use HTTP/3 over QUIC with http_version="v3". SOCKS5 proxy support extends to UDP in version 0.15.0, which also added HTTP/3 fingerprinting.
A complete example: from 403 to 200, then parse
The worked example uses a live public classifieds homepage and reads only its public structure: the page title and top-level category names. Before pointing a scraper at any site, read its robots.txt and terms, and keep the request rate polite.
Start with the version almost everyone writes first. A stock requests.get() returned 403 on three separate runs. The URL was valid and the site was serving real users; the request was turned away at the TLS handshake because a python-requests fingerprint is a known quantity.
Changing the import and adding impersonate="chrome" returned 200 with 386,012 bytes of HTML on every run. The only change was that the handshake looked like Chrome's instead of Python's.
A 200 is a means, not an end. Guard the status before parsing, because parsing a block page as if it were data corrupts a dataset quietly. Guard the nodes you pull as well, so a missing title yields None rather than an AttributeError during an overnight run. With curl_cffi 0.15.0 and Python 3.12, the title parsed as leboncoin, site de petites annonces gratuites, and the public category headings parsed as ['Recherches récentes', 'Immobilier', 'Véhicules', 'Matériel pro', 'Mode', 'Maison & Jardin'].
Where curl_cffi stops, and how to escalate
curl_cffi solves one problem completely: TLS fingerprint detection. It does not run JavaScript, and it does not keep redirect history. When the block is a JavaScript challenge rather than a fingerprint gate, curl_cffi has done all it can.
The escalation table below maps a failure signal to the cheapest fix that addresses it. Read the signal, match the rung. The point of ordering the fixes this way is to keep you from spinning up a browser farm for a problem a proxy would have solved: in practice, a large share of "I need a headless browser" conclusions turn out to sit on the first or second rung, where impersonation plus a trustworthy proxy already clears the block. The same ordering keeps you from the opposite mistake, hammering a 403 from one IP when the response was actually a cf_clearance cookie that no fingerprint change can produce.
At the top rung, a real JavaScript challenge needs a browser engine. Running one yourself at scale means a headless Chrome farm, its memory cost, the fingerprinting headless browsers themselves leak, and the upkeep. A web scraping API that runs a browser is the alternative.
Decision checklist before you change tools
Use this checklist to avoid reaching for a heavier tool too early or staying on a lighter one too long.
- Did stock requests fail and impersonate="chrome" succeed? The block was a TLS-fingerprint gate. curl_cffi alone is the fix.
- Does impersonation still fail from a datacenter IP? The problem is IP trust, not the fingerprint. Add residential proxies to the impersonated request.
- Do you see a cf_clearance cookie, a "checking your browser" page, or a Turnstile widget? The block is a JavaScript challenge. Use a real browser or a rendering API.
- Do you need the full redirect chain? curl_cffi follows redirects but does not record their history. Wrap it yourself or use a managed API that includes it.
- Are you sending many parallel requests from one IP? Concurrency at a browser fingerprint is still concurrency. Cap fan-out and slow down.
Field and control map
The controls below are the ones this guide relies on. They are request or session arguments, not a separate configuration file.
- impersonate: a rolling alias such as "chrome" or a pinned label such as "chrome131". Sets the browser profile for the request or session.
- proxies: a dict with http and https entries. SOCKS5 uses a socks5:// URL.
- Session(impersonate=...): keeps cookies, headers, and the profile across calls.
- AsyncSession(impersonate=...): the async equivalent, used with asyncio.gather.
- ja3=, akamai=, extra_fp=: raw fingerprint arguments for non-browser targets.
- http_version="v3": opts into HTTP/3 over QUIC; the default is HTTP/2.
- status_code, text, json(), headers: the response surface shared with requests.
| Client | Sync / async | HTTP/2 | TLS fingerprint it sends |
|---|---|---|---|
| requests | Sync only | No | Fixed Python signature, easily flagged |
| aiohttp | Async only | No | Fixed Python signature, easily flagged |
| httpx | Both | Yes | Fixed Python signature, easily flagged |
| curl_cffi | Both | Yes | A real browser (Chrome, Safari, Firefox, Edge) |
Capability summary; the fingerprint column is the decisive one for protected sites.
| Stock requests | curl_cffi impersonate="chrome" | |
|---|---|---|
| JA3 hash | a48c0d5f95b1ef98f560f324fd275da1 | a real Chrome JA3 (e.g. f8e631a31940996f349d5de95e74f930) |
| Cipher order (start) | 771,4866-4867-4865,... | 771,4865-4866-4867,... (Chrome's order) |
| HTTP/2 fingerprint | None. It connects over HTTP/1.1 | Chrome's HTTP/2 settings fingerprint (52d84b11...) |
| User-Agent on the wire | python-requests/2.34.2 | A real Chrome User-Agent |
Dated reference captured with curl_cffi 0.15.0; fingerprints change with library and browser releases.
| Failure signal | What it means | The fix |
|---|---|---|
| Blocked on stock requests, fixed by impersonate | A TLS-fingerprint gate | curl_cffi alone, nothing more |
| Still blocked from a datacenter IP, even with impersonation | Low IP trust, not a fingerprint problem | Add residential proxies to the impersonated request |
| cf_clearance cookie, "checking your browser", or a Turnstile widget | A JavaScript challenge | A real browser or a web scraping API that runs one |
| You need the full redirect chain or history | curl_cffi follows redirects but does not record their history | Wrap it yourself, or move to a managed API that includes it |
Read in order; match the failure signal to the cheapest fix that addresses it.
Frequently asked questions
What is curl_cffi used for?
It makes HTTP requests that look like a real browser at the network level, which is what gets a Python scraper past TLS fingerprint blocks. It impersonates a browser's TLS and HTTP/2 handshake, not just the User-Agent header. It is also a fast general-purpose HTTP client built on libcurl, but impersonation is its most common use.
How is curl_cffi different from Python requests?
The difference is the TLS handshake. requests sends a fixed fingerprint that anti-bot systems recognize as Python; curl_cffi sends a real browser fingerprint chosen with the impersonate= argument. The API is otherwise close to identical, so from curl_cffi import requests gives you the same .get(), .status_code, and .text you already use.
Does curl_cffi bypass Cloudflare?
For Cloudflare's TLS-fingerprint detection, yes. For its JavaScript challenges, no. curl_cffi sends a browser-grade handshake, which clears the passive fingerprint checks that stop most Python scrapers, but it runs no JavaScript, so it cannot solve a Turnstile widget or a "checking your browser" interstitial. A cf_clearance cookie or a challenge page in the response is the tell that you have hit the JavaScript wall; at that point you need a real browser or a rendering API, not a better fingerprint.
Which browsers can curl_cffi impersonate?
It impersonates recent versions of Chrome, Safari, Firefox, Edge, and Tor, as well as iOS and Android variants, via pinned labels such as chrome131 or rolling aliases such as chrome. The exact list changes with nearly every release because profiles are added when a browser's real fingerprint changes, so the supported-targets page in the official docs is the reliable source.
How do I use curl_cffi asynchronously?
Reach for AsyncSession together with asyncio.gather when you have many URLs to fetch at once. The async client is part of the base package, so the same pip install curl-cffi covers it with no extra to add. Open the session as async with AsyncSession(impersonate="chrome") as session:, then gather the session.get() coroutines. Each request keeps the same browser fingerprint, and leaving the async with block closes the connections so a long-running scraper does not exhaust file descriptors.
Why is curl_cffi still getting blocked?
If curl_cffi is still blocked, the block is no longer about your TLS fingerprint, so a different fingerprint will not help. The three usual causes are a low-trust datacenter IP (fix with residential proxies), a JavaScript challenge that needs a browser engine curl_cffi does not have (fix with a rendering API or a real browser), or a request rate aggressive enough to look automated regardless of fingerprint (slow down). Match the cause to the fix.
Is web scraping with curl_cffi legal?
The library itself is legal; it is an HTTP client. Whether a given scrape is legal depends on the site's terms of service, its robots.txt, the nature of the data, and the laws of your jurisdiction, not on the tool you used. As a general rule, scraping public data is defensible, while scraping behind a login or collecting personal data is far riskier.
pip install curl-cffi