Crawlee for Python Tutorial: HTTP, Playwright, and Proxy Examples
Crawlee is an open-source Python web scraping library built by the Apify team. It bundles HTTP scraping, headless browser automation, link queuing, and data storage into one structured crawler framework. This guide walks through installation, a basic BeautifulSoup crawler, a Playwright browser crawler, and proxy management with sessions and tiers. Each section pairs the exact code module with a short explanation of what the code does and when to use it.
What Crawlee for Python provides
Crawlee wraps several scraping tools into a single package. It offers typed Python classes, a request queue that visits each URL once, automatic storage of scraped records as JSON files, and a request handler pattern that keeps your extraction logic separate from crawling mechanics. You can switch between an HTTP crawler and a headless browser crawler with minimal code changes, and Crawlee handles concurrency through asyncio while scaling based on available system resources. Proxy rotation is built in: you supply a list of proxy URLs, and Crawlee tries them, discarding ones that time out or return error codes such as 401 and 403.
Installation and extras
Install the base package with pip using the package name crawlee. If you plan to use BeautifulSoup or Playwright, install the corresponding extras. The Playwright extra also requires its own browser dependencies. The exact commands are preserved in the code modules below.
Basic usage: BeautifulSoupCrawler for static pages
The first example uses BeautifulSoupCrawler because the target blog pages have no JavaScript-rendered content and basic HTTP is sufficient. The crawler is initialized, a default request handler is defined, and the run starts with one URL. Crawlee adds that URL to a queue and visits each queued URL once, even if the same URL is added multiple times. Inside the handler, context.soup gives you a BeautifulSoup object for extracting data. The example selects blog cards, pushes a dictionary with url, title, author, and date for each card using context.push_data, and then enqueues pagination links with a CSS selector. Each push_data call saves a JSON file in the storage/defaults/dataset directory, sequentially numbered as 000000001.json, 000000002.json, and so on. The example run produced 158 JSON files; one sample record is shown in the JSON module. The full crawler code is in the private source module and is not reproduced here.
PlaywrightCrawler for browser-rendered pages
When a page needs a real browser, switch to PlaywrightCrawler. The structure stays similar: initialize the crawler, define a request handler, and call run with a starting URL. In the example, the crawler is configured with max_requests_per_crawl=5, headless=False, and browser_type='firefox'. The handler checks the URL: if it contains /charts, it enqueues book links using a CSS selector; if it contains /dp/ or /gp/, it extracts the product title, byline, and description from the page and pushes the data. After the run, crawler.export_data('books.json') writes all collected records to a JSON file. Because headless mode is disabled, you can watch the browser windows open and close as concurrency runs. The exact code is in the Playwright module.
Proxy rotation with ProxyConfiguration
Crawlee's ProxyConfiguration accepts a list of proxy URLs. In the demonstration, three proxy URLs are supplied and four requests are made to ipinfo.io with unique query parameters. The proxies rotate in round-robin order, so the first and fourth requests use the same proxy. The example output shows three unique locations: Lake Mary, Florida; Zagreb, Zagreb; and Gladeview, Florida. This illustrates that with three proxies, you cycle through them in sequence. The exact code and output are in the proxy modules.
Session management per proxy
To keep a consistent session for each proxy, add use_session_pool=True and persist_cookies_per_session=True to the crawler constructor. This maps one session to each proxy and reuses that session whenever the proxy is used. The change is small: the proxy configuration stays the same, and the crawler gains two keyword arguments. The exact code is in the session module.
Tiered proxies: cheap first, premium on failure
Not all proxies perform equally. A cost-effective strategy is to try cheaper proxies first and fall back to more expensive ones only when the first tier fails. Crawlee supports this with tiered_proxy_urls, a list of lists. The first inner list contains the first tier of proxy URLs; if those fail, the next tier is used. The example shows two tiers: free proxies first, then premium proxies. The exact code is in the tiered proxy module.
Decision checklist: which crawler and proxy setup to use
Use this checklist to choose the right configuration for your task.
- If the page content is present in the initial HTML response and does not require JavaScript execution, start with BeautifulSoupCrawler. It is lighter and faster.
- If the page relies on JavaScript to render content, or if you need to interact with the page (click, scroll, type), use PlaywrightCrawler. You can run it headless or with a visible browser for debugging.
- If you have a small set of proxies and want simple rotation, use ProxyConfiguration with proxy_urls. Requests will cycle through the list in round-robin order.
- If you need to maintain login state or cookies per proxy, enable use_session_pool and persist_cookies_per_session.
- If you want to minimize cost by trying cheaper proxies first, use tiered_proxy_urls with two or more tiers.
- Always verify that your use of proxies and crawling complies with the target site's terms and applicable laws.
Field and control map for the examples
The table below summarizes the key constructor parameters and handler context attributes used in the examples. This is a narrative reference, not a substitute for the exact code modules.
| Parameter or attribute | Used in | Purpose |
|---|---|---|
| max_requests_per_crawl | PlaywrightCrawler | Limits the number of requests per crawl |
| headless | PlaywrightCrawler | When False, shows the browser GUI |
| browser_type | PlaywrightCrawler | Selects the browser engine, e.g., 'firefox' |
| proxy_configuration | Both crawlers | Accepts a ProxyConfiguration instance |
| use_session_pool | BeautifulSoupCrawler | Enables a pool of sessions |
| persist_cookies_per_session | BeautifulSoupCrawler | Persists cookies for each session |
| context.request.url | Request handler | The URL being processed |
| context.soup | BeautifulSoupCrawlingContext | BeautifulSoup object for parsing |
| context.page | PlaywrightCrawlingContext | Playwright page object for browser interaction |
| context.push_data | Request handler | Stores a dictionary as a JSON record |
| context.enqueue_links | Request handler | Adds links matching a selector to the queue |
| crawler.export_data | After run | Writes collected data to a file |
Summary
Crawlee for Python brings together HTTP scraping, browser automation, queuing, storage, and proxy management in one library. The examples show a basic BeautifulSoup crawler for static pages, a Playwright crawler for browser-rendered pages, and proxy configurations ranging from simple rotation to session persistence and tiered fallback. The main benefit is cleaner, more maintainable crawling code with less boilerplate.
Frequently asked questions
Do I need to install Playwright separately when using Crawlee?
Yes. If you install the Playwright extra with pip install 'crawlee[playwright]', you must also run playwright install to install the browser dependencies. The exact commands are in the installation code modules.
How does Crawlee avoid visiting the same URL twice?
Crawlee uses a request queue and ensures each URL is crawled only once, even if you add the same URL multiple times. This is handled automatically by the framework.
What is the difference between proxy rotation and tiered proxies?
Proxy rotation cycles through a list of proxies in round-robin order for each request. Tiered proxies let you define groups of proxies: Crawlee tries the first tier, and if those proxies fail (for example, due to timeouts or 401/403 errors), it moves to the next tier. Rotation is about distributing requests; tiering is about fallback strategy.
Can I see the browser while Crawlee runs Playwright?
Yes. Set headless=False in the PlaywrightCrawler constructor. The browser window will open and you can watch the automation. This is useful for debugging.
{
"url": "/blog/how-to-scrape-tiktok/",
"title": "How to Scrape TikTok in 2026: Scrape Profile Stats and Videos",
"author": "Ismail Ajagbe",
"date": "2021-09-27"
}