Scrapy-Playwright Tutorial: Dynamic Page Scraping Tasks
Scrapy-Playwright pairs Scrapy's crawling engine with a real browser, so requests can render JavaScript, respond to clicks and scrolling, and return a page you can parse with selectors. This guide walks through the setup and the practical actions you'll need for JavaScript-heavy sites: waiting for content, typing into fields, capturing screenshots, paginating, intercepting AJAX, logging in, blocking heavy resources, limiting browser pages, restarting crashed browsers, and routing through proxies.
What Scrapy-Playwright gives you
The integration lets a Scrapy spider hand selected requests to Playwright. You keep Scrapy's scheduling, item pipeline and selector API, while gaining the ability to render JavaScript and drive page interactions. Typical jobs include extracting content that only appears after a script runs, clicking buttons to reveal hidden data, scrolling until a list stops growing, and reading structured JSON that a page fetches in the background. Requests you don't flag for Playwright continue through Scrapy's default download handler, so you can mix static and rendered pages in one crawl.
Prerequisites and environment
You need Python installed, a code editor such as Visual Studio Code, and basic familiarity with CSS selectors and browser DevTools. Some Scrapy and Playwright knowledge helps, but the steps below are self-contained. Create an isolated virtual environment so the Scrapy-Playwright package and browser binaries don't collide with other projects. The commands below are the exact ones used in the walkthrough; run them in order.
Create the environment and install browsers
Create and activate the virtual environment, then install the package and browser binaries. The full install pulls Firefox, WebKit and Chromium; if you only need Chromium, install just that browser. The walkthrough uses Chromium.
Start a Scrapy project and spider
Generate the project, move into it, and create a spider. The example spider targets Google Finance for NVIDIA's stock page. After generating the spider, return to the parent directory and open the project in your editor.
Configure Scrapy settings for Playwright
Two settings are essential. First, replace the default HTTP and HTTPS download handlers with the Scrapy-Playwright handler so flagged requests are routed to the browser. Second, set the Twisted reactor to the asyncio-compatible one; without it, Playwright's asynchronous work fails. Optional settings choose the browser type and pass launch options such as headless mode.
Python example
DOWNLOAD_HANDLERS = {
"http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
"https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}Python example
# settings.py
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"Python example
# Enable Scrapy-Playwright
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"
DOWNLOAD_HANDLERS = {
"http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
"https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}
### (Optional)
PLAYWRIGHT_BROWSER_TYPE = "chromium" # Choose 'chromium', 'firefox', or 'webkit'
PLAYWRIGHT_LAUNCH_OPTIONS = {
"headless": False, # Set to True if you prefer headless mode
}Scrape a JavaScript-rendered page
Flag a request with the playwright meta key and add page methods that run before your callback. In the Google Finance example, the spider waits for the price element with wait_for_selector, then the parse method reads the text with a CSS selector. The captured run logged a price of $163.18 for the GOOG quote page. Run the spider from the project directory with scrapy crawl google_finance.
Click buttons and reveal hidden data
Use PageMethod("click", selector) to drive buttons. The example opens the Share popup on a Google Finance quote page and then clicks the Copy link button. Set playwright_include_page to True when you need the page object in the response for further work.
Wait for elements, load states and time
Three wait strategies cover most dynamic pages. wait_for_selector pauses until a specific element exists, which prevents acting on markup that hasn't rendered. wait_for_load_state waits for a page-level state: "load" for a fully loaded page, "domcontentloaded" for the DOM, and "networkidle" when no more than two connections have been active for at least 500 ms. wait_for_timeout pauses for a fixed number of milliseconds and is best reserved for animations or transitions where event-based waits don't apply.
Type into fields and extract text
PageMethod("fill", selector, text) writes into an input, and PageMethod("press", selector, key) sends a key such as Enter. The YouTube example waits for the search field, fills it with "NASA", presses Enter, and waits for channel results. Extracting text afterward uses the same selector approach as any Scrapy spider; the example reads a subscriber count with a CSS selector and logs it.
Capture screenshots
PageMethod("screenshot", path=...) saves an image of the page. The format can be jpg or png. Add full_page=True to capture the whole document rather than the viewport, and omit_background=True to keep transparency. Screenshots are useful for confirming that a page reached the expected state during a run.
Run custom JavaScript on the page
PageMethod("evaluate", script) runs JavaScript in the page context. The example selects the price element, converts the displayed USD value to EUR with a mock rate, and writes the new text back before taking a screenshot. The parse method then reads the modified value. This technique is useful when you need to change the DOM or trigger behavior that Playwright's built-in methods don't cover.
Handle infinite scrolling
Infinite-scroll pages add content as you move down. A custom evaluate script can scroll to the bottom, wait for new content, compare the page height, and stop when the height stops changing. The example applies this to a Nike new-shoes page and then extracts product card titles. The loop waits three seconds between scrolls so the page has time to load.
Follow pagination across multiple pages
When a site uses a More or Next link, follow that link in your callback. The Hacker News example reads the a.morelink href, joins it to the current URL, and yields a new request. It accepts an optional max_pages argument so you can cap the crawl, and it reuses the same Playwright page object to avoid opening a fresh browser session for every page.
Capture AJAX responses
Instead of parsing rendered HTML, you can intercept the network requests that supply the data. The Nike example registers a route handler, filters requests to the Nike API, fetches the response body, parses it as JSON, and stores it. A scroll loop triggers more product loads, and the collected data is written to a JSON file at the end. This produces structured records that are often cleaner than the rendered page.
Automate form submissions and logins
Fill the username and password fields with PageMethod("fill"), click the submit button, and optionally take a screenshot to confirm the result. Scrapy honors robots.txt by default, so a login page may be blocked; the walkthrough shows the ROBOTSTXT_OBEY setting and notes that disabling it is risky and that site terms should be checked. Keep credentials out of committed code.
Python example
ROBOTSTXT_OBEY = FalseKeep a session across parallel requests
When a crawl needs to stay logged in, set playwright_include_page=True on the login request and on subsequent requests so the same page and session are reused. The example logs in, then sends parallel requests to several Hacker News pages and extracts headlines from each.
Block unwanted requests
Images, media and stylesheets can slow a crawl without contributing data. Define an abort function and assign it to PLAYWRIGHT_ABORT_REQUEST. The example blocks by resource type and by file extension, then scrolls and extracts product names from the remaining HTML.
Manage pages, contexts and browser restarts
PLAYWRIGHT_MAX_PAGES_PER_CONTEXT caps how many pages can be open in one browser context, and CONCURRENT_REQUESTS controls how many requests Scrapy processes at once. For long jobs, PLAYWRIGHT_RESTART_DISCONNECTED_BROWSER lets a crashed or disconnected browser restart automatically so the crawl can continue.
Python example
PLAYWRIGHT_MAX_PAGES_PER_CONTEXT = 5 # Allow up to 5 pages per contextPython example
# settings.py
CONCURRENT_REQUESTS = 8 # Set to 8 concurrent requests at a timePython example
PLAYWRIGHT_RESTART_DISCONNECTED_BROWSER = TrueRoute requests through proxies
Anti-scraping measures such as rate limits, IP bans and CAPTCHAs are common. A global proxy can be set in PLAYWRIGHT_LAUNCH_OPTIONS with server, username and password fields; omit the credentials for unauthenticated proxies. To use a different proxy per request, pass playwright_context_kwargs with a new context. Managing proxies manually adds operational work; a managed scraping API is one alternative when you'd rather not handle rotation and anti-bot bypass yourself.
Python example
PLAYWRIGHT_LAUNCH_OPTIONS = {
"proxy": {
"server": "http://proxy_ip:proxy_port", # Replace with your proxy server
"username": "your_username", # Proxy username
"password": "your_password", # Proxy password
}
}Python example
PLAYWRIGHT_LAUNCH_OPTIONS = {
"proxy": {
"server": "http://proxy_ip:proxy_port",
}
}| Situation | Preferred wait | Why |
|---|---|---|
| A specific element must exist before you click or read it | wait_for_selector | Event-driven; stops as soon as the element appears |
| The whole page or its network activity must settle | wait_for_load_state | "load", "domcontentloaded" and "networkidle" cover different page states |
| An animation or transition needs a fixed pause | wait_for_timeout | Use sparingly; fixed delays add unnecessary time |
Guidance reflects the walkthrough's examples; actual timing depends on the target page.
| Name | Type | Purpose |
|---|---|---|
| playwright | request meta key | Routes the request through Playwright |
| playwright_include_page | request meta key | Includes the Playwright page object in the response |
| playwright_page_methods | request meta key | List of PageMethod actions to run before the callback |
| playwright_context | request meta key | Selects or creates a browser context |
| playwright_context_kwargs | request meta key | Per-request context options such as a proxy |
| playwright_page | request meta key | Reuses an existing page object |
| DOWNLOAD_HANDLERS | settings.py | Routes HTTP/HTTPS through the Scrapy-Playwright handler |
| TWISTED_REACTOR | settings.py | Sets the asyncio-compatible reactor |
| PLAYWRIGHT_BROWSER_TYPE | settings.py | Chooses chromium, firefox or webkit |
| PLAYWRIGHT_LAUNCH_OPTIONS | settings.py | Browser launch options including headless and proxy |
| PLAYWRIGHT_ABORT_REQUEST | settings.py | Function that blocks unwanted requests |
| PLAYWRIGHT_MAX_PAGES_PER_CONTEXT | settings.py | Caps open pages per context |
| PLAYWRIGHT_RESTART_DISCONNECTED_BROWSER | settings.py | Restarts a disconnected browser |
| CONCURRENT_REQUESTS | settings.py | Number of simultaneous Scrapy requests |
| ROBOTSTXT_OBEY | settings.py | Whether Scrapy honors robots.txt |
Names and types are taken from the walkthrough's configuration examples.
| Symptom | Likely cause | First check |
|---|---|---|
| Request rejected with a robots.txt message | ROBOTSTXT_OBEY is enabled and the page is disallowed | Review the site's robots.txt and terms before changing the setting |
| Selector returns nothing after a click | Element not yet rendered or selector changed | Add wait_for_selector and verify the selector in DevTools |
| Async requests fail immediately | Twisted reactor not set to the asyncio-compatible one | Confirm TWISTED_REACTOR in settings.py |
| Browser disconnects mid-crawl | Long or resource-heavy run | Enable PLAYWRIGHT_RESTART_DISCONNECTED_BROWSER |
Diagnostic guidance derived from the walkthrough's configuration and examples.
Frequently asked questions
Do I need to enable Playwright for every request?
No. Only requests with the playwright meta key set to True are handled by the Playwright download handler; other requests use Scrapy's default handler.
Which wait should I use for a page with ongoing network activity?
wait_for_load_state with "networkidle" is the most reliable for pages that keep making background requests. Use wait_for_selector when you only need one element, and reserve wait_for_timeout for animations or transitions.
Can I keep a login session across multiple requests?
Yes. Set playwright_include_page=True on the login request and on later requests so the same Playwright page and session are reused.
How do I stop a crawl from running forever on a paginated site?
Pass a max_pages argument to the spider. Without it, the spider follows the More link until no further link is found.
Why might a login page be blocked before Playwright runs?
Scrapy honors robots.txt by default. If the page is disallowed, the request is rejected before rendering. Review the site's terms before changing ROBOTSTXT_OBEY.
[google_finance] DEBUG: Google stock price (EUR): 141.40 €
[nike_shoes] DEBUG: Product: Nike Metcon 9
[nike_shoes] DEBUG: Product: Nike Dunk Low Twist
[nike_shoes] DEBUG: Product: Air Max 1
[nike_shoes] DEBUG: Product: Nike Air Max 1 EasyOn
...
[hacker_news] DEBUG: Post title: Gleam: A Basic Introduction
[hacker_news] DEBUG: Post title: Bear is a tool that generates a compilation database for Clang tooling
[hacker_news] DEBUG: Post title: A Lisp compiler to RISC-V written in Lisp
[hacker_news] DEBUG: Post title: Grokking at the edge of linear separability
[hacker_news] DEBUG: Post title: Techie took five minutes to fix problem Adobe and Microsoft couldn't solve
...
[scrapy.downloadermiddlewares.robotstxt] DEBUG: Forbidden by robots.txt: <GET https://news.ycombinator.com/login>
[abort_request] DEBUG: Product: Nike Dunk Low By You
[abort_request] DEBUG: Product: Nike LeBron 9 Low
[abort_request] DEBUG: Product: Air Jordan 6 Retro Low x PSG
...