Skip to content

Scrapy-Playwright Tutorial: Dynamic Page Scraping Tasks

Scrapy-Playwright pairs Scrapy's crawling engine with a real browser, so requests can render JavaScript, respond to clicks and scrolling, and return a page you can parse with selectors. This guide walks through the setup and the practical actions you'll need for JavaScript-heavy sites: waiting for content, typing into fields, capturing screenshots, paginating, intercepting AJAX, logging in, blocking heavy resources, limiting browser pages, restarting crashed browsers, and routing through proxies.

What Scrapy-Playwright gives you

The integration lets a Scrapy spider hand selected requests to Playwright. You keep Scrapy's scheduling, item pipeline and selector API, while gaining the ability to render JavaScript and drive page interactions. Typical jobs include extracting content that only appears after a script runs, clicking buttons to reveal hidden data, scrolling until a list stops growing, and reading structured JSON that a page fetches in the background. Requests you don't flag for Playwright continue through Scrapy's default download handler, so you can mix static and rendered pages in one crawl.

Prerequisites and environment

You need Python installed, a code editor such as Visual Studio Code, and basic familiarity with CSS selectors and browser DevTools. Some Scrapy and Playwright knowledge helps, but the steps below are self-contained. Create an isolated virtual environment so the Scrapy-Playwright package and browser binaries don't collide with other projects. The commands below are the exact ones used in the walkthrough; run them in order.

Create the environment and install browsers

Create and activate the virtual environment, then install the package and browser binaries. The full install pulls Firefox, WebKit and Chromium; if you only need Chromium, install just that browser. The walkthrough uses Chromium.

Start a Scrapy project and spider

Generate the project, move into it, and create a spider. The example spider targets Google Finance for NVIDIA's stock page. After generating the spider, return to the parent directory and open the project in your editor.

Configure Scrapy settings for Playwright

Two settings are essential. First, replace the default HTTP and HTTPS download handlers with the Scrapy-Playwright handler so flagged requests are routed to the browser. Second, set the Twisted reactor to the asyncio-compatible one; without it, Playwright's asynchronous work fails. Optional settings choose the browser type and pass launch options such as headless mode.

Python example

DOWNLOAD_HANDLERS = {
    "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}

Python example

# settings.py
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"

Python example

# Enable Scrapy-Playwright
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"
DOWNLOAD_HANDLERS = {
    "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}

### (Optional)
PLAYWRIGHT_BROWSER_TYPE = "chromium"  # Choose 'chromium', 'firefox', or 'webkit'
PLAYWRIGHT_LAUNCH_OPTIONS = {
    "headless": False,  # Set to True if you prefer headless mode
}

Scrape a JavaScript-rendered page

Flag a request with the playwright meta key and add page methods that run before your callback. In the Google Finance example, the spider waits for the price element with wait_for_selector, then the parse method reads the text with a CSS selector. The captured run logged a price of $163.18 for the GOOG quote page. Run the spider from the project directory with scrapy crawl google_finance.

Click buttons and reveal hidden data

Use PageMethod("click", selector) to drive buttons. The example opens the Share popup on a Google Finance quote page and then clicks the Copy link button. Set playwright_include_page to True when you need the page object in the response for further work.

Wait for elements, load states and time

Three wait strategies cover most dynamic pages. wait_for_selector pauses until a specific element exists, which prevents acting on markup that hasn't rendered. wait_for_load_state waits for a page-level state: "load" for a fully loaded page, "domcontentloaded" for the DOM, and "networkidle" when no more than two connections have been active for at least 500 ms. wait_for_timeout pauses for a fixed number of milliseconds and is best reserved for animations or transitions where event-based waits don't apply.

Type into fields and extract text

PageMethod("fill", selector, text) writes into an input, and PageMethod("press", selector, key) sends a key such as Enter. The YouTube example waits for the search field, fills it with "NASA", presses Enter, and waits for channel results. Extracting text afterward uses the same selector approach as any Scrapy spider; the example reads a subscriber count with a CSS selector and logs it.

Capture screenshots

PageMethod("screenshot", path=...) saves an image of the page. The format can be jpg or png. Add full_page=True to capture the whole document rather than the viewport, and omit_background=True to keep transparency. Screenshots are useful for confirming that a page reached the expected state during a run.

Run custom JavaScript on the page

PageMethod("evaluate", script) runs JavaScript in the page context. The example selects the price element, converts the displayed USD value to EUR with a mock rate, and writes the new text back before taking a screenshot. The parse method then reads the modified value. This technique is useful when you need to change the DOM or trigger behavior that Playwright's built-in methods don't cover.

Handle infinite scrolling

Infinite-scroll pages add content as you move down. A custom evaluate script can scroll to the bottom, wait for new content, compare the page height, and stop when the height stops changing. The example applies this to a Nike new-shoes page and then extracts product card titles. The loop waits three seconds between scrolls so the page has time to load.

Follow pagination across multiple pages

When a site uses a More or Next link, follow that link in your callback. The Hacker News example reads the a.morelink href, joins it to the current URL, and yields a new request. It accepts an optional max_pages argument so you can cap the crawl, and it reuses the same Playwright page object to avoid opening a fresh browser session for every page.

Capture AJAX responses

Instead of parsing rendered HTML, you can intercept the network requests that supply the data. The Nike example registers a route handler, filters requests to the Nike API, fetches the response body, parses it as JSON, and stores it. A scroll loop triggers more product loads, and the collected data is written to a JSON file at the end. This produces structured records that are often cleaner than the rendered page.

Automate form submissions and logins

Fill the username and password fields with PageMethod("fill"), click the submit button, and optionally take a screenshot to confirm the result. Scrapy honors robots.txt by default, so a login page may be blocked; the walkthrough shows the ROBOTSTXT_OBEY setting and notes that disabling it is risky and that site terms should be checked. Keep credentials out of committed code.

Python example

ROBOTSTXT_OBEY = False

Keep a session across parallel requests

When a crawl needs to stay logged in, set playwright_include_page=True on the login request and on subsequent requests so the same page and session are reused. The example logs in, then sends parallel requests to several Hacker News pages and extracts headlines from each.

Block unwanted requests

Images, media and stylesheets can slow a crawl without contributing data. Define an abort function and assign it to PLAYWRIGHT_ABORT_REQUEST. The example blocks by resource type and by file extension, then scrolls and extracts product names from the remaining HTML.

Manage pages, contexts and browser restarts

PLAYWRIGHT_MAX_PAGES_PER_CONTEXT caps how many pages can be open in one browser context, and CONCURRENT_REQUESTS controls how many requests Scrapy processes at once. For long jobs, PLAYWRIGHT_RESTART_DISCONNECTED_BROWSER lets a crashed or disconnected browser restart automatically so the crawl can continue.

Python example

PLAYWRIGHT_MAX_PAGES_PER_CONTEXT = 5  # Allow up to 5 pages per context

Python example

# settings.py
CONCURRENT_REQUESTS = 8  # Set to 8 concurrent requests at a time

Python example

PLAYWRIGHT_RESTART_DISCONNECTED_BROWSER = True

Route requests through proxies

Anti-scraping measures such as rate limits, IP bans and CAPTCHAs are common. A global proxy can be set in PLAYWRIGHT_LAUNCH_OPTIONS with server, username and password fields; omit the credentials for unauthenticated proxies. To use a different proxy per request, pass playwright_context_kwargs with a new context. Managing proxies manually adds operational work; a managed scraping API is one alternative when you'd rather not handle rotation and anti-bot bypass yourself.

Python example

PLAYWRIGHT_LAUNCH_OPTIONS = {
    "proxy": {
        "server": "http://proxy_ip:proxy_port",  # Replace with your proxy server
        "username": "your_username",  # Proxy username
        "password": "your_password",  # Proxy password
    }
}

Python example

PLAYWRIGHT_LAUNCH_OPTIONS = {
    "proxy": {
        "server": "http://proxy_ip:proxy_port",
    }
}
Wait strategy decision checklist
SituationPreferred waitWhy
A specific element must exist before you click or read itwait_for_selectorEvent-driven; stops as soon as the element appears
The whole page or its network activity must settlewait_for_load_state"load", "domcontentloaded" and "networkidle" cover different page states
An animation or transition needs a fixed pausewait_for_timeoutUse sparingly; fixed delays add unnecessary time

Guidance reflects the walkthrough's examples; actual timing depends on the target page.

Playwright meta and settings map
NameTypePurpose
playwrightrequest meta keyRoutes the request through Playwright
playwright_include_pagerequest meta keyIncludes the Playwright page object in the response
playwright_page_methodsrequest meta keyList of PageMethod actions to run before the callback
playwright_contextrequest meta keySelects or creates a browser context
playwright_context_kwargsrequest meta keyPer-request context options such as a proxy
playwright_pagerequest meta keyReuses an existing page object
DOWNLOAD_HANDLERSsettings.pyRoutes HTTP/HTTPS through the Scrapy-Playwright handler
TWISTED_REACTORsettings.pySets the asyncio-compatible reactor
PLAYWRIGHT_BROWSER_TYPEsettings.pyChooses chromium, firefox or webkit
PLAYWRIGHT_LAUNCH_OPTIONSsettings.pyBrowser launch options including headless and proxy
PLAYWRIGHT_ABORT_REQUESTsettings.pyFunction that blocks unwanted requests
PLAYWRIGHT_MAX_PAGES_PER_CONTEXTsettings.pyCaps open pages per context
PLAYWRIGHT_RESTART_DISCONNECTED_BROWSERsettings.pyRestarts a disconnected browser
CONCURRENT_REQUESTSsettings.pyNumber of simultaneous Scrapy requests
ROBOTSTXT_OBEYsettings.pyWhether Scrapy honors robots.txt

Names and types are taken from the walkthrough's configuration examples.

Failure distinction checklist
SymptomLikely causeFirst check
Request rejected with a robots.txt messageROBOTSTXT_OBEY is enabled and the page is disallowedReview the site's robots.txt and terms before changing the setting
Selector returns nothing after a clickElement not yet rendered or selector changedAdd wait_for_selector and verify the selector in DevTools
Async requests fail immediatelyTwisted reactor not set to the asyncio-compatible oneConfirm TWISTED_REACTOR in settings.py
Browser disconnects mid-crawlLong or resource-heavy runEnable PLAYWRIGHT_RESTART_DISCONNECTED_BROWSER

Diagnostic guidance derived from the walkthrough's configuration and examples.

Frequently asked questions

Do I need to enable Playwright for every request?

No. Only requests with the playwright meta key set to True are handled by the Playwright download handler; other requests use Scrapy's default handler.

Which wait should I use for a page with ongoing network activity?

wait_for_load_state with "networkidle" is the most reliable for pages that keep making background requests. Use wait_for_selector when you only need one element, and reserve wait_for_timeout for animations or transitions.

Can I keep a login session across multiple requests?

Yes. Set playwright_include_page=True on the login request and on later requests so the same Playwright page and session are reused.

How do I stop a crawl from running forever on a paginated site?

Pass a max_pages argument to the spider. Without it, the spider follows the More link until no further link is found.

Why might a login page be blocked before Playwright runs?

Scrapy honors robots.txt by default. If the page is disallowed, the request is rejected before rendering. Review the site's terms before changing ROBOTSTXT_OBEY.

[google_finance] DEBUG: Google stock price (EUR): 141.40 €
[nike_shoes] DEBUG: Product: Nike Metcon 9
[nike_shoes] DEBUG: Product: Nike Dunk Low Twist
[nike_shoes] DEBUG: Product: Air Max 1
[nike_shoes] DEBUG: Product: Nike Air Max 1 EasyOn
...
[hacker_news] DEBUG: Post title: Gleam: A Basic Introduction
[hacker_news] DEBUG: Post title: Bear is a tool that generates a compilation database for Clang tooling
[hacker_news] DEBUG: Post title: A Lisp compiler to RISC-V written in Lisp
[hacker_news] DEBUG: Post title: Grokking at the edge of linear separability
[hacker_news] DEBUG: Post title: Techie took five minutes to fix problem Adobe and Microsoft couldn't solve
...
[scrapy.downloadermiddlewares.robotstxt] DEBUG: Forbidden by robots.txt: <GET https://news.ycombinator.com/login>
[abort_request] DEBUG: Product: Nike Dunk Low By You
[abort_request] DEBUG: Product: Nike LeBron 9 Low
[abort_request] DEBUG: Product: Air Jordan 6 Retro Low x PSG
...