Skip to content

Playwright for Python Web Scraping: Build a Multi-Page Scraper

Playwright drives Chromium, Firefox, and WebKit through one Python API, which makes it a practical choice when a scraping task needs a real browser. This guide builds a scraper in stages: first a single-page screenshot or text capture, then a sitemap-to-CSV URL list, then threaded processing with retries, and finally waits and interactions for content that appears after JavaScript runs.

What Playwright for Python is

Playwright is Microsoft's browser-automation library, available for Python and JavaScript. It opens pages, clicks and fills controls, reads rendered text, and captures screenshots. For scraping, the useful part is that it executes the same page code a browser would, so content added by scripts becomes readable after it appears.

Headless mode runs the browser without a visible window. That suits batch jobs where nobody needs to watch the page. If you are debugging a selector, launch with a visible window instead so you can see what the page actually renders.

Playwright compared with Selenium

Both tools automate browsers, but they differ in scope and ergonomics. Use this checklist when choosing:

  • Engine coverage: one Playwright install can drive Chromium, Firefox, and WebKit, so a cross-engine test matrix does not require a separate driver per browser.
  • Component-heavy pages: shadow DOM and web-component trees were part of the design target, which helps on sites built from encapsulated widgets.
  • Predictable scripting: the same API shape applies across engines, so you write one code path instead of branching per browser.
  • Waiting model: auto-waiting around actions and selectors cuts down on hand-tuned sleeps that break when a page slows down.

If your existing suite is already invested in Selenium, the migration cost may outweigh these advantages. For a new scraping project, the waiting model alone is often enough reason to start with Playwright.

Environment setup

You need Python 3, a Mac, Linux, or Windows 10+ machine, a terminal, and an editor. A basic grasp of Python is assumed.

With Poetry, initialize a project, add Playwright to pyproject.toml, install, and create main.py. The exact commands are preserved in the setup command modules accompanying this page. If you prefer pip, install Playwright directly and then install the browser binaries; the browser install step is easy to forget and produces a launch error when skipped.

Run the script through Poetry or directly with python3 main.py, depending on how you installed the package. The original walkthrough also points to an external GitHub repository for its sample code; treat that as a third-party reference rather than a bundled download, and substitute {{OWNER_SKILLS_REPOSITORY}} if you maintain your own copy.

Python example

playwright = "^1.45.1"

Scrape or screenshot one page

The smallest useful script launches Chromium, opens a URL, optionally saves a full-page PNG, and closes the browser. The full_page option matters on long pages because it captures the whole scrollable document rather than the viewport.

When the --screenshot flag is absent, the enhanced version reads the page title and the text inside a chosen selector, then writes both to a text file. The filename is derived from the title by stripping characters outside word characters, whitespace, and hyphens, then replacing spaces with underscores. That keeps the output safe across operating systems.

Run it as python3 main.py <url> [--screenshot]. Replace <url> with the target page and add the flag only when you want an image instead of text.

Process many pages from a sitemap

For a whole site, start by collecting URLs. The approach here scans a sitemap XML file with Requests and BeautifulSoup, follows nested sitemap entries recursively, and appends loc, lastmod, and priority values to urls.csv. Missing attributes are written as n/a. The header row is written only when the file does not already exist, so repeated runs append rather than duplicate the header.

The scraper then reads the loc column from that CSV and visits each URL. The command-line argument changes from a single URL to the CSV path, and the same screenshot-or-text branch applies to every row.

The source tutorial also shows an optional client wrapper for routing sitemap fetches through a paid proxy service. That wrapper is not part of this page's deliverables: if you want an owned equivalent, record the package under {{OWNER_PACKAGE}} and supply credentials through {{OWNER_API_KEY}} rather than pasting a key into the script.

Python example

requests = "^2.31"
beautifulsoup4 = "^4.12"
lxml = "^5.1" # this library might not be needed in your setup

Parallelism and retries

Sequential visits are slow on large lists. Playwright is not thread-safe, so the script creates a separate sync_playwright context per worker and uses ThreadPoolExecutor with max_workers=5. Keeping the pool small also limits request pressure on the target site.

Retries use exponential backoff: the first retry waits 1 second, the next 2, then 4, and so on, with a default of two retries. The process_url function tracks attempts and re-runs the whole browser session on failure. After the final attempt it prints the error and moves on.

A useful distinction when debugging: a missing selector is not a retryable network failure. The script writes "No requested selector found" into the output file rather than raising, so check the selector before assuming the site blocked you.

Wait for and interact with dynamic content

Content that loads after the initial HTML needs an explicit wait. page.wait_for_selector("h1") blocks until the element exists, and page.wait_for_response with a URL and status predicate waits for a specific network call. Use the selector wait when the DOM is the signal; use the response wait when an API call is the signal.

Interaction follows the same model: fill fields, click a submit button, and wait for navigation before reading the next page. This is how a scraper reaches content behind a form without pretending to be a plain HTTP client.

Handle JavaScript-rendered content and infinite scroll

For JavaScript-driven pages, wait for the network response that carries the data, then read page.content() after it resolves. When an element is rendered by script, wait_for_selector followed by inner_text extracts the visible text.

Infinite scroll needs a loop: scroll to the bottom, wait a fixed interval for new content, and repeat. The example does five rounds with a two-second pause each. Tune both numbers to the site; too short a pause captures a partial list, and too many rounds wastes time.

Reduce the chance of being blocked

Sites may throttle or block automated traffic. Practical mitigations include routing requests through proxies so they do not all originate from one IP, rotating user-agent strings so requests resemble different browsers, and ignoring hidden honeypot elements that only bots would touch. Always read the target site's robots.txt before crawling.

A managed scraping API can absorb proxy rotation, headless rendering, and anti-bot handling if you would rather not maintain that layer. The tutorial's author recommends one such service and lists its headless browsers, proxy rotation, premium proxies, and Google search scraping as reasons. That is the author's recommendation, not a capability of this page; evaluate any provider against your own volume, budget, and compliance needs before committing.

Frequently asked questions

Do I need to install browser binaries separately?

Yes. Installing the Playwright package is not enough; run playwright install (or poetry run playwright install) to download the browsers. Skipping this step causes a launch failure.

Why does the parallel script create a new Playwright context per URL?

Playwright's sync API is not thread-safe. Each worker opens its own sync_playwright context so threads do not share browser state. The example caps the pool at five workers to limit load on the target site.

The output file says 'No requested selector found'. What went wrong?

The selector passed to query_selector did not match any element on the rendered page. Check the selector against the live DOM, and confirm the page finished loading before the query runs. This is a selector problem, not a retryable network error.

What columns does the URL CSV need?

The scraper reads the loc column. The sitemap script writes loc, lastmod, and priority, using n/a when an attribute is absent. If you supply your own CSV, include a loc header.

How does the exponential backoff work?

With backoff_factor=1 and retries=2, the first retry sleeps 1 second and the second sleeps 2 seconds. The delay doubles on each attempt so a struggling server gets progressively more recovery time.