Skip to content

Common questions about web scraping with Playwright

Playwright drives a real browser, so most scraping problems come down to timing, locating the right element, or keeping a session alive between runs.

OUTPUT FORMATSHTMLJSONMarkdown

This page is a question index: each entry points to a focused answer for one task. Use the decision table below to work out which capability you need, then open the matching question.

Start with the task, not the API

Before opening a specific question, name the problem you actually have. Scraping failures with a browser automation tool usually fall into one of five buckets:

  • The page or element is not ready yet (timing).
  • The element exists but your locator does not match it (selection).
  • The site expects a prior session, login or consent state (cookies).
  • You need something other than text — an image, a file, or a network response (capture).
  • The run is slow or noisy because it loads assets you do not need (performance).

Matching your problem to a bucket first saves you from rewriting selectors when the real issue is a missing wait, or adding waits when the selector is simply wrong.

Timing versus selection: a distinction worth getting right

These two failures look identical in a stack trace but need opposite fixes.

A timing failure means the document is still loading or the element is rendered later by script. The fix is to wait for the page or the element before interacting.

A selection failure means the element is already there, but your CSS or XPath expression points somewhere else — a different sibling, a stale class name, or a container that no longer exists. The fix is to inspect the DOM and correct the locator.

A quick test: if you pause the script manually and the element appears, it was timing. If it still is not found, it is selection.

A minimal sequence using captured capabilities

The questions on this page cover the pieces of a typical scrape. Put together in order, a simple run looks like this:

  1. Open the page and wait until it has loaded (see the page-load question).
  2. Optionally block resource types you do not need, such as images or fonts, to speed the run up.
  3. Locate the elements you want with a CSS selector, or with XPath when the structure is easier to express that way.
  4. If the site needs a prior session, load saved cookies before navigating.
  5. Capture the result: read the element text, take a screenshot, or wait for a download event.
  6. If data arrives via background calls, listen for the requests and responses rather than scraping the rendered DOM.

Each step has its own question below; the sequence is the part that is easy to get wrong.

Pre-run checklist

  • Have you decided whether you need a wait, a better locator, or both?
  • Are cookies or a saved session required before the first navigation?
  • Can you skip heavy resources without losing the data you need?
  • Is the data in the DOM, or only in a background response?
  • For downloads, are you listening for the download event before triggering it?
  • Running inside a notebook? Confirm the async setup works there before debugging selectors.

Browse the questions

Each entry below opens a focused answer for that task. The list covers page-load waiting, ownership and purpose of the tool, resource blocking, background request capture, file downloads, CSS and XPath selection, local files, Jupyter notebooks, cookies, and screenshots.

Which Playwright task do you need?
Your problemCapability to reach forTypical symptom if skipped
Page content missing on first readWait for page loadElement not found or empty text
Element present but not matchedCSS or XPath selectorLocator resolves to nothing or the wrong node
Site requires prior sessionSave and load cookiesRedirected to login or consent wall
Run is slow or bandwidth-heavyBlock resourcesLong runtimes loading unused assets
Data only appears via XHR/fetchCapture background requests and responsesRendered DOM lacks the values you need
Need a file from the pageDownload handlingClick happens but no file is saved
Need a visual recordScreenshotNo artifact to compare or review
Working from a local HTML fileLoad local filesNavigation to a file path fails
Prototyping interactivelyRun in Jupyter notebooksAsync loop errors in the notebook
Need to know who maintains itOwnership and purposeUnclear support and roadmap

Task mapping is a reader aid built from the question topics on this page; it is not a benchmark of speed or reliability.

Frequently asked questions

Do I need a wait before every selector?

No. Add a wait when the element is rendered after load or the document is still loading. If the element is already in the DOM and your locator is wrong, a wait will not help — fix the selector instead.

Should I use CSS or XPath selectors?

Both are supported. CSS is usually shorter for class, id and attribute matching; XPath is useful when you need to walk up from a node or match by text and position. Pick whichever expresses your target most readably.

Why save cookies instead of logging in each run?

Saving cookies lets you reuse an authenticated or consented session across runs, which avoids repeating the login flow and reduces the chance of tripping rate limits on the sign-in step.

When should I capture background requests instead of scraping the DOM?

When the values you want are delivered by XHR or fetch calls and are not reliably present in the rendered markup. Listening to responses gives you the structured payload directly.