Common questions about web scraping with Playwright
Playwright drives a real browser, so most scraping problems come down to timing, locating the right element, or keeping a session alive between runs.
This page is a question index: each entry points to a focused answer for one task. Use the decision table below to work out which capability you need, then open the matching question.
Start with the task, not the API
Before opening a specific question, name the problem you actually have. Scraping failures with a browser automation tool usually fall into one of five buckets:
- The page or element is not ready yet (timing).
- The element exists but your locator does not match it (selection).
- The site expects a prior session, login or consent state (cookies).
- You need something other than text — an image, a file, or a network response (capture).
- The run is slow or noisy because it loads assets you do not need (performance).
Matching your problem to a bucket first saves you from rewriting selectors when the real issue is a missing wait, or adding waits when the selector is simply wrong.
Timing versus selection: a distinction worth getting right
These two failures look identical in a stack trace but need opposite fixes.
A timing failure means the document is still loading or the element is rendered later by script. The fix is to wait for the page or the element before interacting.
A selection failure means the element is already there, but your CSS or XPath expression points somewhere else — a different sibling, a stale class name, or a container that no longer exists. The fix is to inspect the DOM and correct the locator.
A quick test: if you pause the script manually and the element appears, it was timing. If it still is not found, it is selection.
A minimal sequence using captured capabilities
The questions on this page cover the pieces of a typical scrape. Put together in order, a simple run looks like this:
- Open the page and wait until it has loaded (see the page-load question).
- Optionally block resource types you do not need, such as images or fonts, to speed the run up.
- Locate the elements you want with a CSS selector, or with XPath when the structure is easier to express that way.
- If the site needs a prior session, load saved cookies before navigating.
- Capture the result: read the element text, take a screenshot, or wait for a download event.
- If data arrives via background calls, listen for the requests and responses rather than scraping the rendered DOM.
Each step has its own question below; the sequence is the part that is easy to get wrong.
Pre-run checklist
- Have you decided whether you need a wait, a better locator, or both?
- Are cookies or a saved session required before the first navigation?
- Can you skip heavy resources without losing the data you need?
- Is the data in the DOM, or only in a background response?
- For downloads, are you listening for the download event before triggering it?
- Running inside a notebook? Confirm the async setup works there before debugging selectors.
Browse the questions
Each entry below opens a focused answer for that task. The list covers page-load waiting, ownership and purpose of the tool, resource blocking, background request capture, file downloads, CSS and XPath selection, local files, Jupyter notebooks, cookies, and screenshots.
| Your problem | Capability to reach for | Typical symptom if skipped |
|---|---|---|
| Page content missing on first read | Wait for page load | Element not found or empty text |
| Element present but not matched | CSS or XPath selector | Locator resolves to nothing or the wrong node |
| Site requires prior session | Save and load cookies | Redirected to login or consent wall |
| Run is slow or bandwidth-heavy | Block resources | Long runtimes loading unused assets |
| Data only appears via XHR/fetch | Capture background requests and responses | Rendered DOM lacks the values you need |
| Need a file from the page | Download handling | Click happens but no file is saved |
| Need a visual record | Screenshot | No artifact to compare or review |
| Working from a local HTML file | Load local files | Navigation to a file path fails |
| Prototyping interactively | Run in Jupyter notebooks | Async loop errors in the notebook |
| Need to know who maintains it | Ownership and purpose | Unclear support and roadmap |
Task mapping is a reader aid built from the question topics on this page; it is not a benchmark of speed or reliability.
Frequently asked questions
Do I need a wait before every selector?
No. Add a wait when the element is rendered after load or the document is still loading. If the element is already in the DOM and your locator is wrong, a wait will not help — fix the selector instead.
Should I use CSS or XPath selectors?
Both are supported. CSS is usually shorter for class, id and attribute matching; XPath is useful when you need to walk up from a node or match by text and position. Pick whichever expresses your target most readably.
Why save cookies instead of logging in each run?
Saving cookies lets you reuse an authenticated or consented session across runs, which avoids repeating the login flow and reduces the chance of tripping rate limits on the sign-in step.
When should I capture background requests instead of scraping the DOM?
When the values you want are delivered by XHR or fetch calls and are not reliably present in the rendered markup. Listening to responses gives you the structured payload directly.