Skip to content

Puppeteer Scraping: Build a Browser Scraper for Dynamic Pages

Puppeteer lets a Node.js program drive a real Chromium browser, so pages whose content appears only after JavaScript runs can be opened, interacted with, and read as the finished DOM. This guide walks through setup, navigation, extraction, pagination, and export, then explains where local browser automation struggles and how a managed API can be combined with the same workflow.

When Puppeteer is the right tool

Use Puppeteer when the data you need is created in the browser rather than present in the first HTML response. A plain HTTP fetch can miss prices, reviews, lazy-loaded lists, and modal content that only appear after scripts run. A headless browser waits for rendering and exposes the final DOM, and it can also dismiss consent dialogs, type into search fields, scroll to load more items, trigger hover states, and capture screenshots.

A practical decision rule: start with a lightweight request scraper for static HTML; switch to Puppeteer when interaction or rendered content is required; consider a managed API when blocks, proxy rotation, or scale become the main problem.

Set up Node.js and Puppeteer

Install Node.js first, then confirm the runtime and package manager are available (module code0002). Create a project folder, initialise a package file (module code0003), and install Puppeteer (module code0004). The walkthrough uses a Wikipedia article as a safe test target and an index.js file for the workflow.

Launch, navigate, and verify

Launch Chromium in headless mode and open a new page (modules code0007 and code0008). Navigate to the target URL with a network-idle wait condition and a timeout so a stalled page fails instead of hanging forever (module code0009). Log the HTTP status and take a full-page screenshot to verify what the browser actually rendered (module code0010). Run the script with module code0012 and check for a successful status before adding extraction.

Extract with $eval, $$eval, or XPath

Choose the selector method by shape of the data:

  • $eval for one element, such as the page heading (module code0013)
  • $$eval for many matching elements, such as all section headings (module code0014)
  • XPath when a path expression is clearer than a CSS selector, for example the first non-empty paragraph (module code0015)

Trim text content to remove surrounding whitespace. If a selector returns nothing, treat that as a signal to inspect the rendered page rather than assuming the data is absent.

Paginate without infinite loops

Wrap extraction in a loop and look for a next-page link after each pass. If the link exists, wait for navigation and click it; if it does not, break out of the loop (modules code0016 and code0017). In the example, following a link to a related article demonstrates the pattern, but the selector must be changed for each real site. Always keep a timeout on navigation so a missing link or stalled page does not loop indefinitely.

Save results to CSV and JSON

Import the Node.js file system module (module code0018), collect results in an array (module code0019), and push each extracted value into it alongside the console output (module code0020). After the loop, build a CSV string with a header row and write both logs.csv and logs.json (module code0021). The complete assembled script is provided as protected module code0001.

Challenges at scale

Local Puppeteer scraping has predictable limits:

  • Performance: each instance runs a full Chromium browser, using substantial CPU and memory even in headless mode
  • Speed: full rendering adds seconds per page compared with a lightweight HTTP request
  • Bot detection: default headers, missing user activity, and rapid navigation can trigger blocks or CAPTCHAs
  • Consistency: infinite loaders and dynamic DOM updates can cause timeouts or partial loads
  • Scalability: running many browser instances needs sandbox flags, disk space, and dependency care in CI/CD

A useful checklist before scaling: measure memory per page, cap concurrency, set every wait to a timeout, randomise delays and headers where appropriate, and plan proxy rotation for protected targets.

Combining Puppeteer with a managed API

The source workflow shows a direct Puppeteer visit to a protected directory site returning HTTP 403, while the same flow routed through a managed HTML API succeeded. The API example keeps the Puppeteer logic and changes only the destination URL, adding parameters that enable a stealth proxy and JavaScript rendering (modules code0023 and code0024). Use {{OWNER_API_BASE_URL}} as the configured origin and {{OWNER_API_KEY}} for the reader's own account key. Keep existing selectors and parsing logic, and let the API handle rendered HTML and proxy rotation. Any trial or pricing terms must be confirmed via {{OWNER_TRIAL_TERMS}} before publication.

Extraction method map
MethodUse it whenExample in this guide
$evalOne elementPage heading
$$evalMany elementsSection headings
XPathPath expression is clearerFirst non-empty paragraph

Selector choice depends on the target page structure; verify against the rendered DOM.

Frequently asked questions

Can I scrape any website with Puppeteer?

Often yes for public pages, but aggressive anti-bot controls can block default automation fingerprints. Custom headers, delays, proxies, or routing through a managed API may be needed, and only public data should be targeted.

Is Puppeteer scraping fast?

It is slower than raw HTML fetching because each page must load and execute JavaScript. That overhead adds seconds per page and becomes harder to sustain at high concurrency.

Do I need proxies with Puppeteer?

For protected sites, proxies are commonly needed because repeated requests from one address can trigger rate limits or CAPTCHAs. Rotating proxies and geo-targeting can be automated through a managed service instead of manual setup.

What is the difference between Puppeteer and a managed scraping API?

Puppeteer is a self-hosted browser automation library. A managed scraping API is a hosted service returning rendered HTML with proxy handling. The two can be combined by keeping Puppeteer selectors and flow while sending target URLs through the API.