Skip to content

Puppeteer Web Scraping in Node.js

Puppeteer is a JavaScript library from Google that gives you a programmatic handle on Chrome and Firefox. You can start a browser, open a tab, navigate, type, click, screenshot, and read the rendered DOM. This guide walks through a small but realistic Node.js scraper: it opens a subreddit, searches for a term, waits for client-rendered results, scrolls to load more, and prints post titles and links as JSON.

Getting started: Node.js and the Puppeteer package

You need Node.js 22 or newer and a working npm installation. Create a project directory and install the puppeteer package. The package brings its own Chrome setup, so you do not need a separate browser download. If you prefer to point Puppeteer at an existing browser, install puppeteer-core instead and pass executablePath to its launch() method.

The setup commands are shown in the protected setup module code0001. After installation, you can write your first script.

First step: capture a screenshot

A screenshot is a good smoke test: it proves the browser launched, the page loaded, and the page handle is working. The example script in code0002 opens the MovieMistakes subreddit, saves a PNG, and closes the browser. Save it as index.js and run it with node index.js. A Chrome window should appear briefly, then a file named moviemistakes.png should appear in the project directory.

The script has six moving parts: import the library, launch the browser, open a new tab, navigate with goto(), capture with screenshot(), and close with browser.close(). Setting headless to false makes the window visible, which is useful while debugging. The import statement is used instead of require() because ES modules are now mainstream.

Scraping Reddit: search, wait, and collect

The goal is to extract post titles and links for a search term. The scraper loads the MovieMistakes subreddit, uses the search box at the top, submits the query, waits for results, scrolls to load more posts, and returns the data as JSON.

Reddit uses custom web components, so the search input lives inside a shadow DOM. Puppeteer's >>> operator pierces that boundary. The selector reddit-search-large >>> input[name="q"] targets the search field, and Page.type() enters the term. The example uses 'comedy'. Pressing Enter submits the search, as shown in code0003 and code0004.

Because the search page is generated on the client side, results may take a few seconds to appear. Page.waitForSelector() pauses until the results container exists; the example waits up to 10 seconds (code0005).

Reddit also uses infinite scrolling. The helper functions in code0006 extract titles and URLs, scroll to the bottom, wait for the page height to grow, and repeat until 100 items are collected or the loop exits. The delay() helper gives the page time to render. The extraction selector is #main-content > div > search-telemetry-tracker a[data-testid="post-title-text"].

Finally, code0007 calls scrapeItems(), closes the browser, and prints the array. The full assembled script is in code0008, and code0009 shows the run command. A sample of the output shape appears in code0010: each item has a title and a url.

Decision checklist before you run the scraper

Use this checklist to avoid the most common setup mistakes:

  • Node.js 22 or newer is installed and npm works.
  • You chose puppeteer (bundled Chrome) or puppeteer-core (existing browser plus executablePath).
  • The project uses ES modules (type="module") so the import statement works.
  • The search selector matches the current page structure, including the shadow-DOM boundary.
  • The waitForSelector timeout is long enough for client-rendered results.
  • The infinite-scroll loop has a stop condition (the example stops at 100 items).
  • The browser is closed in every path, including errors.

Field and control map for the script

The table below maps each script action to its Puppeteer call and the input or output it produces. Use it as a quick reference when adapting the example to another site.

ActionPuppeteer callInputOutput
Launch browserpuppeteer.launch()options such as headlessbrowser handle
Open tabbrowser.newPage()nonepage handle
Navigatepage.goto()URLloaded page
Type into searchpage.type()selector and texttext entered
Submit searchpage.keyboard.press()key namesearch submitted
Wait for resultspage.waitForSelector()selector and timeoutelement present
Extract itemspage.evaluate()extraction functionarray of items
Scrollpage.evaluate()scroll scriptnew page height
Closebrowser.close()nonebrowser closed

Telling failures apart

Three failures look similar but have different causes. A waitForSelector timeout means the results container never appeared; check the selector and increase the timeout. A shadow-DOM selector miss means the >>> path or the input name changed; inspect the page and update the selector. An infinite-scroll loop that exits early usually means the page height stopped growing or the item count never reached the target; check the stop condition and the delay. In all three cases, the browser should still be closed so the process can exit.

Summary

Puppeteer gives you full control over a browser from JavaScript. A few lines can launch Chrome, navigate, type, wait, scroll, and extract structured data. That makes it a better fit than plain HTTP libraries for JavaScript-heavy sites and single-page applications. The example in this guide is intentionally small, but the same pattern scales: launch, navigate, interact, wait, extract, close.

Rate limits, throttling, IP blocks, and other anti-scraping measures can still stop a scraper. A managed scraping API is one way to reduce that friction. If you want to try one, the signup page is linked in the source registry.

Frequently asked questions

Which Node.js version does this Puppeteer tutorial need?

Node.js 22 or newer with a working npm installation. The setup module code0001 creates a project directory, installs puppeteer, and sets the package type to module.

When should I use puppeteer-core instead of puppeteer?

Use puppeteer-core if you want to drive an existing browser installation rather than the Chrome that ships with the puppeteer package. In that case you pass executablePath to launch().

Why does the search selector use >>>?

Reddit places the search input inside a custom element's shadow DOM. The >>> operator lets Puppeteer cross that boundary so page.type() can reach input[name="q"].

How does the scraper handle infinite scrolling?

It scrolls to the bottom, waits for the page height to increase, pauses briefly, and repeats until it has collected 100 items or the loop exits. The helper functions are in code0006.

What does the final output look like?

An array of objects, each with a title and a url. The sample in code0010 shows two entries from the MovieMistakes subreddit.

[
  {
    title: 'Quicksand on Netflix',
    url: 'https://www.reddit.com/r/MovieMistakes/comments/e2mew7/quicksand_on_netflix/'
  },
  {
    title: 'Robert Downey Jr. Clipping Through a CGI Dragon in Dolittle (2020)',
    url: 'https://www.reddit.com/r/MovieMistakes/comments/nqd55i/robert_downey_jr_clipping_through_a_cgi_dragon_in/'
  },

  .... more data here