Scrape Infinite Scroll Pages With Puppeteer: Scroll, Wait, Extract
A plain HTTP fetch returns only the first view of a feed that grows on scroll. Puppeteer drives a real browser, so you can move down the page, let JavaScript fetch more records, and collect each rendered batch. This guide explains the loop, the timing checks that keep it reliable, and the lighter alternatives to try first when a hidden API is available.
The basic loop: scroll, wait for growth, extract
Static parsers miss records that only appear after JavaScript runs. A headless browser solves that by performing the same actions a visitor performs: open the page, scroll, pause, then read what has rendered. The core pattern has four parts. First, define an extraction function that runs inside the page and returns the current items. Second, scroll to the bottom. Third, wait until the document grows or a short delay passes. Fourth, extract again and repeat until you reach the target count or the height stops changing.
Before writing the loop, check your prerequisites: ES6 JavaScript, promises with async/await, and Node.js installed. Install Puppeteer with npm, which also brings a compatible Chromium build; Linux may need extra system dependencies. Keep every helper the extraction function needs inside that function itself, because it is serialized and executed in the browser context where Node.js variables are not available. Inspect the live page in the browser console to confirm the tags and classes you plan to select, and prefer stable hooks such as data-testid or aria-label over generated class names that can change between deployments.
Timing, exit conditions, and a pre-run checklist
Timing decides whether a run succeeds. After each scroll, wait for the page height to increase before extracting, and add a millisecond delay so slow fetches can finish. Log each iteration so you can see whether progress stalls. If the height stops changing within the wait window, end the loop cleanly instead of spinning forever. The same exit logic answers the common failure of scrolling without limit: stop at a target item count, a maximum number of attempts, or a flat scrollHeight.
For the runnable example, the script launches Chromium, opens a page at a fixed viewport, navigates to the demo feed, collects ten items, writes them to items.txt, and closes the browser. Setting headless to false lets you watch the scrolling while you debug. If the target uses a Load More button instead of automatic loading, click that button first with a short delay, then resume the scroll loop. View the output with the file-open command shown in the accompanying modules, or tail the last lines in a terminal.
A useful decision checklist before you run: confirm the selector still matches, confirm the wait condition matches how the site loads, set a target count, set a scroll delay, and decide what happens when height stops growing. That checklist prevents the two most common outcomes: extracting an empty first batch and looping indefinitely.
When a hidden API beats DOM scraping
Scrolling is not always the best way to get the data. Many feeds fire XHR or Fetch requests to hidden endpoints that return clean JSON. Listening for those responses lets you parse structured records directly, often faster and with more reliable ordering than reading changing DOM nodes. To find the endpoint, open the Network tab, scroll the page, and filter for XHR or Fetch traffic. Once you know the pattern, you may be able to call it with a plain HTTP client and skip the browser entirely. Cheerio remains a good fit when the full HTML already arrives in the initial response.
Use this comparison to choose: if the URL changes per page and no JavaScript is needed, a lightweight HTTP approach is fastest. If the URL stays fixed and content appears only after scrolling, use Puppeteer. If a JSON endpoint drives the feed, intercepting or calling that endpoint is usually cleaner than parsing markup. Puppeteer is the fallback when neither a stable URL nor a usable endpoint exists.
Keep long runs stable: memory, pacing, and detection
Long feeds accumulate thousands of DOM nodes, which can exhaust memory and crash the browser. Two controls keep a long run stable. Periodically remove already-collected elements from the DOM, keeping only the last few. And write results to disk in batches instead of holding everything in one array until the end. Together they keep memory use low across hundreds or thousands of scroll steps.
Performance and detection are related. Jumping straight to the bottom is both abrupt for lazy loading and a clear automation signal. Moving in smaller increments, for example a few hundred pixels at a time with a brief pause, lets the browser process content in smaller chunks and looks more like a person scrolling. Vary the delay and distance, use a realistic viewport, and wait for key resources before the next action. On aggressively protected sites, instant bottom jumps can trigger bot detection, so human-like pacing matters more than raw speed.
Trade-offs and failure modes to plan for
Infinite scrolling suits discovery browsing and small screens, and it keeps people engaged, but it slows the page as content piles up, makes it hard to return to a specific position, and can hide footer links. For scrapers, the practical risks are different: generated class names break selectors, failed requests stall loading, and excessive memory ends runs early. Defensive habits help: stable attributes, error handling around the loop, retries for failed steps, and a log entry per failed iteration. With those in place, the scroll-wait-extract loop becomes a dependable starting point for dynamic feeds.
| Check | Pagination | Infinite scroll |
|---|---|---|
| URL changes | Yes | No |
| JavaScript required | No | Yes |
| Best tool | Cheerio / HTTP client | Puppeteer |
| Speed | Fast | Slower |
URL and JavaScript behaviour determine the right tool; speed follows from that choice.
Frequently asked questions
How do I make Puppeteer scroll to the bottom of a page?
Run browser-context code that moves the viewport to the current document height, repeat that step in a loop, and wait after each move for the height to grow before extracting again.
Why is my Puppeteer script not loading infinite scroll content?
The usual cause is extracting too soon. Add a timeout or wait until the page height has increased, and use a millisecond delay when content loads slowly.
Can I scrape an infinite scroll page without Puppeteer?
Yes, when scrolling triggers background XHR or Fetch calls. Find that hidden endpoint in the Network tab and request its JSON directly with a simpler HTTP client.
Is emulating life-like scrolling necessary for scraping?
On heavily protected sites, yes. An instant jump to the bottom can look automated, so smaller steps and varied delays better resemble a person browsing.
How do I stop Puppeteer from scrolling forever?
Set an exit condition: a target item count, a cap on scroll attempts, or a check that document.body.scrollHeight has stopped increasing.
What causes infinite scrolling scrapers to stop unexpectedly?
Failed network requests, changed page structure, or excessive memory use can end a run. Wrap the loop in error handling, retry failed steps, and log each failed iteration.