Skip to content

Scrape Infinite Scroll Pages in Node.js: scroll_y vs infinite_scroll

Pages that append content as you scroll return only their first batch to a plain request, because nothing triggers the next load.

OUTPUT FORMATSHTMLJSONMarkdown

This guide shows how to drive a JavaScript Scenario from Node.js so the page keeps loading, using the Quotes to Scrape scroll page as the practice target. You will see the no-scroll baseline, a manual pixel-scroll approach, and the automatic bottom-scroll instruction, plus how to tell the three common failure modes apart.

What changes when a page scrolls instead of paginating

Traditional pagination gives you a next-page link; infinite scroll replaces that with a fetch triggered by position on the page. A request that never scrolls therefore captures only the initial batch. The plan here is two-stage: first confirm the page renders at all, then add scroll instructions so later batches arrive before the response is returned.

Setup and prerequisites

You need Node.js 24 or later and an API key for the scraping service, which you can find in its dashboard after creating an account. Initialize a project and install the Node.js SDK; the exact install commands are supplied as a protected module and should be used verbatim rather than retyped.

Step 1: Confirm the baseline without scrolling

Send the request with a wait condition on the quote element so the response is not returned before the first batch renders. The script writes the HTML to a file and counts quote blocks. On the practice page this baseline yields 10 quotes, which is the expected no-scroll result, not a bug. Replace the placeholder API key with your own before running.

Step 2: Manual scrolling with scroll_y

The scroll_y instruction moves the viewport down by a pixel count you choose, so you control each step. A working pattern is a large scroll, a pause, then another scroll and pause. Two settings matter here: resource blocking must be turned off so the page's CSS and images load, and the total scenario time must stay within 40 seconds. Without CSS the initial quotes collapse into a short block that fits the window, the page cannot scroll, and the count stays at 10. With the setting corrected, the captured run returned all 100 quotes; if yours returns fewer, lengthen the waits or add more scroll-and-wait pairs while staying inside the time limit.

Step 3: Automatic scrolling with infinite_scroll

The infinite_scroll instruction repeatedly jumps to the bottom of the page with a pause between jumps. Two parameters shape it: max_count caps how many scrolls occur, and delay sets the pause in milliseconds. The captured run with max_count 3 and delay 2000 returned all 100 quotes, though results depend on how fast each batch loads; raise delay or max_count if you fall short.

Choosing between the two instructions

Use scroll_y when you need pixel-level control or want to stop at a specific depth. Use infinite_scroll when you simply want the page driven to its end and are happy to cap the number of jumps. Both require resource loading to be enabled for this page and both must finish inside the 40-second scenario window.

Failure checklist

Zero quotes: the wait selector never matched, usually because the page had not rendered its first batch. Stuck at 10 quotes: CSS was blocked, so the page could not scroll. Fewer than expected after scrolling: waits or delay too short, max_count too low, or the scenario hit the 40-second ceiling. Check these in order before changing anything else.

Field and control map

wait_for: CSS selector the response waits for before returning. block_resources: set to false to load CSS and images. js_scenario.instructions: ordered list of actions. scroll_y: pixel offset to scroll down. wait: pause in milliseconds. infinite_scroll.max_count: maximum number of bottom scrolls. infinite_scroll.delay: pause between bottom scrolls in milliseconds.

Frequently asked questions

Why does my request return only 10 quotes?

That is the no-scroll baseline: the response is captured before any scrolling triggers the next batch. Add a scroll_y or infinite_scroll instruction, and make sure block_resources is false so the page can actually scroll.

Why does scrolling still not load more quotes?

If resource blocking stays on, the page's CSS is missing, the initial quotes fit inside the window, and no scroll event fires. Turning block_resources off is the usual fix.

How long can a JavaScript Scenario run?

The whole scenario must finish within 40 seconds, so total wait and delay time has to fit inside that window. If you need more scroll steps, shorten individual pauses rather than exceeding the limit.

Should I use scroll_y or infinite_scroll?

scroll_y gives you pixel-level control and lets you stop at a chosen depth. infinite_scroll drives to the bottom repeatedly and is capped by max_count, with delay controlling the pause between jumps.