Skip to content

How to handle infinite scroll pages in Go

Infinite scroll pages reveal content only after the viewport moves.

OUTPUT FORMATSHTMLJSONMarkdown

A plain HTTP request captures the first batch and nothing more. This guide shows how a Go program can send a scrolling scenario to a scraping API, wait for new items to appear, and write the resulting HTML to disk.

Why a single request stops at the first batch

Many sites reduce initial payload by rendering a small set of items and loading more as the visitor scrolls. On the demo page used here, the first response contains 9 boxes. Scrolling to the bottom triggers another 9, and the cycle repeats. Without a scroll instruction, the response body reflects only the pre-loaded blocks, so any extraction step sees a partial page. The fix is not a different parser; it is a request that tells the browser layer to move down the page before the HTML is captured.

Baseline request: capture the initial HTML

The first program builds a GET request with an API key and a URL-encoded target, reads the response body, and writes it to infinite.html. That file is useful as a control: it shows exactly what the page returns before any scrolling. If the target page has infinite scroll, this baseline will contain only the first batch. Use it to confirm the target URL, the API key, and the file-writing path before adding scroll behavior.

Add scroll_y and wait steps through js_scenario

The scrolling version adds a js_scenario parameter. Its value is a JSON object with an instructions array. Each instruction is either a scroll_y step, which moves the page vertically by a pixel amount, or a wait step, which pauses for a number of milliseconds. In the example, the sequence is scroll_y 1080, wait 500, scroll_y 1080, wait 500. The JSON is URL-encoded before it is appended to the request URL, just like the target URL. The wait steps give the page time to request and render the next batch before the response is captured.

What the expanded result looks like

With two scroll_y instructions and two waits, the saved HTML contains 18 blocks instead of 9. Each scroll triggers another batch of 9, so the count grows in steps. To capture more, add more scroll_y instructions, keeping a wait after each one. The pattern is additive: the number of scroll steps, not the pixel distance alone, determines how many batches are requested. A larger scroll_y value moves farther per step, but the page still needs a pause to load the next set.

Checklist before you run the Go program

  1. Confirm the target page actually uses scroll-triggered loading; a static page needs no scenario. 2. Start with the baseline request and inspect the saved file for the first batch. 3. Add one scroll_y step and one wait step, then compare the block count. 4. Increase the number of scroll_y steps until the saved HTML contains the content you need. 5. Keep the API key out of source control and out of shared logs. 6. If the count does not grow, check whether the wait is long enough for the page's network requests to finish.

Field and control map

The request URL carries three relevant inputs: api_key identifies the account, url is the encoded target page, and js_scenario is the encoded instruction list. Inside js_scenario, scroll_y moves the viewport down by a pixel value, scroll_x moves it horizontally, and wait pauses for a millisecond value. The example uses only scroll_y and wait. The response body is the HTML after the scenario runs, which the Go program writes to a file. The protected code modules code0001 and code0002 accompany this page with the exact request-building and file-saving routines.

Common failure distinctions

If the saved HTML matches the baseline, the scenario may not have been sent or the page may not use scroll-triggered loading. If the count grows once but stops, the number of scroll_y steps is likely too low. If the count grows inconsistently, the wait may be shorter than the page's loading time. If the request fails before any HTML is saved, check the API key and URL encoding rather than the scroll values. These are separate failure modes and should be tested one at a time.

Frequently asked questions

Do I need scroll_x as well as scroll_y?

Only if the page loads content when moved horizontally. The demo page loads more blocks when scrolled vertically, so scroll_y is sufficient there. scroll_x is available for pages that reveal content along the horizontal axis.

How many scroll steps do I need?

There is no fixed number. Each scroll_y instruction triggers one additional batch on the demo page. Start with one step, compare the saved block count with the baseline, and add steps until the HTML contains the content you need.

Why is the wait step important?

The page needs time to request and render the next batch after a scroll. The example waits 500 milliseconds after each scroll_y instruction. If the saved HTML grows inconsistently, a longer wait may help.

Can I save the result to a different file name?

Yes. The file path is passed to the save function. The example uses infinite.html, but any writable path works.