Skip to content

How to Handle Infinite Scroll Pages in Python

Infinite scroll loads content in batches as the visitor moves down the page, so a single request often captures only the first batch.

OUTPUT FORMATSHTMLJSONMarkdown

This guide shows how to drive the page with a js_scenario built from scroll_y and wait instructions, then save the fuller HTML for parsing. The demo page starts with 9 boxes; after two scroll-and-wait steps, 18 boxes were present.

Why a plain fetch returns only the first batch

Infinite scroll is a loading technique that reduces the amount of data sent to a client up front. The server delivers an initial set of items, and more arrive only when the browser reaches the bottom of the visible area. A request that does not scroll therefore sees the initial set and nothing more. On the demo page used here, that initial set is 9 boxes. Without a scroll_y instruction, extracting the full list is not practical because the remaining items were never requested by the page.

The two instruction fields that matter

The js_scenario parameter accepts an instructions list. Each entry is a small object. Two fields do the work in this example:

  • scroll_y — a vertical scroll distance in pixels. The default window height in the service is 1080 pixels, so a value of 1080 moves roughly one viewport down. Adding another scroll_y entry continues from the new position.
  • wait — a pause in milliseconds. A value of 500 pauses for half a second, giving the page time to fetch and render the next batch before the next instruction runs.

A scroll_x field exists for horizontal movement, but the demo page scrolls vertically, so scroll_y is the relevant one here. Pairing each scroll with a wait is what makes the sequence reliable: the scroll triggers the load, and the wait gives it time to finish.

Worked example: from 9 boxes to 18

The demo page is a simple HTML file at the demo base URL. A first request without scrolling returns the 9 pre-loaded boxes. Adding two scroll_y steps of 1080 pixels, each followed by a 500 ms wait, produced 18 boxes in the saved HTML. That is two batches of 9, matching the page's behavior of loading 9 more each time the bottom is reached.

To go further, repeat the scroll-and-wait pair. Each additional pair should trigger another batch, provided the page still has items to load. The exact number of pairs you need depends on how many batches the page serves and how tall each batch is.

Checklist before you add more scrolls

Use this checklist to decide what to change when the result is short:

  1. Did you include scroll_y at all? If not, you will only ever see the initial batch.
  2. Did each scroll_y have a wait after it? A scroll without a pause can fire before the next batch renders.
  3. Is the scroll distance at least one viewport? The default height is 1080 pixels; a smaller value may not reach the trigger point.
  4. Did the page stop growing? If two consecutive scroll-and-wait pairs add nothing, the page has likely reached its end.
  5. Are you saving the HTML after the last wait? The saved file reflects the page state at the moment the response completes.

Two different short-result failures

A short result can come from two distinct causes, and they need different fixes. If scroll_y is missing entirely, the page never asks for more items, so you get exactly the initial batch — 9 boxes on the demo. If scroll_y is present but the wait is too short, the scroll fires and the page starts loading, but the response may be captured before the new items appear. The first case is a missing instruction; the second is a timing problem. Check which one you have before adding more scroll steps, because adding scrolls will not fix a missing scroll_y, and adding waits will not help if no scroll is present.

Frequently asked questions

What does scroll_y do in a js_scenario?

scroll_y moves the page vertically by the number of pixels you specify. In the example, 1080 pixels matches the default window height, so each instruction moves about one viewport down and triggers the next batch of items to load.

Why is a wait instruction needed after each scroll?

The scroll tells the page to load more, but the loading takes time. A wait of 500 milliseconds pauses the scenario for half a second so the new items can render before the next instruction or the final save. Without a wait, the captured HTML may still show the previous batch.

How many scroll_y instructions do I need?

There is no fixed number. Each scroll-and-wait pair loads one more batch on a page that behaves like the demo. Add pairs until two consecutive pairs stop adding new items, which indicates the end of the content.

Can I scroll horizontally instead?

Yes. The scenario supports scroll_x for horizontal movement. Use it when the page loads items as you move sideways rather than down. The demo page scrolls vertically, so scroll_y is the field used in the example.