Skip to content

Scraping infinite scroll pages with Ruby

Many sites load a small first batch of items and add more only when the visitor scrolls.

OUTPUT FORMATSHTMLJSONMarkdown

A plain HTTP request captures just that first batch. This guide shows how to send scroll instructions from Ruby so the response includes additional items.

What changes when a page uses infinite scroll

A page can return its initial content without any scrolling. On the demo page used here, that first response contains 9 boxes. The remaining boxes are not present until the page is scrolled. If your Ruby request stops at the first response, you get the preloaded set only.

A scroll-enabled request tells the page to move down or across and then waits before the response is captured. The example below scrolls twice and waits 500 milliseconds after each scroll, which produces 18 blocks in the saved HTML.

Decision checklist: plain request or scroll request

Use a plain request when the page already contains every item you need in its first response. Use a scroll-enabled request when the page loads more items only after a scroll event.

Check these points before choosing:

  • Does the first response contain the full set, or only a preloaded batch?
  • Does the page scroll vertically, horizontally, or both?
  • How many scroll steps are needed to reach the target content?
  • How long should the request wait after each scroll so the new content can load?
  • Does the saved HTML contain the additional items you expected?

Request field map

The request uses these fields:

  • api_key: your API key.
  • url: the page to fetch.
  • js_scenario: a JSON object with an instructions array.
  • instructions: an ordered list of steps. Each step can be a scroll_y value, a scroll_x value, or a wait value.
  • scroll_y: vertical scroll distance in pixels.
  • scroll_x: horizontal scroll distance in pixels.
  • wait: pause in milliseconds before the next step.

The order matters. In the example, the sequence is scroll_y 1080, wait 500, scroll_y 1080, wait 500. More scroll_y steps can be added to capture further batches.

Worked example: from 9 boxes to 18 blocks

Start with a request that has no scroll_y parameter. Save the response body to an HTML file. When you open that file, you should see the 9 preloaded boxes.

Next, add a js_scenario with two scroll_y steps of 1080 pixels and a 500-millisecond wait after each. Send the request again and save the new response. The saved HTML should now contain 18 blocks. To capture more, add further scroll_y and wait pairs in the same instructions array.

Failure distinction: missing content vs. blocked request

If the saved HTML contains only the first batch, the page likely needs additional scroll steps or longer waits. If the request itself fails, the problem is at the HTTP level, not the scroll configuration. Check the error message and the request URL before changing the scroll instructions.

Frequently asked questions

Do I need both scroll_y and scroll_x?

Only if the page loads content in both directions. The example uses scroll_y for vertical scrolling. scroll_x is available for horizontal scrolling.

Why does the first request show only 9 boxes?

The page preloads 9 boxes. Additional boxes are loaded only after the page is scrolled. A request without scroll_y captures the preloaded set.

How do I capture more than 18 blocks?

Add more scroll_y instructions with waits between them. Each scroll step can trigger another batch.