Skip to content

Scrolling and Loading More Content in a JavaScript Scenario

Lazy loading hides content until the browser believes someone is looking at it.

OUTPUT FORMATSHTMLJSONMarkdown

A JavaScript scenario can imitate that behavior: move an inner container to its bottom, keep scrolling the page while it grows, or press a Load more control until it disappears. The three recipes below cover those cases. Each one needs a CSS selector or timing value adjusted for the target page, and every scenario shares one hard ceiling: 40 seconds of execution, after which the request times out.

Pick the pattern that matches the page

Start by watching how the page behaves in a normal browser. If new rows appear inside a table, feed, or chart while you scroll that box, use the inner-element pattern. If the whole document keeps getting taller as you move down, use continuous scrolling. If a visible button adds the next batch, use the click loop. The three patterns are not interchangeable: scrolling the window does nothing for a fixed-height container, and clicking a button that never appears simply exits.

Pattern 1: scroll an inner element to its end

Tables, graphs, and similar widgets sometimes hold content that only renders once the container itself is scrolled. The scenario sets the container's scrollTop to its scrollHeight, which jumps it to the bottom in one step. Replace '#example' with the CSS selector of the scrollable element, not of the page body. If the container loads in stages, a single jump may not be enough; in that case move to the continuous approach and target the element instead of the window.

Pattern 2: keep scrolling while the page grows

When content arrives in batches as the page moves, a repeating scroll is more reliable than one large jump. The script opens with a wait, then starts an interval that scrolls down 1500 pixels every 700 milliseconds. After each step it compares the document height with the previous reading. A change resets the stability counter; no change for 5 seconds clears the interval, and a final wait lets the last batch settle. Two values deserve attention: the 1500-pixel step and the 700-millisecond interval. A smaller step with a longer interval is gentler on slow pages, but it also consumes more of the 40-second budget.

Pattern 3: click Load more until it is gone

Some pages never grow on scroll; they wait for a button press. The loop looks up the selector stored in s, stops when the element is missing or its computed display is none, otherwise scrolls it into view, clicks, waits one second, and tries again. Set s to the button's CSS selector. The one-second pause is what gives the next batch time to appear; shortening it risks clicking a button that is still disabled or detached.

Stay inside the 40-second execution limit

The scenario has a maximum execution time of 40 seconds, and requests that exceed it time out. That cap covers every step in the scenario, including the leading and trailing waits. A rough budget helps: two 3-second waits plus a 5-second stability stop already use 11 seconds before any scrolling happens, leaving under 30 seconds for the loop itself. If a page needs more time than that, split the work across requests or reduce the waits rather than letting the request fail.

Before you extract, confirm the content arrived

A scenario that finishes without error has not necessarily loaded everything. Check that the expected number of rows or items is present, that the Load more control is truly absent, and that the document height stopped changing. If the selector matched nothing, the click loop exits immediately and looks successful. If the page grew but the new items sit in a shadow root or an iframe, scrolling the main document will not reach them.

Choosing a scrolling pattern
Page behaviorPatternWhat the selector points atStops when
New content appears inside a scrollable box such as a table or chartInner-element scrollThe scrollable containerThe container is jumped to its bottom once
The document keeps getting taller as you move downContinuous scrollingNot applicable; the script scrolls the windowNo height change for 5 seconds
A Load more or Show more button adds the next batchClick loopThe buttonThe button is missing or display is none

Pattern choice depends on how the target page actually loads content; verify behavior in a browser before running the scenario.

Timing values in the continuous scrolling script
ValueRoleEffect of changing it
1500 pixelsScroll step per intervalSmaller steps reach the bottom more slowly
700 millisecondsInterval between scroll stepsLonger intervals give slow pages more time but use more of the 40-second budget
5000 millisecondsStability threshold before the loop stopsA higher value waits longer for late content
3000 milliseconds (leading and trailing)Wait before and after the loopExtra settling time counts against the 40-second limit

Values are the defaults shown in the script; adjust them to the target page while keeping total execution under 40 seconds.

Frequently asked questions

How long can a JavaScript scenario run?

Execution is capped at 40 seconds. Anything still running at that point causes the request to time out, so waits and loops must fit inside that window.

Why did my continuous scroll script stop early?

The loop ends when the document height has not changed for 5 seconds. If the page loads slowly, raise the stability threshold or lengthen the interval; if the page never grows at all, the pattern is wrong for that site.

The click loop finished instantly. What went wrong?

Most likely the selector matched nothing, so the loop broke on its first check. Confirm the button's CSS selector and that it is visible when the scenario starts.

Can I scroll a table inside the page instead of the whole window?

Yes. Point the inner-element pattern at the table's scrollable container and set its scrollTop to its scrollHeight. Scrolling the window will not move a fixed-height container.

{
  "evaluate": "document.querySelector('#example').scrollTop = document.querySelector('#example').scrollHeight"
}