Scrolling and Loading More Content in a JavaScript Scenario
Lazy loading hides content until the browser believes someone is looking at it.
A JavaScript scenario can imitate that behavior: move an inner container to its bottom, keep scrolling the page while it grows, or press a Load more control until it disappears. The three recipes below cover those cases. Each one needs a CSS selector or timing value adjusted for the target page, and every scenario shares one hard ceiling: 40 seconds of execution, after which the request times out.
Pick the pattern that matches the page
Start by watching how the page behaves in a normal browser. If new rows appear inside a table, feed, or chart while you scroll that box, use the inner-element pattern. If the whole document keeps getting taller as you move down, use continuous scrolling. If a visible button adds the next batch, use the click loop. The three patterns are not interchangeable: scrolling the window does nothing for a fixed-height container, and clicking a button that never appears simply exits.
Pattern 1: scroll an inner element to its end
Tables, graphs, and similar widgets sometimes hold content that only renders once the container itself is scrolled. The scenario sets the container's scrollTop to its scrollHeight, which jumps it to the bottom in one step. Replace '#example' with the CSS selector of the scrollable element, not of the page body. If the container loads in stages, a single jump may not be enough; in that case move to the continuous approach and target the element instead of the window.
Pattern 2: keep scrolling while the page grows
When content arrives in batches as the page moves, a repeating scroll is more reliable than one large jump. The script opens with a wait, then starts an interval that scrolls down 1500 pixels every 700 milliseconds. After each step it compares the document height with the previous reading. A change resets the stability counter; no change for 5 seconds clears the interval, and a final wait lets the last batch settle. Two values deserve attention: the 1500-pixel step and the 700-millisecond interval. A smaller step with a longer interval is gentler on slow pages, but it also consumes more of the 40-second budget.
Pattern 3: click Load more until it is gone
Some pages never grow on scroll; they wait for a button press. The loop looks up the selector stored in s, stops when the element is missing or its computed display is none, otherwise scrolls it into view, clicks, waits one second, and tries again. Set s to the button's CSS selector. The one-second pause is what gives the next batch time to appear; shortening it risks clicking a button that is still disabled or detached.
Stay inside the 40-second execution limit
The scenario has a maximum execution time of 40 seconds, and requests that exceed it time out. That cap covers every step in the scenario, including the leading and trailing waits. A rough budget helps: two 3-second waits plus a 5-second stability stop already use 11 seconds before any scrolling happens, leaving under 30 seconds for the loop itself. If a page needs more time than that, split the work across requests or reduce the waits rather than letting the request fail.
Before you extract, confirm the content arrived
A scenario that finishes without error has not necessarily loaded everything. Check that the expected number of rows or items is present, that the Load more control is truly absent, and that the document height stopped changing. If the selector matched nothing, the click loop exits immediately and looks successful. If the page grew but the new items sit in a shadow root or an iframe, scrolling the main document will not reach them.
| Page behavior | Pattern | What the selector points at | Stops when |
|---|---|---|---|
| New content appears inside a scrollable box such as a table or chart | Inner-element scroll | The scrollable container | The container is jumped to its bottom once |
| The document keeps getting taller as you move down | Continuous scrolling | Not applicable; the script scrolls the window | No height change for 5 seconds |
| A Load more or Show more button adds the next batch | Click loop | The button | The button is missing or display is none |
Pattern choice depends on how the target page actually loads content; verify behavior in a browser before running the scenario.
| Value | Role | Effect of changing it |
|---|---|---|
| 1500 pixels | Scroll step per interval | Smaller steps reach the bottom more slowly |
| 700 milliseconds | Interval between scroll steps | Longer intervals give slow pages more time but use more of the 40-second budget |
| 5000 milliseconds | Stability threshold before the loop stops | A higher value waits longer for late content |
| 3000 milliseconds (leading and trailing) | Wait before and after the loop | Extra settling time counts against the 40-second limit |
Values are the defaults shown in the script; adjust them to the target page while keeping total execution under 40 seconds.
Frequently asked questions
How long can a JavaScript scenario run?
Execution is capped at 40 seconds. Anything still running at that point causes the request to time out, so waits and loops must fit inside that window.
Why did my continuous scroll script stop early?
The loop ends when the document height has not changed for 5 seconds. If the page loads slowly, raise the stability threshold or lengthen the interval; if the page never grows at all, the pattern is wrong for that site.
The click loop finished instantly. What went wrong?
Most likely the selector matched nothing, so the loop broke on its first check. Confirm the button's CSS selector and that it is visible when the scenario starts.
Can I scroll a table inside the page instead of the whole window?
Yes. Point the inner-element pattern at the table's scrollable container and set its scrollTop to its scrollHeight. Scrolling the window will not move a fixed-height container.
{
"evaluate": "document.querySelector('#example').scrollTop = document.querySelector('#example').scrollHeight"
}