Scraping Infinite Scroll Pages from C# with a Scroll Scenario
Many sites send only a first batch of content and append more as the visitor moves down the page.
A plain HTTP request stops at that first batch. This guide shows how to drive the scroll from a C# client by attaching a JavaScript scenario to the request, using a demo page that starts with nine boxes and adds nine more each time the bottom is reached.
What changes when a page loads on scroll
>Infinite scroll is one of several ways sites reduce the amount of data sent in an initial response. The first response for the demo page contains nine boxes. The remaining boxes are created only after the viewport moves to the end of the document, and the same pattern repeats on each further scroll. That means the HTML you receive depends on what the browser did before the response was captured. If nothing scrolled, you get the opening batch and nothing else.
Two consequences follow for a C# client:
- A request that only passes the target URL is enough to confirm the page responds, but it is not enough to collect the appended items.
- The scroll has to be expressed as instructions the fetching service executes in the page, not as code in your C# process. Your program builds the instruction set and sends it as a request parameter.
Step 1: Confirm the baseline without scrolling
Start by requesting the demo page with no scroll instructions and writing the response to a file. The C# sample below builds a query string with the credential and the target URL, issues the request, and saves the body as InfiniteScroll_NoScroll.html. Reading that file shows the first nine pre-loaded blocks. Treat this run as your baseline: it tells you how much content arrives before any scrolling, which is exactly the amount a scroll-free request will always return on this kind of page.
This modern HttpClient alternative makes the same GET /api/v1/ request as the HttpWebRequest example, with api_key and url query fields. Set APP_BASE_URL, DEMO_BASE_URL and API_KEY; the demo origin must serve /infinite_scroll.html. The first call sends no js_scenario and saves the baseline HTML.
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Net;
using System.Net.Http;
using System.Text.Json;
using System.Threading.Tasks;
string apiOrigin = Environment.GetEnvironmentVariable("APP_BASE_URL")
?? throw new InvalidOperationException("Set APP_BASE_URL");
string demoOrigin = Environment.GetEnvironmentVariable("DEMO_BASE_URL")
?? throw new InvalidOperationException("Set DEMO_BASE_URL");
string key = Environment.GetEnvironmentVariable("API_KEY")
?? throw new InvalidOperationException("Set API_KEY");
using var handler = new HttpClientHandler
{
AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate
};
using var transport = new HttpClient(handler);
async Task<string> FetchPageAsync(string? scenario = null)
{
var fields = new Dictionary<string, string>
{
["api_key"] = key,
["url"] = demoOrigin.TrimEnd('/') + "/infinite_scroll.html"
};
if (scenario != null) fields["js_scenario"] = scenario;
string query = string.Join("&", fields.Select(field =>
Uri.EscapeDataString(field.Key) + "=" + Uri.EscapeDataString(field.Value)));
return await transport.GetStringAsync(apiOrigin.TrimEnd('/') + "/api/v1/?" + query);
}
await File.WriteAllTextAsync("before.html", await FetchPageAsync());Step 2: Add scroll and wait instructions
The second sample keeps the same request shape and adds a js_scenario value containing four instructions: scroll down by 1080, wait 500 milliseconds, scroll down by 1080 again, then wait 500 milliseconds once more. The result is written to InfiniteScroll_WithScroll.html and contains eighteen blocks, double the baseline. Each scroll step moves the viewport; each wait step gives the page time to append the next batch before the response is captured.
The pattern scales by repetition rather than by a single large jump: add another scroll_y and wait pair for each additional batch you need. The horizontal counterpart, scroll_x, follows the same instruction format when a page loads content sideways.
Use FetchPageAsync and the imports from the baseline example. The ordered instructions retain two 1080-pixel scrolls, each followed by a 500 ms wait. JsonSerializer creates valid JSON for the js_scenario query value, and the second result is saved separately for comparison.
var scenario = new
{
instructions = new object[]
{
new { scroll_y = 1080 },
new { wait = 500 },
new { scroll_y = 1080 },
new { wait = 500 }
}
};
await File.WriteAllTextAsync("after.html",
await FetchPageAsync(JsonSerializer.Serialize(scenario)));Request fields used in the example
The two samples touch four request parameters. Map them before adapting the code:
- api_key — carries the credential; the sample holds it in a constant named API_KEY.
- url — the page to fetch; the demo target ends in infinite_scroll.html.
- js_scenario — a JSON instruction list; the working example uses scroll_y and wait entries.
- The base endpoint — the sample appends the query string to a base URL constant.
Keep the credential out of source control and supply it from configuration or environment at run time.
Checklist before you adapt the code
- Decide the direction: scroll_y for vertical feeds, scroll_x for horizontal ones.
- Count the batches you need and add one scroll plus wait pair per batch beyond the first.
- Keep the wait long enough for the site's own loading to finish; 500 ms is what the demo uses.
- Save each run to a separate file so you can compare baseline and scrolled output.
- Confirm the extra items are actually present in the saved HTML before parsing them.
Telling two failures apart
If the saved file holds only the opening batch, the fetch itself worked and the scroll instructions did not produce more content — check the scenario value, the direction, and whether the waits are long enough. If the request throws or returns an error instead of HTML, the problem is upstream of scrolling: the credential, the endpoint, or the target URL. Separating these two cases saves time, because adding more scroll steps cannot fix a request that never succeeded.
| Parameter | Role in the request | Example value |
|---|---|---|
| api_key | Credential for the request | held in the API_KEY constant |
| url | Target page to fetch | the demo infinite_scroll.html page |
| js_scenario | Instruction list executed in the page | scroll_y 1080, wait 500, scroll_y 1080, wait 500 |
Values reflect the two captured samples only; adapt them to your own target and credential handling.
Frequently asked questions
Why is a wait step needed between scrolls?
The scroll moves the viewport, but the next batch is appended after that movement. The wait gives the page time to finish adding it before the response is captured. The example uses 500 milliseconds twice.
How do I get more than two extra batches?
Repeat the scroll_y and wait pair. Two pairs produced eighteen blocks from a nine-block baseline, so each additional pair is expected to bring in the next batch.
What if the page loads content sideways?
Use scroll_x in place of scroll_y. It is the horizontal counterpart and sits in the same instruction list format.
What does a request without scroll instructions return?
Only the content present at load time — nine blocks on the demo page. That is a useful baseline, not a failure of the request.