Run Concurrent Scraping Requests from C#
A single-threaded scraper handles one URL, waits for the response, writes the file, then moves on.
When you have a queue of pages, that waiting time dominates. This guide shows a small C# console program that starts more than one scrape at once, and explains what actually controls how many calls can be in flight.
Why parallelism changes scraping speed
>>>
The C# program
>>>
Set APP_BASE_URL, SITE_BASE_URL and API_KEY. This keeps the documented two Thread instances and their separate /blog/ and /documentation/ outputs. HttpClient provides an alternative transport for the same GET /api/v1/ api_key/url query contract; each thread builds its own fields. Join waits for both threads before disposing the shared client, and HTTP/file failures are reported separately.
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Net;
using System.Net.Http;
using System.Threading;
string apiOrigin = Environment.GetEnvironmentVariable("APP_BASE_URL")
?? throw new InvalidOperationException("Set APP_BASE_URL");
string siteOrigin = Environment.GetEnvironmentVariable("SITE_BASE_URL")
?? throw new InvalidOperationException("Set SITE_BASE_URL");
string key = Environment.GetEnvironmentVariable("API_KEY")
?? throw new InvalidOperationException("Set API_KEY");
using var handler = new HttpClientHandler
{
AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate
};
using var transport = new HttpClient(handler);
void SavePage(string target, string outputFile)
{
try
{
var fields = new Dictionary<string, string> { ["api_key"] = key, ["url"] = target };
string query = string.Join("&", fields.Select(field =>
Uri.EscapeDataString(field.Key) + "=" + Uri.EscapeDataString(field.Value)));
string html = transport.GetStringAsync(apiOrigin.TrimEnd('/') + "/api/v1/?" + query)
.GetAwaiter().GetResult();
File.WriteAllText(outputFile, html);
}
catch (HttpRequestException error) { Console.Error.WriteLine("HTTP: " + error.Message); }
catch (IOException error) { Console.Error.WriteLine("File: " + error.Message); }
}
var blog = new Thread(() => SavePage(siteOrigin.TrimEnd('/') + "/blog/", "blog.html"));
var docs = new Thread(() => SavePage(siteOrigin.TrimEnd('/') + "/documentation/", "documentation.html"));
blog.Start();
docs.Start();
blog.Join();
docs.Join();What each request sends
>>>
Two different failures hide in one catch
>>>
A worked two-thread run
>>>
Before you raise the thread count
>>>
| Field | Where it comes from | What it controls |
|---|---|---|
| api_key | query["api_key"] = API_KEY | identifies the account making the call |
| url | query["url"] = uri | the page to be fetched |
| BASE_URL | {{APP_BASE_URL}}/api/v1/? | the API endpoint the query string is appended to |
Field names and values are taken from the sample program; the endpoint host is a configuration placeholder.
| Symptom | Where it happens | What the sample returns |
|---|---|---|
| Request throws (network, auth, HTTP error) | Get() called inside Scrape() | exception propagates; no file is written |
| File write throws | inside the try around File.CreateText | Scrape() returns false |
| Success | file written | Scrape() returns true |
This reflects the sample's control flow only; it is not a general error taxonomy.
Frequently asked questions
How many requests can run at once?
>>>
Why use Thread instead of Task here?
>>>
What happens if one scrape fails?
>>>
Can I reuse the same output path for every URL?
>>>