Skip to content

Run Concurrent Scraping Requests from C#

A single-threaded scraper handles one URL, waits for the response, writes the file, then moves on.

OUTPUT FORMATSHTMLJSONMarkdown

When you have a queue of pages, that waiting time dominates. This guide shows a small C# console program that starts more than one scrape at once, and explains what actually controls how many calls can be in flight.

Why parallelism changes scraping speed

>>>

The C# program

>>>

Set APP_BASE_URL, SITE_BASE_URL and API_KEY. This keeps the documented two Thread instances and their separate /blog/ and /documentation/ outputs. HttpClient provides an alternative transport for the same GET /api/v1/ api_key/url query contract; each thread builds its own fields. Join waits for both threads before disposing the shared client, and HTTP/file failures are reported separately.

using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Net;
using System.Net.Http;
using System.Threading;

string apiOrigin = Environment.GetEnvironmentVariable("APP_BASE_URL")
    ?? throw new InvalidOperationException("Set APP_BASE_URL");
string siteOrigin = Environment.GetEnvironmentVariable("SITE_BASE_URL")
    ?? throw new InvalidOperationException("Set SITE_BASE_URL");
string key = Environment.GetEnvironmentVariable("API_KEY")
    ?? throw new InvalidOperationException("Set API_KEY");
using var handler = new HttpClientHandler
{
    AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate
};
using var transport = new HttpClient(handler);

void SavePage(string target, string outputFile)
{
    try
    {
        var fields = new Dictionary<string, string> { ["api_key"] = key, ["url"] = target };
        string query = string.Join("&", fields.Select(field =>
            Uri.EscapeDataString(field.Key) + "=" + Uri.EscapeDataString(field.Value)));
        string html = transport.GetStringAsync(apiOrigin.TrimEnd('/') + "/api/v1/?" + query)
            .GetAwaiter().GetResult();
        File.WriteAllText(outputFile, html);
    }
    catch (HttpRequestException error) { Console.Error.WriteLine("HTTP: " + error.Message); }
    catch (IOException error) { Console.Error.WriteLine("File: " + error.Message); }
}

var blog = new Thread(() => SavePage(siteOrigin.TrimEnd('/') + "/blog/", "blog.html"));
var docs = new Thread(() => SavePage(siteOrigin.TrimEnd('/') + "/documentation/", "documentation.html"));
blog.Start();
docs.Start();
blog.Join();
docs.Join();

What each request sends

>>>

Two different failures hide in one catch

>>>

A worked two-thread run

>>>

Before you raise the thread count

>>>

Request field map used by the sample
FieldWhere it comes fromWhat it controls
api_keyquery["api_key"] = API_KEYidentifies the account making the call
urlquery["url"] = urithe page to be fetched
BASE_URL{{APP_BASE_URL}}/api/v1/?the API endpoint the query string is appended to

Field names and values are taken from the sample program; the endpoint host is a configuration placeholder.

Failure distinction in the sample
SymptomWhere it happensWhat the sample returns
Request throws (network, auth, HTTP error)Get() called inside Scrape()exception propagates; no file is written
File write throwsinside the try around File.CreateTextScrape() returns false
Successfile writtenScrape() returns true

This reflects the sample's control flow only; it is not a general error taxonomy.

Frequently asked questions

How many requests can run at once?

>>>

Why use Thread instead of Task here?

>>>

What happens if one scrape fails?

>>>

Can I reuse the same output path for every URL?

>>>