Scrape a webpage from C# and save its HTML
This guide shows a small C# workflow for requesting a page through an API, receiving its HTML as text, and then either displaying that text or saving it locally.
It is useful when you want the fetching handled remotely while parsing and storage remain in your own application.
Make the API request from C#
Start with a reusable async helper that accepts the address to fetch. It escapes that address before placing it in the request URI, sets the Bearer authorization header, performs the GET request, verifies the response status, and reads the body as a string. A separate entry point supplies the target address and decides what to do with the returned markup. For a quick check, printing the response to the console is enough to confirm that the request succeeded and that HTML came back.
Run as a modern .NET console program after setting APP_BASE_URL, API_KEY and TARGET_URL. This preserves the documented GET /api/v1/ request, url query field and Authorization: Bearer header. GetStringAsync raises on an unsuccessful HTTP status instead of treating its body as successful HTML.
using System;
using System.IO;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Threading.Tasks;
string apiOrigin = Environment.GetEnvironmentVariable("APP_BASE_URL")
?? throw new InvalidOperationException("Set APP_BASE_URL");
string token = Environment.GetEnvironmentVariable("API_KEY")
?? throw new InvalidOperationException("Set API_KEY");
using var transport = new HttpClient();
transport.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", token);
Task<string> FetchHtmlAsync(string target)
{
string address = apiOrigin.TrimEnd('/') + "/api/v1/?url=" + Uri.EscapeDataString(target);
return transport.GetStringAsync(address);
}
string html = await FetchHtmlAsync(Environment.GetEnvironmentVariable("TARGET_URL")
?? throw new InvalidOperationException("Set TARGET_URL"));
Console.WriteLine(html);Save the returned HTML locally
The second example follows the same request pattern but changes the destination of the result. Instead of printing the HTML, the program writes the returned string to a local file named blog.html and reports the saved path. That pattern separates fetching from handling: once the helper returns text, your code can print it, write it to disk, parse it, or pass it to another processing step. To scrape a different page, replace only the target address passed to the helper.
Use FetchHtmlAsync, File and the client setup from the preceding example, and set SITE_BASE_URL to the website origin. The /blog/ response is saved after the same authenticated request succeeds.
string siteOrigin = Environment.GetEnvironmentVariable("SITE_BASE_URL")
?? throw new InvalidOperationException("Set SITE_BASE_URL");
string blogHtml = await FetchHtmlAsync(siteOrigin.TrimEnd('/') + "/blog/");
await File.WriteAllTextAsync("blog.html", blogHtml);
Console.WriteLine("Saved blog.html");Adaptation checklist
Before adapting the examples, work through this checklist:
- Confirm the API base address and key are configured rather than left as placeholders.
- Pass the exact page address you want through the helper so it is escaped correctly.
- Decide whether the returned HTML should be printed, saved, parsed, or stored.
- For repeated requests in a production application, reuse one HttpClient instance instead of creating a new client for each request.
- Treat the single-request client creation in the examples as a simplification for readability, not as the recommended pattern for a larger scraper.
Frequently asked questions
What does the C# Get helper do?
The helper builds an escaped request URI from the target address, attaches the key as a Bearer token in the Authorization header, sends the request with HttpClient, checks for success, and returns the response body as a string. The entry point then either prints that string or writes it to a file.
Should a production scraper create a new HttpClient for every request?
For a one-off demonstration, creating a client per request keeps the code short. For an application issuing many requests, reuse a single HttpClient instance instead of creating a new one each time.