Skip to content

C# Web Scraping: From a Console App to a Working CSV Pipeline

This guide walks through building a C# scraper that actually produces usable output: it fetches a page, parses HTML with HtmlAgilityPack, cleans the extracted values, and writes rows to CSV with CsvHelper. It then covers the situations that break simple scrapers — pages that need JavaScript, sites that expect particular request headers, and sites that span many pages — and shows when a rendering API is enough versus when a local browser is worth the overhead.

What you will build

A small console project that requests a page, loads the returned HTML into HtmlAgilityPack, selects product cards, and maps each card to a Product object with a title, a decimal price and an absolute URL. The same objects are then written to products.csv. The pipeline has three stages: fetch, parse and normalize, then export. Keeping those stages separate makes it easier to swap the fetch layer later without rewriting the parsing logic.

Environment setup

Install the .NET SDK and Visual Studio Code, then add the C# Dev Kit extension for IntelliSense, debugging and project templates. Confirm the install from a terminal with the command in the .NET CLI check module. If the command prints version, runtime and path details, the toolchain is ready. Create the project with the console scaffold module, which produces a folder containing a default Program.cs — the file where the scraping code will live.

Add the parsing and export libraries

Two NuGet packages carry most of the work. HtmlAgilityPack parses and navigates HTML through XPath or DOM traversal. CsvHelper writes records to CSV and handles quoting and encoding details. Add both from the project folder using the package install module, then check the project file for the package references shown in the project file module. For CSS selector support, add the separate CssSelectors package from the CSS selector install module; HtmlAgilityPack does not include CSS selectors by default.

Fetch HTML and parse it

HtmlAgilityPack does not make network requests — it only parses what you give it. Fetching is handled by HttpClient, or by a rendering API that returns rendered HTML. The endpoint shape is in the endpoint module: an api_key parameter and a url parameter, with the target URL encoded through Uri.EscapeDataString so query characters are not mangled. A short HttpClient timeout plus a CancellationToken prevents the app from hanging on a slow or blocked request, and an HttpRequestException catch keeps network failures from crashing the run.

Select elements with XPath or CSS

HtmlAgilityPack supports XPath natively through SelectNodes and SelectSingleNode. A selector such as //span[@class='titleline']/a targets anchor elements inside a specific span class; if the site changes its markup, that selector is the first thing to revisit. Relative href values should be resolved against the page base URI with new Uri(baseUri, href) so every link is absolute. If you prefer CSS syntax, the CssSelectors package adds QuerySelector and QuerySelectorAll, which read like document.querySelectorAll in JavaScript. Both styles can coexist in one project.

Forwarding custom headers to the target site

Some sites change layout, content or language depending on the visitor they think they are serving — desktop versus mobile, one locale versus another, bot versus human. A rendering API can relay your own headers to the destination, but only when you opt in. Set the forward_headers flag from the header-forwarding parameter module on the request, and prefix every header you want passed through with spb-. The prefix is mandatory: the service strips it and sends the remainder as a normal header, so spb-user-agent arrives as User-Agent and spb-accept-language arrives as Accept-Language. The C# header-forwarding example module shows the full request built with HttpClient.DefaultRequestHeaders.Add for both headers. Header forwarding can be combined with proxy and rendering options for finer control; the service documentation lists the full set of supported header rules.

Model and clean the extracted data

Define a Product class with Title, Price and Url properties so each scraped card becomes one typed instance. Cleaning matters as much as selection: HtmlEntity.DeEntitize converts encoded entities such as & back to their literal characters, replacing the non-breaking space character with a normal space removes invisible gaps, and a final Trim clears leftover whitespace. Prices arrive with a currency symbol, so strip it and parse with decimal.TryParse using NumberStyles.Any and CultureInfo.InvariantCulture to get a reliable decimal.

Write the CSV

CsvHelper writes a header row automatically when you call WriteRecords. Configure it with CultureInfo.InvariantCulture so decimal points stay as dots regardless of the machine locale, and set NewLine to Environment.NewLine. The output path is built with Path.Combine against AppContext.BaseDirectory. Encoding choice depends on the consumer: UTF-8 with a BOM helps older Excel versions on Windows read non-ASCII characters correctly, while UTF-8 without a BOM is the better fit for APIs and automated pipelines. Newer Excel versions usually detect UTF-8 without a BOM, but older versions may not.

Handling JavaScript-rendered pages

A plain HttpClient request returns the initial HTML only. If the content you need is injected by JavaScript, the parsed page can look empty even though a browser shows it fine. There are two routes. The first is a rendering API: append the render_js flag from the rendering flag module to the request and the service loads the page in a real browser before returning the final HTML. The second is local browser automation with Selenium WebDriver, which gives live DOM access and lets you interact with the page — clicking, scrolling, filling forms — at the cost of running a full browser process and maintaining drivers.

Local Selenium setup

Add Selenium.WebDriver and Selenium.WebDriver.ChromeDriver from the Selenium install module. The ChromeDriver package bundles and manages the matching driver version, so no manual driver download or path configuration is needed. Launch Chrome with headless, disable-gpu and no-sandbox arguments, navigate to the URL, and call driver.Quit() when finished. For element extraction, FindElements(By.CssSelector(...)) returns all matching nodes; wrap each card lookup in a try/catch for NoSuchElementException so malformed cards are skipped rather than aborting the loop. A WebDriverWait that checks document.readyState and the presence of at least one card prevents scraping before the page has settled.

Crawling multiple pages

Pagination turns a single-page scraper into a small crawler. Put pending URLs in a Queue<string>, track visited URLs in a HashSet<string> with OrdinalIgnoreCase comparison, and stop after a maxPages limit. On each page, extract the product cards, resolve the next link to an absolute URL, and enqueue it. Writing each record to CSV as it is found keeps memory flat on longer crawls. If you also follow product detail pages, keep a second HashSet for item URLs so the same product is not stored twice. If a page needs JavaScript to load its content, add the render_js flag to the request rather than introducing a local browser; if it varies by visitor, forward headers with the spb- prefix as described above.

Choosing a fetch strategy

Use this comparison to pick the lightest option that meets the requirement. The costs below describe setup and control trade-offs, not measured performance.

Failure checklist

When a run produces nothing useful, identify which stage failed before changing selectors. An HttpRequestException points to a network or status problem, not a parsing problem. A 429 response means the request rate is too high; add exponential backoff, randomize delays, cache pages already fetched, and slow the worker down. A successful request that yields zero matching nodes usually means the markup changed or the content is JavaScript-injected — check the raw response first, then consider the render_js flag. A page that returns the wrong layout or language may need forwarded headers with the spb- prefix. A CSV that opens with garbled characters is an encoding issue, not a data issue; revisit the BOM decision. Always review a site's robots.txt and terms of service before scraping.

Where to take it next

CSV is a good fit for small exports and spreadsheet review. Once the dataset grows or needs updates, move to a database: SQLite for local lightweight storage, PostgreSQL or MySQL for scale, and an ORM such as Entity Framework Core to upsert by a unique key like the product URL. Keeping the fetch, parse and store stages separate means the storage change does not touch the parsing code.

Fetch strategy comparison
ApproachSetupHandles JavaScriptPage interactionResource cost
HttpClient onlyNone beyond .NETNoNoLow
Rendering API with render_jsAPI key and one query parameterYesNoOffloaded to the service
Local Selenium WebDriverTwo NuGet packages plus ChromeYesYes — clicks, scrolls, form inputFull browser process per session

Setup and control descriptions only; no measured performance comparison.

Product model and CSV field map
FieldTypeSource on pageNormalization applied
Titlestringh3 > a title attribute, falling back to link textHtmlEntity.DeEntitize, non-breaking space replaced, Trim
Pricedecimalp.price_color inner textCurrency symbol stripped, decimal.TryParse with InvariantCulture
Urlstringh3 > a href attributeResolved to absolute URL against the page base

Describes the Product model and cleaning steps used in the guide.

CSV encoding decision
ConsumerEncoding choiceReason
Excel on Windows (especially older versions)UTF-8 with BOMHelps Excel detect encoding and render non-ASCII characters
APIs and automated pipelinesUTF-8 without BOMAvoids a leading byte-order marker in machine-parsed files
Newer Excel versionsUTF-8 without BOM usually worksDetection improved, but older versions may still misread

Encoding guidance reflects the behavior described for Excel versions at the time of writing.

Header forwarding checklist
StepWhat to doWhy it matters
Enable forwardingAdd forward_headers=true to the requestWithout the flag, prefixed headers are not relayed to the target
Prefix each headerStart every forwarded header name with spb-The prefix is mandatory; the service strips it before sending
Name the real headerspb-user-agent becomes User-Agent; spb-accept-language becomes Accept-LanguageThe target site sees a normal header, not the prefixed name
Combine with other optionsPair with proxy and rendering parametersGives finer control over how the page is served

Rules as described in the supplied header-forwarding example; consult the service documentation for the full list of supported headers.

Frequently asked questions

Which libraries do I need for C# web scraping?

HtmlAgilityPack for parsing HTML with XPath or CSS selectors, CsvHelper for CSV export, and HttpClient for network requests. For JavaScript-heavy pages, add a rendering API call with render_js or use Selenium WebDriver for full browser control.

How do I scrape multiple pages?

Keep pending URLs in a Queue and visited URLs in a HashSet. Find the next link on each page, resolve it to an absolute URL, enqueue it, and stop at a maxPages limit or when the queue empties.

Can a C# scraper read content loaded by JavaScript?

Yes, but not with plain HttpClient, which only fetches the initial HTML. Add render_js=true to a rendering API request, or run Selenium WebDriver locally to execute scripts and read the live DOM.

How can I reduce the chance of being blocked?

Rotate proxies, send realistic User-Agent and Accept-Language headers through the forward_headers flag with the spb- prefix, keep request rates polite, cache pages already fetched, and review robots.txt and the site's terms before scraping.

Should I use CSS selectors or XPath?

Both work. XPath is native to HtmlAgilityPack and suits deeply nested or table-like layouts. CSS selectors, available through HtmlAgilityPack.CssSelectors.NetCore, often read more naturally for simpler modern markup. Pick based on the target page and your team's familiarity.

How do I store results beyond CSV?

Use a database once the dataset grows or needs updates: SQLite for local lightweight storage, PostgreSQL or MySQL for scale. An ORM such as Entity Framework Core lets you upsert by a unique key like the product URL to avoid duplicates.

What should I do about 429 errors?

A 429 means too many requests. Add exponential backoff with retries, randomize delays, cache pages already fetched, and use rotating proxies. If the errors persist, slow the scraper or move to a queue-based worker that controls throughput.

{{APP_BASE_URL}}/api/v1?api_key=YOUR_API_KEY&url=https%3A%2F%2Fexample.com&render_js=true