Skip to content

Web Scraping with Html Agility Pack

Html Agility Pack gives C# developers a DOM-oriented way to locate elements, walk between parent and child nodes, and read both visible text and attributes such as links. This guide shows a complete workflow: install the library, fetch a page, inspect its structure, extract the top ten Hacker News stories, and return them as JSON. It also shows how to reuse the same parser when a page needs a real browser to render its content.

Install Html Agility Pack

Create a .NET project first, then add Html Agility Pack through NuGet in Visual Studio via Tools, NuGet Package Manager and Manage NuGet Packages for Solution. Search for the package and install it into the solution dependencies. If the namespace import is missing, add the import shown in protected module code0001 to each file that uses the library.

Inspect the page before parsing

Before writing selectors, inspect the target page in a browser. In the Hacker News example, each story sits in a table row. Useful markers include the title, rank, story link and score classes, together with anchor and span elements that hold the values to extract. Noting those markers first prevents guessing selectors later.

Fetch HTML with native C# libraries

For pages whose markup is available immediately, fetch the HTML with native C# HTTP support. Add the imports in protected module code0002, then use the small fetch method in protected module code0003 to load the Hacker News homepage into a response variable. A breakpoint with the HTML visualizer is a quick way to confirm the markup arrived before parsing begins.

Parse rows with LINQ and XPath

Load the fetched markup into an HtmlDocument. Use LINQ to collect table rows whose class contains athing and keep only the first ten, as shown in protected module code0004. Then loop over those rows and use XPath to read the rank, story title, link address and score for each one, as shown in protected module code0005. LINQ handles the broad selection and count; XPath handles the precise per-row fields.

Turn parsed rows into JSON

Create a small HackerNewsItems class in the same namespace as the parsing code, with fields for rank, title, url and score. Install Newtonsoft.Json from NuGet and add the import in protected module code0006. Populate one HackerNewsItems object per row, add each object to a list, and serialize the list to JSON, as shown in protected module code0007. JSON is a practical handoff format because most APIs, applications and languages can consume it. Protected module code0008 combines fetching, parsing and serialization end to end.

Use Selenium when rendering matters

Some sites build their visible content with client-side scripts, so the initial HTML response does not contain the final results. In that case, drive a browser with Selenium, let the page render, and feed the browser page source into the same ParseHtml method. Install Selenium.WebDriver from NuGet and add the imports in protected module code0009. Protected module code0010 shows the headless Chrome setup; adjust the Chrome executable path to match the local machine. Protected module code0011 shows the full browser-based version reusing the existing parser. A headless Firefox driver is also available from NuGet if Chrome is not preferred.

Keep the browser driver in sync

Keep the Selenium library current as Chrome updates. A session creation error stating that the ChromeDriver version only supports a different Chrome version means the driver and browser are out of sync; update the Selenium library in the project to resolve it.

Choose the right tool combination

Html Agility Pack covers parsing well, but it does not render JavaScript by itself. Pages built with frameworks such as React, Angular or Vue may therefore need a browser driver alongside it. XPath remains valuable because it can express more flexible searches than CSS selectors alone. Plan the stack around the page type: direct fetch plus parser for static markup, browser plus parser for rendered content.

Direct fetch or browser fetch?
SituationFetch methodParser inputBest for
Markup is present in the initial responseNative C# HTTP requestResponse HTML stringStatic or server-rendered pages
Content appears only after scripts runSelenium with a browserBrowser page sourceSingle-page applications and client-rendered pages

Decision table derived from the two workflows in the guide; browser path adds rendering at the cost of a driver and browser dependency.

Frequently asked questions

When should I use Selenium instead of a direct HTTP request?

Inspect the rendered page first. If the needed text only appears after client-side scripts run, fetch the page through a browser driver and pass the resulting page source to the same parser. If the markup is already present in the initial response, a direct HTTP request is simpler and faster.

How do LINQ and XPath work together in this example?

Use LINQ to narrow the node set to the rows you care about and limit the count, then use XPath expressions inside each row to read the exact text or attribute for rank, title, link and score. That combination keeps the broad search readable while making the per-item extraction precise.

What does a ChromeDriver version mismatch mean?

The driver and browser versions must match. When Chrome reports that the installed ChromeDriver supports a different Chrome version, update the Selenium library in the project until the driver and browser align.