Skip to content

Web Scraping with Visual Basic

Visual Basic can run a practical scraper without leaving the .NET ecosystem. This guide walks through a Windows Forms app that fetches a page, extracts headings, links and table headings with HtmlAgilityPack, and then adds a PuppeteerSharp path for pages whose content only appears after JavaScript runs.

Choose the right scraping path first

The most important decision comes before any code. If the data is already present in the HTML the server returns, an HTTP client plus an HTML parser is enough. If the data is inserted or updated in the browser after the initial load, a headless browser is needed.

Use this checklist:

  • View the page source, not just the rendered page. If your target text appears in the source, start with the static approach.
  • If the target text is missing from the source but visible in the browser, plan for a headless browser.
  • If the page is a single-page application or loads content asynchronously, assume the static download will be incomplete.
  • If the page is slow or waits on a slow API, plan a longer navigation timeout and an explicit wait for the target element.
  • If the page structure changes often, keep selectors in one place so they are easy to update.

Understand the page structure before writing selectors

Before scraping, inspect the target page's HTML. Identify which tags and attributes hold the data. For a Wikipedia article, for example, sections may contain unordered lists whose list items hold hyperlinks, and tables may use row and heading elements with specific class names. The same inspection habit applies to any site: find the container, then the element inside it that carries the text or attribute you need.

The original walkthrough used a Wikipedia page on the history of programming languages. The exact selectors below are tied to that page's markup at the time of writing, so treat them as examples of the technique rather than universal selectors.

Set up the Visual Basic project

Install Visual Studio if it is not already present, then create a new project. Choose Visual Basic as the language and Windows Forms App (.NET Framework) as the project type. Give the project a name and a save location.

Next, add the HtmlAgilityPack NuGet package. In Solution Explorer, right-click the project, choose Manage NuGet Packages, search for HtmlAgilityPack in the Browse tab, and click Install. HtmlAgilityPack lets you load HTML into a document object, select nodes, and read their text and attributes.

Build the form controls

Open Form1.vb in the designer and add the controls the scraper needs. The field map below shows what each control does and the names used in the sample code.

  • URL input: a TextBox named urlTextbox, used to enter the page address.
  • Output area: a second TextBox named outputTextbox, set to Multiline, used to display extracted text.
  • Scrape trigger: a Button named scrapeButton, used to start the static scrape.
  • Headless trigger: a second Button named HeadlessBtn, used to start the PuppeteerSharp scrape.

Adjust positions and sizes as needed. Keeping the names consistent with the sample code avoids extra renaming work.

Scrape a static page with HtmlAgilityPack

The static scraper follows a short sequence: read the URL from the text box, download the HTML string, load it into an HtmlAgilityPack document, select nodes with XPath, and append the extracted text to the output box.

The complete sample is provided as a protected code module and should be used exactly as supplied. In summary, it selects the first h1 and first p, then selects links inside elements with class div-col, and finally selects table heading cells inside tables that carry several class names. Each selection checks for Nothing before looping, and appends a fallback message when no nodes are found.

To add the code, open the View menu, choose Code or press F7, paste the sample, and save. Run the app with F5 or the green Start button, enter a URL, and click the scrape button.

How the XPath selectors work

Two selectors carry most of the extraction logic in the static example.

The link selector is a union of two paths. The first path reaches anchor elements that live inside a div whose class is div-col; the second path reaches unordered lists inside that same container. The union means the result set can contain both kinds of node. For each node returned, the code reads the href attribute with an empty-string default and appends it to the output.

The table heading selector is a chained filter. It starts from tables and requires four class tokens to be present at once — sidebar, nomobile, nowraplinks and hlist — then descends to th elements whose class includes sidebar-heading. Chaining contains() checks this way is the practical pattern when a page describes one component with several class names, because it does not depend on the exact order or spacing of those names.

If you need a deeper treatment of XPath, the source links to a separate practical XPath article.

Add a headless browser for JavaScript-rendered pages

When the server response lacks the target content, switch to a headless browser. In a Visual Basic project, install PuppeteerSharp through NuGet: right-click the project, choose Manage NuGet Packages, search for PuppeteerSharp, and click Install.

The headless sample is also supplied as a protected code module. It downloads a browser build, launches Chromium in headless mode, opens a new page, navigates to the target URL with a two-minute timeout, waits for the target selector, queries all matching elements, extracts each element's text content, writes the headlines to the output box, and closes the browser.

The example target is the CNN sport page. The selector looks for span elements with a data-editable attribute equal to headline inside a div with class container_lead-plus-headlines-with-images__headline. Because that selector is tied to the page's markup, expect to revise it if the site changes.

Understand the two-minute timeout and the wait step

The headless sample sets a navigation timeout of 120,000 milliseconds, which is two minutes. This gives a slow page more time to finish loading before the script continues. If navigation still exceeds that limit, a TimeoutException is thrown and the scrape stops.

A long timeout is not a substitute for a proper wait. The sample also calls WaitForSelectorAsync so the script pauses until the target element exists in the page. This matters when content is injected after the initial document load.

Keep the two failure modes separate:

  • Navigation timeout: the page did not finish loading within the configured limit.
  • Missing selector: navigation completed, but the element you asked for was not found, often because the markup changed or the content never rendered.

Logging which of these occurred makes selector maintenance much faster.

A worked example: CNN sport headlines

Suppose you want the current headlines from the CNN sport page. The headless sample navigates to that URL, waits for the headline selector, collects every matching span, and writes each headline on its own line in the output box.

If the output is empty, check the selector against the live page before changing the timeout. If the script throws a timeout instead, check whether the page loaded at all and whether the two-minute limit is appropriate for your network. If the output contains headlines but not the ones you expected, the selector may be matching a different module on the page.

This example is a technique demonstration. The page's markup and headline set can change at any time, so verify the selector against the current page before relying on it.

When to move beyond a local scraper

A local Visual Basic scraper is a good fit for learning, for small one-off jobs, and for pages you control or can access reliably. At larger scale, rate limits, proxy rotation, user-agent handling and browser fingerprinting become operational work of their own. A managed scraping API is one option for teams that would rather not maintain that layer.

Any commercial terms, trial allowances or plan details belong to the service you choose and should be confirmed directly with that provider before you rely on them.

Frequently asked questions

Do I need a headless browser for every Visual Basic scraper?

No. If the data is present in the HTML returned by the server, an HTTP client plus HtmlAgilityPack is enough. Use a headless browser only when the content is loaded or changed by JavaScript after the initial page load.

Which NuGet packages does the guide use?

The static example uses HtmlAgilityPack for parsing. The dynamic example uses PuppeteerSharp to control a headless Chromium browser.

Why does the headless example use a two-minute timeout?

The sample sets a navigation timeout of 120,000 milliseconds so a slow page has time to finish loading. If navigation exceeds that limit, a TimeoutException is thrown. The timeout does not replace waiting for a specific selector.

What should I check if the scraper returns no results?

First check whether the selector still matches the page's current markup. If navigation itself failed, look for a timeout instead. Keeping the two failure modes separate makes debugging faster.