Skip to content

HtmlUnit in Java: scrape static and dynamic pages with a headless browser

HtmlUnit is a GUI-less browser for Java that can execute JavaScript and perform AJAX calls. It was primarily designed for automated testing, but the same browser-like controls — loading pages, selecting DOM elements, typing into forms and clicking buttons — make it practical for scraping both static and dynamic sites. This guide walks through a Maven setup, a static-page extraction, a selector-based traversal, and a form-driven dynamic example.

Set up the Maven project

You need Java installed and a build tool such as Maven or Gradle. The examples use Maven; adapt the build-specific steps if you use Gradle. Create a new Maven project with module code0001, move into its directory with code0002, add the HtmlUnit dependency shown in code0003 to the dependencies section of pom.xml, and add the main-class property shown in code0004 to the properties section. You are then ready to write the scraper in App.java.

Scrape a static page with CSS selectors

Start with the Java skeleton in code0005. Create a WebClient as in code0006. For a page that does not need scripts, disable JavaScript with code0007 to reduce work. Load the page with code0008, which returns an HtmlPage. Before writing a selector, inspect the page in the browser and identify stable structure. In the worked example, the target headings are h3 elements inside the third section, so the selector in code0009 combines position and element classes. Pass that selector to querySelectorAll as in code0010, iterate the returned nodes with code0011, cast each node to HtmlHeading3 and print its text using code0012. Run the project with code0014. Decision check: if the selector returns nothing, re-inspect the live DOM first — positional selectors such as nth-child break when sections are added or reordered, so prefer an ID or stable class when one exists.

Traverse siblings and children when classes are missing

Some lists have no useful ID or class. In the footer example, blog links sit in li elements inside a ul that follows an h4 heading. Find the heading by its visible text with the XPath in code0015, move to the next element sibling with code0016, and collect its child nodes with code0017. Child nodes include text and comment nodes, so filter for element nodes before casting, as in code0018. Casting to HtmlAnchor exposes the link text and href; combine the base URI with the href attribute when you need an absolute URL. Use XPath when text is the most stable anchor, CSS when classes are stable, and sibling traversal when structure — rather than naming — defines the relationship.

Automate a form on a dynamic page

For interactive pages, re-enable JavaScript and allow script errors to be logged without aborting the run, as in code0019. Load the example form page with code0020 and take its first form with code0021. Select the search field by name with code0022 and the submit button by its value with code0023. Type the query and click with code0024; the click returns the results page. Read each tr.team row and pull the .year and .wins cells with code0025, trimming whitespace before printing. Worked example: searching for Toronto returns Toronto Maple Leafs rows, from which year and wins are extracted. If script errors appear in the log, check whether they are non-fatal before treating them as failures.

When HtmlUnit fits — and what it does not solve

HtmlUnit suits Java scrapers that need browser behaviour without a visible browser: it supports the W3C DOM standard, CSS and XPath selectors, JavaScript execution, and simulated Firefox, Chrome and Internet Explorer browsers. It handles static extraction, structural traversal and form interaction in one client. For real-world sites, plan separately for rate limits, proxies, bot detection and browser fingerprinting — those concerns sit outside the basic HtmlUnit workflow described here.