Skip to content

Java Web Scraping: A Practical Task Guide

This guide walks through the practical decisions behind a Java web-scraping project: how to fetch pages, choose between a lightweight HTTP client and a real browser, extract structured data, handle forms and authentication, and deploy scrapers in the cloud.

OUTPUT FORMATSHTMLJSONMarkdown

It is organized around tasks rather than library tours, so you can move from a simple HTML page to a JavaScript-heavy application without losing track of what each tool is doing. The material is adapted from the Java Web Scraping Handbook, originally written by Kevin Sahin and published at javawebscrapinghandbook.com; this edition is maintained by {{OWNER_PUBLISHER_NAME}}.

What Java web scraping involves

At its core, a scraper turns a web page into structured data. In Java, that means fetching the page with an HTTP client or headless browser, parsing the returned HTML, and then locating the values you need with selectors or regular expressions. The output is usually a typed object, a JSON document, or a database row. The three stages repeat across projects: fetch the response, locate the relevant nodes or text, and convert the result into a reusable format. A scraper can be run by hand, but the term usually refers to an automated pipeline that downloads a page, extracts the wanted values, and stores them for later analysis.

APIs versus scraping

When a site exposes a clean API, use it. APIs usually return JSON or XML and are easier to consume than HTML. Scraping becomes necessary when no API exists, when the API is rate-limited, or when the available data is incomplete. The trade-off is maintenance: a scraper depends on the page structure, so a markup change can break extraction. A practical rule is to check for an official API first, then fall back to scraping only the pages you actually need.

Common use cases

Scraping is used across news aggregation, real-estate listings, travel price comparison, e-commerce monitoring, bank account aggregation, data journalism, SEO analysis, market research, and lead generation. The common thread is fragmented data spread across many sites. When an industry has a web presence split across dozens or hundreds of pages, an aggregation opportunity usually exists. The same maintenance risk applies: each source site can change independently.

HTTP and DOM foundations

HTTP is a client/server protocol. A client sends a request with a method, path, headers, and sometimes a body; the server responds with a status code, headers, and content. The important request headers for scraping include Host, User-Agent, Accept, Cookie, and Referer. Status codes fall into familiar classes: 2XX success, 3XX redirect or action required, 4XX client error, and 5XX server error. The browser then parses the HTML into a Document Object Model, a tree of element and text nodes. The DOM you inspect in a browser can differ from the original HTML because JavaScript may have modified it after load.

Choosing an extraction approach

There are two broad approaches. A headless browser handles HTML parsing, cookies, forms, JavaScript execution, and iframes, but carries more overhead. A plain HTTP library plus a parser such as Jsoup is lighter and faster when the page does not require JavaScript. Use the lighter path first; move to a headless browser only when the content is rendered client-side or when you need browser-level behavior such as screenshots or complex form interaction.

XPath and regular-expression extraction

XPath selects nodes by path, predicate, or attribute. It is useful when the page has a stable structure and you can target elements directly. Regular expressions are useful when the data is embedded in a larger block of text, such as a price or date inside a paragraph. In practice, use XPath to narrow the search to the right element, then use a regex to pull a specific value from that element's text. Both techniques are fragile if the page structure or wording changes, so keep selectors as specific as necessary but no more brittle than the data requires.

A worked example: Hacker News items

The Hacker News example shows the full extraction loop. The scraper fetches the page, selects each row with the class athing, then reads the position, title, URL, author, score, and item ID using relative XPath expressions. The extracted values are placed into a plain Java object and serialized to JSON with Jackson. This pattern generalizes: identify the repeating container, select fields relative to that container, and map the result into a typed object before exporting. The exact selectors are tied to the page markup, so verify them against the current DOM before relying on them.

Forms, logins, and cookies

Form handling is often the difference between a public-page scraper and one that can reach authenticated data. A form has an action URL and a method, usually POST for login or account creation. Inputs carry a name and a value; the browser sends those name-value pairs in the request body. After a successful login, the server typically sets a session cookie, which the client must send on subsequent requests. Hidden inputs, including CSRF tokens, must be preserved exactly as the server issued them. A reliable login flow therefore means: load the form, read every required input including hidden ones, submit the form, and verify success by checking for a known post-login element rather than assuming the request worked. The worked examples in this guide use a practice sandbox at {{OWNER_SANDBOX_URL}}; confirm that URL with the owner before relying on it.

JavaScript-rendered pages

Modern front-end frameworks often render content after the initial HTML load. A lightweight headless browser may fetch both the HTML and the JavaScript but still leave the page incomplete because it cannot run every script. There are two practical responses: use a more capable browser such as headless Chrome, or inspect the network calls the page makes and reproduce the underlying API requests directly. The second approach is often faster and more stable, but it requires understanding the request parameters and authentication. When the page depends on WebSockets or complex client-side state, a real browser is usually the safer choice.

Headless Chrome and Selenium

Headless Chrome runs the browser without a visible window and is controlled through Selenium. The WebDriver object sends commands to the browser, and WebElement objects represent DOM nodes. Useful methods include get, getCurrentUrl, getPageSource, navigate, switchTo, manage cookies, findElement, click, getText, sendKeys, and getAttribute. Waiting is a common source of failures: a fixed sleep may be too short or too long, so prefer explicit waits such as WebDriverWait with an ExpectedCondition that matches the element you need. For infinite scroll, you can scroll the window, call a JavaScript function directly, or call the underlying API; the API approach is usually the most efficient when it is available.

Captchas, PDFs, and OCR

Captcha solving is slow and costly, so the first goal should be to avoid triggering captchas through polite request rates and realistic headers. When a captcha must be solved, third-party services can return a token that is placed into the page's hidden response field before form submission; any named service, pricing, or terms must be confirmed by the owner (OWNER_CAPTCHA_SERVICE_TERMS) before publication. PDF extraction follows a different path: download the file, verify its content type, then use a PDF library to extract text and apply a regex to the relevant line. OCR is the fallback for text embedded in images; Tesseract can recognize text, but accuracy depends on language data, image quality, and preprocessing such as cropping or contrast adjustment.

Staying unblocked

Anti-bot systems look at headers, request timing, IP reputation, cookies, and hidden elements. Sending a realistic set of browser-like headers is a basic step. Randomizing the User-Agent helps, but it must be consistent with the rest of the request. Proxies distribute requests across IP addresses; free proxy lists are often unreliable, so paid or self-hosted pools are usually better. TOR hides the origin IP but its exit nodes are public and often blocked. Timing matters: a fixed click cadence is a strong bot signal, so introduce random pauses between requests. Hidden form inputs and invisible elements can be traps; preserve hidden inputs and check visibility before interacting with elements.

Serverless deployment

Cloud functions suit scraping workloads because scraping is I/O-bound and can be parallelized. A function can be triggered by an HTTP request, a queue message, a log event, or an IoT event. The deployment flow typically involves creating a project from an archetype, testing locally, and deploying with a build tool. Cold starts can make the first invocation slow, so plan for that if latency matters. The function can return JSON directly or write results to a database or queue. The same scraper logic used locally can be packaged into a function, but credentials and configuration should be managed through the platform rather than hard-coded. Provider-specific instructions, including Azure function deployment, must be confirmed by the owner (OWNER_CLOUD_PROVIDER_TERMS) before publication.

Choosing a scraping approach
ApproachBest forTrade-off
Plain HTTP client + parserStatic HTML, simple pages, high volumeCannot execute JavaScript or handle complex browser behavior
Lightweight headless browser (HtmlUnit)Forms, cookies, some JavaScript, moderate complexityMay fail on modern JavaScript frameworks
Full headless browser (Chrome + Selenium)JavaScript-heavy pages, screenshots, complex interactionHigher resource use and slower execution
Direct API callsPages that load data through a discoverable APIRequires reverse-engineering request parameters and authentication

This is a decision aid, not a benchmark. The right choice depends on the target page and the data you need.

Form and session control map
ControlWhat to checkWhy it matters
Form action and methodWhere the request goes and whether it is GET or POSTDetermines the request shape
Visible inputsName and value for each fieldThese become request parameters
Hidden inputsCSRF tokens and other hidden valuesMust be preserved exactly; changing them can cause rejection
CookiesSession cookie set after loginRequired for authenticated follow-up requests
Success indicatorA known element or text after submissionConfirms the request actually worked

Use this as a checklist when a form submission fails or when authenticated pages return unexpected content.

Frequently asked questions

Should I use HtmlUnit or headless Chrome for Java scraping?

Start with HtmlUnit when the page is mostly static or uses simple forms. Move to headless Chrome when the page relies on modern JavaScript frameworks, WebSockets, or complex client-side rendering. The lighter tool is faster and easier to run; the full browser is more capable but heavier.

Why does my Java login scraper fail even when the credentials are correct?

Common causes include missing hidden fields such as CSRF tokens, not preserving cookies between requests, submitting to the wrong action URL, or checking for success incorrectly. Inspect the actual POST request in the browser's developer tools and compare it with the request your scraper sends.

Why is the data missing from the HTML my scraper downloads?

The content may be rendered by JavaScript after the initial HTML load. Check the network panel for API calls that return the data directly, or use a browser that executes JavaScript. If the page uses a discoverable API, calling that API is often simpler and more stable than rendering the full page.

How can I reduce captcha challenges while scraping?

Use realistic headers, rotate User-Agents and proxies where appropriate, add random delays between requests, and avoid aggressive request rates. Captcha solving services are slow and costly, so prevention is usually the better strategy.

curl https://api.coinmarketcap.com/v1/ticker/ethereum/?convert=EUR
{
    id: "ethereum",
    name: "Ethereum",
    symbol: "ETH",
    rank: "2",
    price_usd: "414.447",
    price_btc: "0.0507206",
    24h_volume_usd: "1679960000.0",
    market_cap_usd: "39748509988.0",
    available_supply: "95907342.0",
    total_supply: "95907342.0",
    max_supply: null,
    percent_change_1h: "0.64",
    percent_change_24h: "13.38",
    percent_change_7d: "25.56",
    last_updated: "1511456952",
    price_eur: "349.847560557",
    24h_volume_eur: "1418106314.76",
    market_cap_eur: "33552949485.0"
}
curl https://api.e-commerce.com/products
curl https://api.e-commerce.com/products/123
GET /how-to-log-in-to-almost-any-websites/ HTTP/1.1
Host: ksah.in
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Encoding: gzip, deflate, sdch, br
Connection: keep-alive
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36
POST /login HTTP/1.1
Host: example.com
Content-Type: application/x-www-form-urlencoded
uname=username&psw=my_great_password
GET /sample_page.html HTTP/1.1
Host: www.example.org
Cookie: cookie_name=cookie_value
WebRequest[<url="https://www.javawebscrapingsandbox.com/account/login",
POST, EncodingType[name=application/x-www-form-urlencoded],
[csrf_token=1524752332##6997dd9d5ed448484131add18b41a4263541b5c2,
 [email protected],
 password=test],
 {Origin=https://www.javawebscrapingsandbox.com/account/login,
 Accept=text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,
 Cache-Control=max-age=0,
 Referer=https://www.javawebscrapingsandbox.com/account/login,
 Accept-Encoding=gzip, deflate}, null>]
[
    {
        id: 31,
        name: "Marmot Drop Line Men's Jacket, Lightweight 100-Weight Sweater Fleece",
        price: "74.96",
        url: "https://www.amazon.com/gp/product/B075LC96R2/ref=ox_sc_sfl_title_39?ie=UTF8"
    },
    {
        id: 32,
        name: "ASUS ZenPad 3S 10 9.7" (2048x1536), 4GB RAM, 64GB eMMC, 5MP Front / 8MP Rear Camera, Android 6.0, Tablet, Titanium Gray (Z500M-C1-GR)",
        price: "296.07",
        url: "https://www.amazon.com/dp/B01MATMXZV?tag=thewire06-20"
    },
    {
        id: 33,
        name: "LG Electronics OLED65C7P 65-Inch 4K Ultra HD Smart OLED TV (2017 Model)",
        price: "2596.99",
        url: "https://www.amazon.com/gp/product/B01NAYM1TP/ref=ox_sc_sfl_title_35?ie=UTF8"
    },
    ...
]
curl https://${function_url}/api/hnitems?pageNumber=3
[
    {
        "title": "Nvidia Can Artificially Create Slow Motion That Is Better Than a 300K FPS Camera (vice.com)",
        "url": "https://motherboard.vice.com/en_us/article/ywejmy/nvidia-ai-slow-motion-better-than-a-300000-fps-camera",
        "author": "jedberg",
        "score": 27,
        "position": 121,
        "id": 17597105
    },
    {
        "title": "Why fundraising is a terrible experience for founders: Lessons learned (kapwing.com)",
        "url": "https://www.kapwing.com/blog/the-terrible-truths-of-fundraising/",
        "author": "jenthoven",
        "score": 74,
        "position": 122,
        "id": 17594807
    },
    {
        "title": "Why No HTTPS? (whynohttps.com)",
        "url": "https://whynohttps.com",
        "author": "iafrikan",
        "score": 62,
        "position": 123,
        "id": 17599022
    },
   //...