Java Web Scraping: A Practical Task Guide
This guide walks through the practical decisions behind a Java web-scraping project: how to fetch pages, choose between a lightweight HTTP client and a real browser, extract structured data, handle forms and authentication, and deploy scrapers in the cloud.
It is organized around tasks rather than library tours, so you can move from a simple HTML page to a JavaScript-heavy application without losing track of what each tool is doing. The material is adapted from the Java Web Scraping Handbook, originally written by Kevin Sahin and published at javawebscrapinghandbook.com; this edition is maintained by {{OWNER_PUBLISHER_NAME}}.
What Java web scraping involves
At its core, a scraper turns a web page into structured data. In Java, that means fetching the page with an HTTP client or headless browser, parsing the returned HTML, and then locating the values you need with selectors or regular expressions. The output is usually a typed object, a JSON document, or a database row. The three stages repeat across projects: fetch the response, locate the relevant nodes or text, and convert the result into a reusable format. A scraper can be run by hand, but the term usually refers to an automated pipeline that downloads a page, extracts the wanted values, and stores them for later analysis.
APIs versus scraping
When a site exposes a clean API, use it. APIs usually return JSON or XML and are easier to consume than HTML. Scraping becomes necessary when no API exists, when the API is rate-limited, or when the available data is incomplete. The trade-off is maintenance: a scraper depends on the page structure, so a markup change can break extraction. A practical rule is to check for an official API first, then fall back to scraping only the pages you actually need.
Common use cases
Scraping is used across news aggregation, real-estate listings, travel price comparison, e-commerce monitoring, bank account aggregation, data journalism, SEO analysis, market research, and lead generation. The common thread is fragmented data spread across many sites. When an industry has a web presence split across dozens or hundreds of pages, an aggregation opportunity usually exists. The same maintenance risk applies: each source site can change independently.
HTTP and DOM foundations
HTTP is a client/server protocol. A client sends a request with a method, path, headers, and sometimes a body; the server responds with a status code, headers, and content. The important request headers for scraping include Host, User-Agent, Accept, Cookie, and Referer. Status codes fall into familiar classes: 2XX success, 3XX redirect or action required, 4XX client error, and 5XX server error. The browser then parses the HTML into a Document Object Model, a tree of element and text nodes. The DOM you inspect in a browser can differ from the original HTML because JavaScript may have modified it after load.
Choosing an extraction approach
There are two broad approaches. A headless browser handles HTML parsing, cookies, forms, JavaScript execution, and iframes, but carries more overhead. A plain HTTP library plus a parser such as Jsoup is lighter and faster when the page does not require JavaScript. Use the lighter path first; move to a headless browser only when the content is rendered client-side or when you need browser-level behavior such as screenshots or complex form interaction.
XPath and regular-expression extraction
XPath selects nodes by path, predicate, or attribute. It is useful when the page has a stable structure and you can target elements directly. Regular expressions are useful when the data is embedded in a larger block of text, such as a price or date inside a paragraph. In practice, use XPath to narrow the search to the right element, then use a regex to pull a specific value from that element's text. Both techniques are fragile if the page structure or wording changes, so keep selectors as specific as necessary but no more brittle than the data requires.
A worked example: Hacker News items
The Hacker News example shows the full extraction loop. The scraper fetches the page, selects each row with the class athing, then reads the position, title, URL, author, score, and item ID using relative XPath expressions. The extracted values are placed into a plain Java object and serialized to JSON with Jackson. This pattern generalizes: identify the repeating container, select fields relative to that container, and map the result into a typed object before exporting. The exact selectors are tied to the page markup, so verify them against the current DOM before relying on them.
JavaScript-rendered pages
Modern front-end frameworks often render content after the initial HTML load. A lightweight headless browser may fetch both the HTML and the JavaScript but still leave the page incomplete because it cannot run every script. There are two practical responses: use a more capable browser such as headless Chrome, or inspect the network calls the page makes and reproduce the underlying API requests directly. The second approach is often faster and more stable, but it requires understanding the request parameters and authentication. When the page depends on WebSockets or complex client-side state, a real browser is usually the safer choice.
Headless Chrome and Selenium
Headless Chrome runs the browser without a visible window and is controlled through Selenium. The WebDriver object sends commands to the browser, and WebElement objects represent DOM nodes. Useful methods include get, getCurrentUrl, getPageSource, navigate, switchTo, manage cookies, findElement, click, getText, sendKeys, and getAttribute. Waiting is a common source of failures: a fixed sleep may be too short or too long, so prefer explicit waits such as WebDriverWait with an ExpectedCondition that matches the element you need. For infinite scroll, you can scroll the window, call a JavaScript function directly, or call the underlying API; the API approach is usually the most efficient when it is available.
Captchas, PDFs, and OCR
Captcha solving is slow and costly, so the first goal should be to avoid triggering captchas through polite request rates and realistic headers. When a captcha must be solved, third-party services can return a token that is placed into the page's hidden response field before form submission; any named service, pricing, or terms must be confirmed by the owner (OWNER_CAPTCHA_SERVICE_TERMS) before publication. PDF extraction follows a different path: download the file, verify its content type, then use a PDF library to extract text and apply a regex to the relevant line. OCR is the fallback for text embedded in images; Tesseract can recognize text, but accuracy depends on language data, image quality, and preprocessing such as cropping or contrast adjustment.
Staying unblocked
Anti-bot systems look at headers, request timing, IP reputation, cookies, and hidden elements. Sending a realistic set of browser-like headers is a basic step. Randomizing the User-Agent helps, but it must be consistent with the rest of the request. Proxies distribute requests across IP addresses; free proxy lists are often unreliable, so paid or self-hosted pools are usually better. TOR hides the origin IP but its exit nodes are public and often blocked. Timing matters: a fixed click cadence is a strong bot signal, so introduce random pauses between requests. Hidden form inputs and invisible elements can be traps; preserve hidden inputs and check visibility before interacting with elements.
Serverless deployment
Cloud functions suit scraping workloads because scraping is I/O-bound and can be parallelized. A function can be triggered by an HTTP request, a queue message, a log event, or an IoT event. The deployment flow typically involves creating a project from an archetype, testing locally, and deploying with a build tool. Cold starts can make the first invocation slow, so plan for that if latency matters. The function can return JSON directly or write results to a database or queue. The same scraper logic used locally can be packaged into a function, but credentials and configuration should be managed through the platform rather than hard-coded. Provider-specific instructions, including Azure function deployment, must be confirmed by the owner (OWNER_CLOUD_PROVIDER_TERMS) before publication.
| Approach | Best for | Trade-off |
|---|---|---|
| Plain HTTP client + parser | Static HTML, simple pages, high volume | Cannot execute JavaScript or handle complex browser behavior |
| Lightweight headless browser (HtmlUnit) | Forms, cookies, some JavaScript, moderate complexity | May fail on modern JavaScript frameworks |
| Full headless browser (Chrome + Selenium) | JavaScript-heavy pages, screenshots, complex interaction | Higher resource use and slower execution |
| Direct API calls | Pages that load data through a discoverable API | Requires reverse-engineering request parameters and authentication |
This is a decision aid, not a benchmark. The right choice depends on the target page and the data you need.
| Control | What to check | Why it matters |
|---|---|---|
| Form action and method | Where the request goes and whether it is GET or POST | Determines the request shape |
| Visible inputs | Name and value for each field | These become request parameters |
| Hidden inputs | CSRF tokens and other hidden values | Must be preserved exactly; changing them can cause rejection |
| Cookies | Session cookie set after login | Required for authenticated follow-up requests |
| Success indicator | A known element or text after submission | Confirms the request actually worked |
Use this as a checklist when a form submission fails or when authenticated pages return unexpected content.
Frequently asked questions
Should I use HtmlUnit or headless Chrome for Java scraping?
Start with HtmlUnit when the page is mostly static or uses simple forms. Move to headless Chrome when the page relies on modern JavaScript frameworks, WebSockets, or complex client-side rendering. The lighter tool is faster and easier to run; the full browser is more capable but heavier.
Why does my Java login scraper fail even when the credentials are correct?
Common causes include missing hidden fields such as CSRF tokens, not preserving cookies between requests, submitting to the wrong action URL, or checking for success incorrectly. Inspect the actual POST request in the browser's developer tools and compare it with the request your scraper sends.
Why is the data missing from the HTML my scraper downloads?
The content may be rendered by JavaScript after the initial HTML load. Check the network panel for API calls that return the data directly, or use a browser that executes JavaScript. If the page uses a discoverable API, calling that API is often simpler and more stable than rendering the full page.
How can I reduce captcha challenges while scraping?
Use realistic headers, rotate User-Agents and proxies where appropriate, add random delays between requests, and avoid aggressive request rates. Captcha solving services are slow and costly, so prevention is usually the better strategy.
curl https://api.coinmarketcap.com/v1/ticker/ethereum/?convert=EUR
{
id: "ethereum",
name: "Ethereum",
symbol: "ETH",
rank: "2",
price_usd: "414.447",
price_btc: "0.0507206",
24h_volume_usd: "1679960000.0",
market_cap_usd: "39748509988.0",
available_supply: "95907342.0",
total_supply: "95907342.0",
max_supply: null,
percent_change_1h: "0.64",
percent_change_24h: "13.38",
percent_change_7d: "25.56",
last_updated: "1511456952",
price_eur: "349.847560557",
24h_volume_eur: "1418106314.76",
market_cap_eur: "33552949485.0"
}
curl https://api.e-commerce.com/products
curl https://api.e-commerce.com/products/123
GET /how-to-log-in-to-almost-any-websites/ HTTP/1.1
Host: ksah.in
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Encoding: gzip, deflate, sdch, br
Connection: keep-alive
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36
POST /login HTTP/1.1
Host: example.com
Content-Type: application/x-www-form-urlencoded
uname=username&psw=my_great_password
GET /sample_page.html HTTP/1.1
Host: www.example.org
Cookie: cookie_name=cookie_value
WebRequest[<url="https://www.javawebscrapingsandbox.com/account/login",
POST, EncodingType[name=application/x-www-form-urlencoded],
[csrf_token=1524752332##6997dd9d5ed448484131add18b41a4263541b5c2,
[email protected],
password=test],
{Origin=https://www.javawebscrapingsandbox.com/account/login,
Accept=text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,
Cache-Control=max-age=0,
Referer=https://www.javawebscrapingsandbox.com/account/login,
Accept-Encoding=gzip, deflate}, null>]
[
{
id: 31,
name: "Marmot Drop Line Men's Jacket, Lightweight 100-Weight Sweater Fleece",
price: "74.96",
url: "https://www.amazon.com/gp/product/B075LC96R2/ref=ox_sc_sfl_title_39?ie=UTF8"
},
{
id: 32,
name: "ASUS ZenPad 3S 10 9.7" (2048x1536), 4GB RAM, 64GB eMMC, 5MP Front / 8MP Rear Camera, Android 6.0, Tablet, Titanium Gray (Z500M-C1-GR)",
price: "296.07",
url: "https://www.amazon.com/dp/B01MATMXZV?tag=thewire06-20"
},
{
id: 33,
name: "LG Electronics OLED65C7P 65-Inch 4K Ultra HD Smart OLED TV (2017 Model)",
price: "2596.99",
url: "https://www.amazon.com/gp/product/B01NAYM1TP/ref=ox_sc_sfl_title_35?ie=UTF8"
},
...
]
curl https://${function_url}/api/hnitems?pageNumber=3
[
{
"title": "Nvidia Can Artificially Create Slow Motion That Is Better Than a 300K FPS Camera (vice.com)",
"url": "https://motherboard.vice.com/en_us/article/ywejmy/nvidia-ai-slow-motion-better-than-a-300000-fps-camera",
"author": "jedberg",
"score": 27,
"position": 121,
"id": 17597105
},
{
"title": "Why fundraising is a terrible experience for founders: Lessons learned (kapwing.com)",
"url": "https://www.kapwing.com/blog/the-terrible-truths-of-fundraising/",
"author": "jenthoven",
"score": 74,
"position": 122,
"id": 17594807
},
{
"title": "Why No HTTPS? (whynohttps.com)",
"url": "https://whynohttps.com",
"author": "iafrikan",
"score": 62,
"position": 123,
"id": 17599022
},
//...