Skip to content

Web Scraping with Groovy

Groovy gives you the reach of the JVM without the ceremony of Java. That makes it a practical choice for scraping tasks that start with a single HTTP request and grow into HTML parsing, login sessions, and full browser automation. This guide walks through the same progression: a plain GET, a POST with JSON, a multi-page comment crawler, a cookie-based password change, and a Selenium crawler for a JavaScript-heavy search page.

Set up Groovy and the Jodd libraries

You need a recent Java runtime, the Groovy SDK, Jodd HTTP for requests, and Jodd Lagarto for HTML parsing. If you use an IDE such as Eclipse or IntelliJ, Groovy support is usually available as a plugin; otherwise install Groovy manually.

Groovy Grape removes the need to configure Maven or Gradle for these examples. Annotate imports with @Grab and Groovy resolves the library at runtime. The complete request examples are supplied as protected code modules and should be used exactly as provided.

Send a GET request and read the response

A GET request is the baseline. Create an HttpRequest with HttpRequest.get(url), call send(), and store the response. Printing the response calls its toString() method and shows the raw HTTP response. The response object also exposes methods such as statusCode() so you can branch on the HTTP status instead of parsing the printed text.

The supplied GET example uses https://example.com and expects a 200 response with an HTML body. Run it as a Groovy script after saving it to a file. If the response is not 200, check the URL, network access, and whether the target expects headers or cookies.

POST form data and parse JSON with JsonSlurper

For a POST, use HttpRequest.post(url), attach form fields with form(name, value, ...), and send. The supplied example posts to https://httpbin.org/post and parses the response body with Groovy's JsonSlurper. The parsed result is a Java map, so you can read fields directly.

httpbin echoes the request back. In the captured output, the two form parameters appear under the form field, which confirms that the request body was encoded as application/x-www-form-urlencoded. If the form field is missing or empty, check that you passed name/value pairs in the correct order and that the response body was read with bodyText() before parsing.

Why Groovy and Jodd keep the code short

Groovy removes mandatory class scaffolding and gives you built-in JSON handling. Jodd HTTP favors method chaining and a small API surface, so a request with headers, form fields, and cookies can stay compact. The trade-off is that you must still inspect each target page and confirm its current HTML structure before relying on selectors.

Crawl Hacker News comments with Lagarto and Jerry

The first full example collects comments from a Hacker News thread. Open the thread in a browser, inspect the comment table, and confirm the selectors before coding. The captured structure uses a table with class comment-tree and rows with class comtr.

The field map below shows how each output field is derived from a comment row. Use it as a checklist when the page structure changes.

After extracting the rows, the script looks for a rel="next" link. If one exists, it builds the next URL from the Hacker News base and repeats. The captured run against thread 26263508 printed four pages, then the collected comment list. Save the list to a file or database once extraction finishes.

Log in and keep a session cookie

The second example changes a Hacker News password. It needs two things: a valid session cookie and a one-time token. Log in with a POST to https://news.ycombinator.com/login, sending acct, pw, and goto as form fields. A successful login returns a 302 status. Read the cookie with cookies() and keep it for later requests.

The example notes that HttpBrowser could also be used, but redirection issues with Hacker News led to the manual request route. If you use HttpBrowser, verify its redirect behavior against the current site before relying on it.

Fetch the one-time token before changing the password

Request https://news.ycombinator.com/changepw with the session cookie. Parse the page with Jerry and read the value of the hidden input named fnid. The supplied selector is input[name="fnid"]. If the token is empty, the session may have expired or the page structure may have changed; re-check the login step before sending the password-change request.

Send the password-change POST and check the redirect

The final POST goes to https://news.ycombinator.com/r with the session cookie and form fields fnid, fnop, oldpw, and pw. The captured code uses fnop=changepw-page. A successful change logs the user out and redirects to /news. The example checks the Location header for news and prints a boolean result.

Failure distinction: a login failure is detected by a status code other than 302, while a password-change failure is detected by a Location header that does not contain news. Check both separately so you do not treat an expired session as a wrong new password.

Know when a single HTTP request is not enough

Static or server-rendered pages can be scraped with one HTTP request and manual DOM parsing. Client-rendered pages send an HTML skeleton with links to JavaScript resources; the visible content appears only after the browser runs that JavaScript. In that case, use a real browser environment such as Selenium with ChromeDriver.

Decision checklist:

  • Does the page source already contain the target text and elements? If yes, start with Jodd HTTP and Lagarto.
  • Does the page show placeholders or empty containers until JavaScript runs? If yes, move to Selenium.
  • Does the target require login, scrolling, or clicking? If yes, use a browser driver and manage cookies there.
  • Does the selector depend on a parent element? Use XPath when CSS cannot express the parent relationship.

Crawl a JavaScript search page with Selenium

The third example collects tweet links for a search query. It uses Selenium Java bindings, a recent Chrome browser, and a ChromeDriver binary that matches the browser version. Set the webdriver.chrome.driver system property if the driver is not in the working directory, then create ChromeOptions with headless mode and a window size.

The script loads https://twitter.com/search with q and src parameters, waits for main section to appear, and then selects tweet links with an XPath expression. The expression targets div elements with data-testid="cellInnerDiv" that contain a time descendant, then selects the time element's parent. That parent is the link element whose href is stored. A Set removes duplicate links. The page is scrolled with PAGE_DOWN and the loop continues until the set reaches MAX_TWEETS.

The captured DOM structure is a dated reference. Twitter changes its markup, so re-inspect the page before reusing the selector. The example also uses a fixed 1.5-second pause after scrolling; adjust waits if the page loads more slowly.

Choose the lightest tool that fits the page

Groovy keeps the code compact across all three examples. Jodd HTTP and Jodd Lagarto are a good fit when the HTML already contains the data. Selenium is the right choice when the browser must execute JavaScript, scroll, or interact with the page. If you need to avoid blocks or manage proxies and request throttling, that is a separate operational concern; plan for it before scaling a crawler.

Hacker News comment field map
Output fieldSource elementSelector or attribute
idcomment rowid attribute of table.comment-tree tr.comtr
userusername linktext of a.hnuser
timeage spantitle attribute of span.age
commentcomment bodytext of div.comment
parentparent linkhref attribute of a:contains(parent)

Selectors reflect the captured Hacker News markup and should be re-checked before reuse.

Static scraping versus headless browser scraping
SignalStatic pathHeadless path
Target text in page sourcePresentAbsent or placeholder
JavaScript execution neededNoYes
Typical toolsJodd HTTP, Jodd LagartoSelenium, ChromeDriver
Interaction neededUsually noneScroll, click, type
Session handlingManual cookiesDriver-managed cookies

Use this as a starting checklist; some sites mix server-rendered and client-rendered content.

Frequently asked questions

Do I need Maven or Gradle to use Jodd with Groovy?

No for these examples. Groovy Grape resolves dependencies from @Grab annotations at runtime. If you move the code into a larger project, use your normal build tool and keep the same library versions or update them deliberately.

When should I use a POST request instead of a GET request?

Use GET when you are retrieving a page or resource without sending data. Use POST when you need to submit form fields such as credentials, a token, or a password change. The Hacker News login and password-change examples both use POST because they send account data to the server.

How do I keep a login session across requests in Groovy?

After a successful login, read the cookie with the response's cookies() method and store it. Pass that stored cookie to later requests with cookies(sessionCookie). If a later request behaves like an anonymous user, check whether the cookie was lost or expired.

Why does the Twitter example use XPath instead of a CSS selector?

The script needs the parent of the time element. CSS selectors in the captured approach cannot select a parent directly, so an XPath expression expresses that relationship in one step. The expression selects div elements with data-testid="cellInnerDiv" that contain a time descendant, then takes the time element's parent.

When should I switch from Jodd HTTP to Selenium?

Switch when the data is not present in the initial HTML and appears only after JavaScript runs. A quick check is to view the page source: if the target content is missing there but visible in the browser, a headless browser is the safer path.

[args:[:], data:, files:[:], form:[param1:value1, param2:value2], headers:[Content-Length:27, Content-Type:application/x-www-form-urlencoded, Host:httpbin.org, User-Agent:Jodd HTTP, json:null, url:https://httpbin.org/post]