Skip to content

Web Scraping in Scala: jsoup, Scala Scraper and Selenium

Scala can scrape the web through several JVM libraries, and the right choice depends on whether the target page is plain HTML or depends on JavaScript. This guide sets up a Scala 3 project, parses static pages with jsoup and Scala Scraper, and uses Selenium when a headless browser is required.

Set up a Scala project

The examples use Scala 3.6.3 and sbt 1.10.7. Start from the official Scala 3 giter8 template, name the project dev.draft, and add the three scraping dependencies to build.sbt. The exact template command and dependency block are supplied as protected code modules code0001 and code0002; use them unchanged so the versions and coordinates stay consistent with the rest of the guide.

Parse static HTML with jsoup

Create JsoupScraper.scala inside src/main/scala/dev/draft. The entry point calls Jsoup.connect(url).get(), which downloads the page and returns a Document. A related method, parse, works only on documents already stored locally, so it does not fetch anything. The exact starter file is supplied as code0003, the download line as code0004, and the title print as code0005; the captured title output appears in code0006.

Select elements with jsoup

The worked example uses the Wikipedia home page and three sections: In the news, On this day, and Did you know. Inspecting the page reveals the container IDs mp-itn, mp-otd, and mp-dyk. Each section is selected with a CSS selector such as #mp-itn b a, which targets links inside bold items within that container. The exact selection lines are code0007, code0009, and code0011, and the captured element output appears in code0008, code0010, and code0012.

CSS selector rules used here:

  • A bare tag name matches every element with that name.
  • A leading # matches an id, as in #bar.
  • Combined selectors narrow the match, for example bar#baz.foo.
  • A space between selectors means a descendant relationship.
  • The > character restricts the match to direct children.

Beyond select, jsoup offers traversal helpers such as next, nextAll, nextSibling, and nextElementSibling.

Extract text and attributes with jsoup

An HTML element exposes three useful retrieval paths: children for nested elements, text for visible strings, and attr for attribute values. After importing scala.jdk.CollectionConverters.*, the example builds a list of title and href pairs from the On this day section. The exact loop is code0013 and the captured output is code0014. A second loop, code0015, prints only the headline text, with output in code0016.

Use Scala Scraper for the same task

Scala Scraper wraps a browser-like abstraction. JsoupBrowser() is the HTML-only implementation: it does not execute JavaScript. Create ScalaScraper.scala in the same package and call browser.get(url) to download and parse a page, or parseFile for an already downloaded document. The exact file is code0017, the browser constructor is code0018, the download line is code0019, the title print is code0020, and the captured title output is code0021.

Select with Scala Scraper

Import the DSL modules to enable the >> operator. The same three Wikipedia sections are selected with elementList and the same CSS selectors. The exact imports are code0022, the three selection lines are code0023, code0025, and code0027, and the captured element output appears in code0024, code0026, and code0028. The descendant-space rule described for jsoup applies identically here.

Extract with Scala Scraper

Scala Scraper mirrors the jsoup extraction model: children returns child elements, text returns text content, and attr reads an attribute. The example builds title and href pairs with the >> operator, shown exactly in code0029, and prints them with output in code0030. A second loop, code0031, prints only the headline text, with output in code0032.

Know when static parsing stops working

jsoup and Scala Scraper are sufficient for static HTML and for validating generated HTML. They cannot render a dynamic site or a single-page application, because JsoupBrowser only processes HTML. When the content you need appears only after JavaScript runs, move to a headless browser such as Selenium.

Scrape JavaScript-rendered pages with Selenium

Selenium drives a real browser and is used for bots, automated tests, and scraping. Download the WebDriver for your browser; the client setup differs between Firefox and Chrome. For the Firefox driver, place the latest geckodriver release on your system PATH. The exact Selenium example is code0033, which repeats the three Wikipedia section selections with By.cssSelector. A second example, code0034, loads a related-words page and reads the dynamically generated words container; the captured output begins in code0035.

Choose the right approach

Use this checklist before writing the scraper:

  • If the target HTML is present in the initial response, start with jsoup or Scala Scraper.
  • If you already have HTML saved locally, use the parse or parseFile path instead of a network fetch.
  • If the data appears only after JavaScript runs, use Selenium with a headless browser.
  • If you need to interact with the page, such as typing or clicking, Selenium is the appropriate tool.
  • If you need to validate generated HTML in unit tests, jsoup and Scala Scraper both support that use.

The field and control map below shows how the same concepts appear in each library.

Summary

A Scala scraping project can start with jsoup for straightforward HTML parsing, move to Scala Scraper when a browser-like abstraction is convenient, and switch to Selenium when JavaScript rendering is unavoidable. The protected code modules in this guide preserve the exact setup, selection, and extraction examples so you can adapt them to your own target pages.

Approach comparison
ApproachHandles static HTMLRuns JavaScriptSelection methodExtraction methods
jsoupYesNodoc.select(css)text, attr, children
Scala Scraper (JsoupBrowser)YesNodoc >> elementList(css)text, attr, children via >>
SeleniumYesYesdriver.findElement(By.cssSelector(css))getText and WebElement methods

Capabilities reflect the captured examples; browser driver setup and supported features may change between releases.

Frequently asked questions

Should I use jsoup or Scala Scraper?

Both parse static HTML. jsoup exposes a direct Document API with select, text, and attr. Scala Scraper adds a browser abstraction and a DSL with the >> operator, and its JsoupBrowser implementation is built on jsoup. Choose based on which API style fits your codebase; neither executes JavaScript.

What is the difference between connect and parse in jsoup?

connect downloads a page and parses it in one step. parse only parses HTML you already have, such as a local file or a stored response. The same distinction appears in Scala Scraper as get versus parseFile.

When do I need Selenium instead of jsoup?

Use Selenium when the content you need is produced by JavaScript, when the site is a single-page application, or when you must interact with the page. jsoup and Scala Scraper only see the HTML delivered in the initial response.