Skip to content

Haskell Web Scraping with Scalpel and Selenium

Haskell's static types can help keep a scraper aligned with the shape of data it expects. This guide covers two practical paths: parsing static HTML with Scalpel, and driving a browser with webdriver when JavaScript creates the content you need.

Choose the approach that matches the page

Static pages are the right starting point when the markup already contains the information you need. Scalpel combines fetching and parsing in a high-level workflow, while Haskell types help describe the expected result before extraction begins. The worked example uses a Wikipedia list of large cities, but the same pattern applies to any table whose rows share a predictable structure.

Set up the Haskell project

Install GHC and Stack through GHCup, create a project with protected module code0001, enter it with code0002, install Scalpel with code0003, and add the dependencies shown in code0004. The text library handles string values. In Main.hs, code0005 imports the required modules and enables OverloadedStrings, which Scalpel needs. Keep building in that file as each scraper is added.

Run a first scraper

A scraper describes what to find in HTML and what to return. The simple heading scraper in code0006 returns the text of the first h1 element. scrapeURL, shown in code0007, binds that scraper to a URL and returns an action that main can run. Code0008 handles both outcomes: a found value is printed, while Nothing produces a clear message. Running the combined program in code0009 with stack run should print the page heading.

Combine selectors and narrow the context

Use attr from code0010 when the value lives in an attribute rather than visible text. The // operator in code0011 matches a descendant, similar to a CSS descendant selector. Attribute predicates such as hasClass, used in code0012, narrow the match further. When a page is easier to understand in stages, chroot selects a container and passes only that subtree to the next scraper, as demonstrated in code0013.

Turn table rows into typed records

Define a City record with name, country, and population fields in code0014. The table scraper in code0015 first enters the matching table, then applies a row scraper to each tr element. For every row, it reads the header as the name and the first two td values as country and population. Missing cells fall back to Not available, and strip removes surrounding whitespace. Code0016 runs the collection, while code0017 shows the complete program. This layered design is useful because each step has one job: locate the table, iterate rows, extract fields, and assemble records.

Use a browser for JavaScript-generated content

Scalpel parses HTML but does not execute JavaScript. If quotes or other content appear only after browser rendering, use webdriver with Selenium instead. The guide's setup needs Selenium server version 2, a ChromeDriver matching the installed Chrome version, and the webdriver dependency in code0018. Start the server with code0019, then connect from Haskell. In code0020, chromeConfig describes the browser session, getQuotes opens the JavaScript version of Quotes to Scrape, finds span.text elements, and returns their text. Running it should open Chrome, load the page, close the session, and print the quotes.

Practical checklist

Before writing code, inspect whether the needed content is present in the initial HTML. If it is, start with Scalpel and keep selectors as narrow as possible. If the content is missing until scripts run, switch to browser automation and verify that the driver and server versions match. For repeated extraction, model the result as a record, handle missing fields explicitly, and test the Nothing path as well as the successful path.

Static HTML or browser automation?
SituationApproachWhy it fits
Content is present in fetched HTMLScalpelParses markup directly and returns typed results
Content appears only after JavaScript runswebdriver with SeleniumControls a browser so rendered elements become available
Data is organized as repeated table rowschroot plus chrootsNarrows context to the table, then applies one row scraper repeatedly

Use the initial HTML test as the deciding factor; browser automation adds server and driver setup.

Frequently asked questions

When should I use Scalpel instead of Selenium in Haskell?

Use Scalpel for pages whose HTML is available without browser execution. Its scrapers locate elements and return text, attributes, or nested results. When content is generated by JavaScript, a plain HTTP fetch will not see the rendered result, so browser automation is needed.

What do I need to set up a Scalpel project?

You need GHC and Stack, installed through GHCup. Then create a project, enter its folder, install Scalpel, and add Scalpel plus text to the dependencies in package.yaml.

How do Scalpel selectors and chroot help with nested HTML?

Use attr when you need an attribute value, // to match nested elements, @: with predicates such as hasClass for attribute conditions, and chroot to narrow the HTML context before running the next scraper.

What is required for the webdriver example?

The webdriver setup in the guide requires a Selenium server version 2 and a matching ChromeDriver placed alongside the code and server. The webdriver dependency must also be added to package.yaml before starting the server.

{-# LANGUAGE OverloadedStrings #-}

module Main (main) where

import           Data.Text
import           Text.HTML.Scalpel

scraper :: IO (Maybe Text)
scraper = scrapeURL "https://en.wikipedia.org/wiki/List_of_largest_cities" heading

  where
    heading :: Scraper Text Text
    heading = text "h1"

main :: IO ()
main = do
  result <- scraper
  case result of
    Just x  -> print x
    Nothing -> print "Didn't find the necessary items."
{-# LANGUAGE OverloadedStrings #-}

module Main (main) where

import           Data.Text
import           Text.HTML.Scalpel


data City = City
  { name       :: Text
  , country    :: Text
  , population :: Text
} deriving (Show, Eq)

allCities :: IO (Maybe ([City]))
allCities = scrapeURL "https://en.wikipedia.org/wiki/List_of_largest_cities" table

  where

    table :: Scraper Text [City]
    table = chroot ("table" @: [hasClass "static-row-numbers"]) cities

    cities :: Scraper Text [City]
    cities = chroots "tr" city

    city :: Scraper Text City
    city = do
      name <- text "th"
      rows <- texts "td"
      let country = getCountry(rows)
      let population = getPopulation(rows)
      return $ City (strip $ name) country population

    getCountry(x: xs) = strip x
    getCountry(_)     = "Not available"

    getPopulation(x: y: xs) = strip y
    getPopulation(_)        = "Not available"


main :: IO ()
main = do
  result <- allCities
  case result of
    Just x  -> print x
    Nothing -> print "Didn't find the necessary items."