OCaml Web Scraping
OCaml is a type-safe functional language, and it can drive a complete scraping workflow: fetch a page, parse the HTML, select elements with CSS, and print the fields you need. For pages that build their content with JavaScript, the same language can control a real browser through the WebDriver protocol. This guide walks through both paths, from project setup to the final output.
Before you start
You need OCaml installed, OPAM initialized, and Dune available. The official OCaml installation instructions cover these steps. For the static example you will install three packages: cohttp, cohttp-lwt-unix, and lambdasoup. For the dynamic example you will also need a browser driver (this guide uses GeckoDriver for Firefox), the matching browser installed, and the ocaml-webdriver library, which is not in the OPAM registry and must be built from its Git repository.
Scraping a static HTML page
The first target is a simple quotes site whose first page lists quotes in div elements with the class quote. Each quote contains a span with class text for the quote and a span with class author for the author name. Because the page is plain HTML, an HTTP client plus an HTML parser is enough.
Create a Dune project, change into its directory, and install the dependencies. Then edit bin/dune so the libraries section includes cohttp, cohttp-lwt-unix, and lambdasoup. The exact project and dependency commands are provided in the code modules below.
In bin/main.ml, open Lwt, Cohttp_lwt_unix, and Soup. The request definition fetches http://quotes.toscrape.com/ and parses the response body into a Lambda Soup node. That definition does not perform the request by itself; the request is executed later when the program runs.
To choose selectors, inspect the page in your browser's developer tools. The quotes are selected with .quote, and within each quote the text and author are selected with .text and .author. Lambda Soup's $ operator returns the first matching element and $$ returns all matches. The code iterates over the quote list, extracts leaf text, trims whitespace, and prints each quote with its author.
After building and executing the project, the program prints every quote on the first page. The full code is in the protected code module, and the exact build and run commands are provided alongside it.
Working with CSS selectors
Lambda Soup supports all CSS3 selectors except a specific set of dynamic and pseudo-element selectors. The excluded list covers link states, user-action states, form states, and pseudo-elements such as ::before and ::after. If a selector depends on interaction or generated content, it will not work here.
A practical example is selecting only the first quote. Instead of taking the first item from the full list, you can use the :first-child selector with the path .quote:first-child .text. The code extracts the leaf text, trims it, and prints it as the first quote. The full program in the protected module combines the full-list output with this single-quote output.
Scraping a JavaScript-rendered page
Lambda Soup cannot execute JavaScript, so it cannot handle pages that load data after the initial HTML arrives. For those pages, use the WebDriver protocol to control a real browser. The browser executes the page's JavaScript, and your OCaml code sends commands to it.
The example target is a site that lists Oscar-winning movies. The list is empty by default, and clicking a year link such as 2015 loads the movie data with JavaScript. Download the WebDriver for your chosen browser and install the browser itself. This guide uses GeckoDriver for Firefox. Clone the ocaml-webdriver repository, install cohttp-async, then build and install the library with Dune.
Create a separate Dune project for the dynamic scraper and add webdriver_cohttp-lwt-unix to the libraries section of bin/dune. In bin/main.ml, open Webdriver_cohttp_lwt_unix and Infix. Two helper functions make the code easier to read: list_iter walks a list of elements, and wait retries a command until an element appears, catching the no_such_element error and sleeping between attempts.
The action sequence navigates to the target page, finds the first link with the CSS selector a.year-link, clicks it, waits for an element with the selector tr.film, then selects all elements matching tr.film .film-title. It iterates over those elements, extracts each movie name with text, and prints it.
Finally, connect to the driver at http://127.0.0.1:4444, which is GeckoDriver's default address. Change the host if you use a different driver, and change the Capabilities.firefox_headless value to a capability accepted by your driver. The program catches Webdriver errors, prints the error string and backtrace to stderr, and exits. Build and execute the project to see the movie names in the output.
Static or dynamic: choosing the right path
Use this comparison to decide which approach fits your target page before writing code.
Reading the two common failures
The dynamic example distinguishes two failure modes. A no_such_element error means the selector did not match an element at the moment the command ran. The wait helper handles this by retrying after a short sleep, which is useful when content appears after a click. A Webdriver error is a broader driver or protocol failure; the example prints its error string and a backtrace to stderr. If you see the second kind, check that the driver is running, that the host address matches the driver's listening address, and that the capability value is accepted by your driver.
Pre-run checklist
For the static project: confirm OCaml, OPAM, and Dune are installed; confirm cohttp, cohttp-lwt-unix, and lambdasoup are installed; confirm bin/dune lists all three libraries; and confirm the target URL and selectors match the page you inspected.
For the dynamic project: confirm the browser and its WebDriver are installed and the driver is running; confirm ocaml-webdriver is built and installed; confirm bin/dune lists webdriver_cohttp-lwt-unix; confirm the host matches the driver's address; and confirm the capability value is accepted. If the page loads data after a click, keep the wait step before selecting the loaded elements.
| Aspect | Static page | JavaScript-rendered page |
|---|---|---|
| Fetching | Cohttp Lwt request | Browser controlled through WebDriver |
| Parsing | Lambda Soup parses the response body | Browser renders the page; OCaml reads elements through the driver |
| Selecting | CSS selectors with $ and $$ | CSS selectors passed to find_first and find_all |
| JavaScript | Not executed | Executed by the browser |
| Extra setup | cohttp, cohttp-lwt-unix, lambdasoup | Browser, WebDriver, ocaml-webdriver, cohttp-async |
| Waiting for content | Not needed for the static example | wait helper retries on no_such_element |
This table summarizes the two workflows shown in the source tutorial; it is not a claim about every page or every library version.
Frequently asked questions
Why can't Lambda Soup scrape the movie list?
Lambda Soup parses HTML but does not execute JavaScript. The movie list is empty in the initial HTML and is filled only after a year link is clicked, so a browser must run the page's scripts before the elements exist.
What does the wait helper do?
It retries a command when the driver reports no_such_element, sleeping briefly between attempts. This handles elements that appear after an interaction such as a click.
Why is the host set to 127.0.0.1:4444?
That is the default address GeckoDriver listens on. If you use a different driver, change the host to match its listening address.
Is ocaml-webdriver available through OPAM?
No. The tutorial says it is not in the OPAM registry, so you clone its Git repository, install cohttp-async, and build and install it with Dune.